Neural structure search device based on template and method therefor

Abstract

The present invention relates to a neural structure search device and a method therefor. The neural structure search device according to one embodiment comprises: a search space configuration unit which configures a plurality of operator templates by matching, to each other, hardware blocks corresponding to the plurality of operators, and which configures a search space on the basis of the plurality of operator templates; and a neural structure search unit for deriving a target neural network through a neural search process based on the search space.

Description

Template-based neural structure search device and method thereof

[0001] The present invention relates to a neural structure search device and method thereof, and more particularly, to a technical idea for searching a neural network structure using a search space composed of a template structure.

[0002] This application is the result of research conducted with the support of the National Research Foundation of Korea (No. RS-2023-00213548) and the National IT Industry Promotion Agency (NIPA) (No. RS-2022-00155966, Artificial Intelligence Convergence Innovation Talent Development (Ewha Womans University), National Project Number: 1711179344, No. 2021-0-02068, Artificial Intelligence Innovation Hub Research and Development) with funding from the government (Ministry of Science and ICT) in 2024.

[0003] NAS (Neural Architecture Search) is a method of AutoML (Automated Machine Learning) that automatically finds the structure of a deep learning model.

[0004] Existing NAS often operates slowly on hardware because they find the structure of deep learning models by considering only the accuracy of the model. Even when considering hardware, there were many NAS techniques that operated tailored to specific hardware.

[0005] Accordingly, interest in the NAS technique (Hardware co-design NAS) that finds a deep learning model and hardware optimized for the model is increasing, but NAS that finds the structure of existing hardware and deep learning model together has the problem of a large search space.

[0006] Specifically, the scope of search in existing technologies has significantly increased as the hardware search space (e.g., width and height of processing elements, buffer size, etc.) has been added to the search space of deep learning models (e.g., number of channels, number of layers, etc.).

[0007] In addition, existing technologies calculate the performance of candidate models and hardware structures (i.e., hardware blocks) in real time during exploration. However, this method is time-consuming and may produce inaccurate measurements, which may result in deriving structures that are not optimal among hardware candidates.

[0008] The present invention seeks to provide a neural architecture search device and method that first finds an optimal hardware structure for each operation and defines a new basic element, operator-hardware pairs, that pair each operation with an optimal hardware block to construct a search space.

[0009] In addition, the present invention aims to provide a neural structure search device and method that can minimize the size of the search space and the time and number of resources required for neural network search based on operator-hardware pairs.

[0010] A neural structure search device according to one embodiment of the present invention may include a search space construction unit that configures a plurality of operator templates by matching hardware blocks corresponding to a plurality of operators, and a neural structure search unit that derives a target neural network through a neural search process based on the search space.

[0011] According to one side, the plurality of operators may include at least one of a skip connection operator, an average pooling operator, a max pooling operator, a standard convolution operator, and a depthwise convolution operator.

[0012] According to one side, the search space configuration unit can configure multiple operator templates by designing hardware blocks corresponding to multiple operators to have 100% utilization.

[0013] According to one side, the neural structure search unit can derive a target neural network through a neural search process based on cell-based search and gradient descent.

[0014] According to one side, the neural structure search unit extracts a plurality of cells each including a target operator template that is composed of a plurality of nodes constituting a target neural network and at least one operator template among a plurality of operator templates, and connects the plurality of nodes to each other, sets a weight for each of the at least one operator templates, and derives at least one candidate neural network corresponding to the target neural network as a result of a learning process that updates the weight.

[0015] According to one side, the neural structure search unit can derive a candidate neural network by applying at least one fully connected layer and a softmax activation function to multiple cells.

[0016] According to one side, the neural architecture search unit can derive a target neural network based on an evaluation process for candidate neural networks.

[0017] According to one side, the neural architecture search unit can derive a target neural network based on an evaluation process that evaluates model accuracy and hardware cost for candidate neural networks.

[0018] According to one side, the neural architecture search unit can derive a target neural network in response to a loss function based on cross-entropy loss and hardware cost.

[0019] A neural structure search method according to one embodiment of the present invention may include a step of configuring a plurality of operator templates by matching hardware blocks corresponding to a plurality of operators in a search space configuration unit, and configuring a search space based on the plurality of operator templates, and a step of deriving a target neural network through a neural search process based on the search space in a neural structure search unit.

[0020] According to one embodiment, the present invention can construct a search space by first finding an optimal hardware structure for each operation and defining an operator-hardware pair that pairs each operation with an optimal hardware block as a new basic element.

[0021] According to one embodiment, the present invention can minimize the size of the search space and the time and number of resources required for neural network search based on operator-hardware pairs.

[0022] FIG. 1 is a drawing for explaining a neural structure exploration device according to an embodiment.

[0023] FIGS. 2A to 2F are drawings specifically illustrating an example of designing a hardware block in a neural structure search device according to one embodiment.

[0024] Figure 3 is a drawing illustrating a neural structure search method according to an embodiment.

[0025] Specific structural or functional descriptions of embodiments according to the concept of the present invention disclosed in this specification are merely illustrative for the purpose of explaining embodiments according to the concept of the present invention, and embodiments according to the concept of the present invention may be implemented in various forms and are not limited to the embodiments described in this specification.

[0026] Embodiments according to the concept of the present invention may have various modifications and take various forms, and thus, embodiments are illustrated in the drawings and described in detail in this specification. However, this is not intended to limit embodiments according to the concept of the present invention to specific disclosed forms, but rather includes modifications, equivalents, or alternatives that fall within the spirit and technical scope of the present invention.

[0027] While terms such as "first" or "second" may be used to describe various components, the components should not be limited by the terms. Terms are used solely to distinguish one component from another. For example, a first component may be referred to as a "second component," and similarly, a second component may also be referred to as a "first component," without departing from the scope of the invention.

[0028] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components in between. Conversely, when a component is referred to as being "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between. Expressions that describe relationships between components, such as "between," "immediately between," or "directly adjacent to," should be interpreted similarly.

[0029] The terminology used herein is for the purpose of describing specific embodiments only and is not intended to limit the present invention. The singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, it should be understood that the terms "comprises" or "has" are intended to specify the presence of a described feature, number, step, operation, component, part, or combination thereof, but do not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0030] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0031]

[0032] Hereinafter, embodiments will be described in detail with reference to the attached drawings. However, the scope of the patent application is not limited or restricted by these embodiments. The same reference numerals in each drawing represent the same components.

[0033]

[0034] FIG. 1 is a drawing for explaining a neural structure exploration device according to an embodiment.

[0035] Referring to FIG. 1, the neural architecture search device (100) can first find the optimal hardware structure for each operation and define an operator-hardware pair that pairs each operation with the optimal hardware block as a new basic element to configure a search space.

[0036] In addition, the neural network search device (100) can minimize the size of the search space and the time and number of resources required for neural network search based on operator-hardware pairs.

[0037] To this end, the neural structure search device (100) may include a search space configuration unit (110) and a neural structure search unit (120).

[0038] A search space configuration unit (110) according to an embodiment can configure a plurality of operator templates (i.e., operator-hardware pairs) by matching hardware blocks corresponding to a plurality of operators, and configure a search space based on the plurality of operator templates.

[0039] In other words, the search space configuration unit (110) can derive each hardware block having an optimal structure corresponding to a plurality of operators in advance, and configure a plurality of operator templates by matching each derived hardware block to each of the corresponding plurality of operators.

[0040] Specifically, the search space configuration unit (110) comprises a plurality of operator templates as in Equation 1 below. ) based on the search space as shown in Equation 2 below. ) can be configured.

[0041] [Formula 1]

[0042]

[0043] [Formula 2]

[0044]

[0045] Here, is the ith operator (where i is a positive integer), is an operator A hardware block that implements , where N represents the number of operators.

[0046] That is, the existing technology uses a method of first stacking operations and then searching the hardware, so that the entire search space is very large as it is the product of the search space of operations and the search space of hardware, but the search space based on multiple operator templates of the present invention can significantly reduce the number of operations by configuring the optimal operator-hardware pair as a template, and accordingly, the neural structure search process is simplified to a problem of only determining the method of stacking operations, thereby minimizing the search time and the number of required resources.

[0047] In addition, the search space based on multiple operator templates can measure the performance of the correct answer candidate through calculations because it finds the optimal hardware structure for the operation before searching the neural architecture. Using these calculations, the performance of the hardware can be measured relatively accurately in a short period of time, so that the most optimal neural network (i.e., deep learning model) and its corresponding hardware structure can be found.

[0048] For example, the plurality of operators may include at least one of a skip connection operator, an average pooling operator, a max pooling operator, a standard convolution operator, and a depthwise convolution operator.

[0049] For a more specific example, the plurality of operators may include, but are not limited to, the operators in Table 1 below.

[0050]

[0051] According to one side, the search space configuration unit (110) can configure multiple operator templates by designing hardware blocks corresponding to multiple operators to have 100% utilization. In other words, the search space configuration unit (110) can maximize efficiency by optimizing the hardware block design so that each component of the hardware block corresponding to each of the multiple operators can be fully utilized.

[0052] A method for designing a hardware block to have 100% utilization in a search space configuration unit (110) according to an embodiment will be described in more detail later with reference to FIGS. 2a to 2f.

[0053] A neural structure search unit (120) according to an embodiment can derive a target neural network (i.e., a deep learning model) through a neural search process based on a search space.

[0054] According to one side, the neural structure search unit (120) can derive a target neural network through a neural search process based on cell-based search and gradient descent, thereby reducing search costs and solving optimization problems.

[0055] Here, gradient descent is one of the optimization algorithms used to train models by adjusting parameters in machine learning and optimization problems. It searches for the minimum value of a function by utilizing the gradient (slope) of a given function, and can derive an optimized value by repeatedly moving in the direction of decreasing gradient.

[0056] According to one side, the neural structure exploration unit (120) can derive a target neural network through a neural exploration process consisting of a cell extraction process, an operator weight setting process, a weight update process, an operator selection process, and an operator evaluation process.

[0057] Specifically, the neural structure exploration unit (120) can extract a plurality of cells each including a plurality of nodes constituting a target neural network and a target operator template that is composed of at least one operator template among a plurality of operator templates and that connects the plurality of nodes to each other.

[0058] For example, the neural structure exploration unit (120) can extract a plurality of cells including normal cells and reduction cells, wherein the normal cells and reduction cells can be arranged in a preset order.

[0059] For more specific examples, a normal cell may be located between two other normal cells, or between another normal cell and a reduction cell, and a reduction cell may be located between two normal cells, or between every M normal cells (where M is a positive integer).

[0060] Next, the neural structure search unit (120) can set the weight of each of at least one operator templates.

[0061] In other words, the neural structure search unit (120) can set a weight for each candidate operator template (i.e., at least one operator template among a plurality of operator templates provided in the search space) for the target operator template.

[0062] For example, the weight of the operator template may be set to an optimized value because if too many candidate operators are used, the weight corresponding to each candidate operator may be too small, which may cause learning not to be performed properly, and the search space may become too large, which may cause learning to take an excessively long time. Here, the optimized value of the weight may be set according to the input of the administrator.

[0063] Next, the neural structure search unit (120) can derive a candidate neural network corresponding to the target neural network as a result of a learning process for updating weights (i.e., a neural network learning process), and derive a target neural network based on an evaluation process for the candidate neural network.

[0064] According to one side, the neural structure exploration unit (120) can derive a candidate neural network by applying at least one fully connected layer and a softmax activation function to multiple cells.

[0065] In other words, the neural structure exploration unit (120) can build a complete network by stacking multiple cells multiple times and applying a fully connected layer and a softmax activation function.

[0066] For example, the neural structure search unit (120) can receive learning data corresponding to the target neural network and update weights through a learning process based on the received learning data.

[0067] Additionally, the neural architecture exploration unit (120) can derive a target neural network based on an evaluation process that applies a candidate neural network to an evaluation set, wherein the evaluation set can include a means for evaluating model accuracy and hardware cost for the candidate neural network.

[0068] For a more specific example, the neural network search unit (120) can derive a target neural network in response to a loss function based on cross-entropy loss and hardware cost.

[0069] Specifically, the neural structure exploration unit (120) can derive a target neural network through an evaluation process based on the following equation 3.

[0070] [Formula 3]

[0071]

[0072] Here, is the operator weight according to the selection of candidate operator templates, is the network weight, means the optimal weight.

[0073] That is, the neural structure search unit (120) finds the optimal weight ( ) as described in Equation 3, the training loss ( ) can be derived by minimizing the training loss ( ) corresponding to the loss function ( ) can be derived through the following equation 4.

[0074] [Formula 4]

[0075]

[0076] Here, is the cross entropy loss, is the hardware cost (e.g., latency, area, energy consumption), means hyperparameter.

[0077] Meanwhile, as shown in Table 2 below, it can be confirmed that the neural structure search device (i.e., TD-NAAS) (100) according to one embodiment has an accuracy increased by 0.8% and 0.2%, and a search speed increased by 2.75 times compared to existing technologies (i.e., EDD, DANCE).

[0078] Here, the experimental results in Table 3 show that the CIFAR-10 dataset was used as learning data, the optimal hardware structure for the operator was determined using Synopsys Design Compiler, the post-synthesis RTL simulation results were used as hardware performance evaluation values, and the deep learning model was trained using PyTorch and a single NVIDIA RTX A6000 GPU.

[0079]

[0080] FIGS. 2A to 2F are drawings specifically illustrating an example of designing a hardware block in a neural structure search device according to one embodiment.

[0081] Referring to FIGS. 2A to 2F, reference numeral 210 is a drawing for explaining a method for optimizing design of a hardware block corresponding to a 3x3 standard convolution operator in a neural structure search device according to an embodiment, and reference numerals 220 to 240 are drawings for explaining a method for optimizing design of a hardware block corresponding to a 2x2 standard convolution operator.

[0082] In addition, drawing reference numeral 250 is a drawing for explaining a method for optimizing design of a hardware block corresponding to a 3x3 average pooling operator in a neural architecture search device, and drawing reference numeral 260 is a drawing for explaining a method for optimizing design of a hardware block corresponding to a 3x3 maximum pooling operator in a neural architecture search device.

[0083] Specifically, a neural structure search device according to an embodiment can configure a plurality of operator templates by matching hardware blocks corresponding to a plurality of operators, and configure a search space based on the plurality of operator templates.

[0084] In other words, the neural network architecture search device defines a new basic element by bundling operators and their corresponding hardware blocks into a single template, and constructs a search space with these basic elements. This reduces the size of the search space compared to existing technologies, enabling efficient search, and thus supporting the derivation of a neural network structure with better performance.

[0085] Additionally, the neural network structure search device can support more efficient processing of neural network structures by optimizing the design so that hardware blocks corresponding to operators always maintain 100% utilization.

[0086] For example, if the operator is a 3x3 standard convolution operator, as illustrated in the drawing reference numeral 210, the neural network search device can configure a hardware block including nine multipliers corresponding to nine activation values ​​(I0 to I8) and nine weights (W0 to W8) and an addition tree that adds them all, and can supply 3x3 convolution windows in a row to the hardware block, so that the hardware block can be designed so that the utilization is always 100%.

[0087] More specifically, as exemplified by the 2x2 convolution operator illustrated in reference numerals 220 to 240, the neural architecture search device can be designed such that by creating four multipliers and constructing a 4:1 addition-tree (reference numeral 220), all of the multipliers (i.e., four multipliers) are in operation, so that the utilization of the hardware block is always 100%.

[0088] In this case, it can be confirmed that the utilization of the hardware block still remains at 100% even when the convolution window is slid from (I0, I1, I5, I6) to (I1, I2, I6, I7) (drawing symbol 230).

[0089] On the other hand, if a hardware block corresponding to a 2x2 convolution operator, such as a neural structure search device according to an embodiment, is not designed to be optimized with 4 multipliers but with 5 multipliers to form a 5:1 addition tree (drawing reference numeral 240), it can be confirmed that only 80% of the multipliers (i.e., 4 out of 5 multipliers) are operated, resulting in inefficiency as the utilization of the hardware block becomes 80%.

[0090] For example, if the operator is a 3x3 average pooling operator, as illustrated in the drawing reference numeral 250, the neural network search device can configure a hardware block including an addition-tree and a final multiplier corresponding to nine activation values ​​(I0 to I8), and can supply a 3x3 pooling window in a row to the hardware block, so that the hardware block can be designed so that the utilization is always 100%.

[0091] In addition, as illustrated in the drawing reference numeral 250, if the operator is a 3x3 max pooling operator, the neural network structure search device can configure a hardware block including 9 comparator-trees corresponding to 9 activation values ​​(I0 to I8), and supply 3x3 pooling windows in a row to the hardware, so that the hardware block can be designed so that the utilization is always 100%.

[0092] Meanwhile, in FIGS. 2A to 2F, for convenience of explanation, the plurality of operators are exemplified as a 3x3 standard convolution operator, a 2x2 standard convolution operator, a 3x3 average pooling operator, and a 3x3 maximum pooling operator, but the plurality of operators according to an embodiment are not limited thereto and may include an NxN (where N is a positive integer) standard convolution operator, an NxN average pooling operator, and an NxN maximum pooling operator.

[0093] Specifically, the neural network search engine is a NxN standard convolutional operator, where N 2 A hardware block including a multiplier and an addition tree that adds all the outputs of the multipliers can be constructed, and NxN convolution windows can be spread out in a row and fed to the hardware block, so that the hardware block can be designed such that the utilization is always 100%.

[0094] Additionally, the neural network search engine is an NxN average pooling operator, where N 2We construct a hardware block that includes an active value adder-tree and a final multiplier, and feed NxN pooling windows in a row to the hardware block (1 / N to the multiplier). 2 ), and this allows the hardware blocks to be designed so that utilization is always 100%.

[0095] Additionally, the neural network search engine is an NxN max pooling operator, where N 2 A hardware block containing a comparator tree can be constructed, and NxN pooling windows can be spread out in a row and supplied to the hardware block, so that the hardware block can be designed such that the utilization is always 100%.

[0096]

[0097] Figure 3 is a drawing illustrating a neural structure search method according to an embodiment.

[0098] In other words, FIG. 3 may be a drawing explaining an operation method of a neural structure exploration device according to an embodiment described through FIGS. 1 to 2f.

[0099] In step 310, the neural structure search method can configure a plurality of operator templates by matching hardware blocks corresponding to a plurality of operators in the search space configuration section, and configure a search space based on the plurality of operator templates.

[0100] In other words, the neural architecture search method can configure multiple operator templates by matching each hardware block having an optimal structure corresponding to multiple operators in advance in the search space configuration section.

[0101] For example, the plurality of operators may include at least one of a skip connection operator, an average pooling operator, a max pooling operator, a standard convolution operator, and a depthwise convolution operator.

[0102] According to one side, in step 310, the neural structure search method can configure multiple operator templates by designing hardware blocks corresponding to multiple operators in the search space configuration section to have 100% utilization.

[0103] In step 320, the neural structure search method can derive a target neural network (i.e., a deep learning model) through a neural search process based on the search space in the neural structure search section.

[0104] According to one side, in step 320, the neural structure search method can derive a target neural network through a neural search process based on cell-based search and gradient descent in the neural structure search section, thereby reducing search costs and solving optimization problems.

[0105] For example, in step 320, the neural structure search method can derive a target neural network through a neural search process consisting of a cell extraction process, an operator weight setting process, a weight update process, an operator selection process, and an operator evaluation process in the neural structure search section.

[0106] Specifically, in step 320, the neural structure search method can extract a plurality of cells, each of which includes a plurality of nodes constituting a target neural network and a target operator template that is composed of at least one operator template among a plurality of operator templates and that connects the plurality of nodes to each other, in the neural structure search unit.

[0107] For example, in step 320, the neural structure search method can extract multiple cells including normal cells and reduction cells in the neural structure search section.

[0108] Next, in step 320, the neural structure search method can set the weight of each of at least one operator templates in the neural structure search section.

[0109] In other words, in step 320, the neural structure search method can set a weight for each candidate operator template (i.e., at least one operator template among multiple operator templates provided in the search space) for the target operator template in the neural structure search unit.

[0110] Next, in step 320, the neural structure search method derives a candidate neural network corresponding to the target neural network as a result of a learning process (i.e., a neural network learning process) that updates weights in the neural structure search unit, and derives the target neural network based on an evaluation process for the candidate neural network.

[0111] For example, in step 320, the neural structure search method can derive a candidate neural network by applying at least one fully connected layer and a softmax activation function to multiple cells in the neural structure search unit.

[0112] Additionally, in step 320, the neural structure search method can receive learning data corresponding to the target neural network in the neural structure search unit and update the weights through a learning process based on the received learning data.

[0113] Additionally, in step 320, the neural architecture search method can derive a target neural network based on an evaluation process of applying a candidate neural network to an evaluation set in the neural architecture search unit, wherein the evaluation set can include a means for evaluating model accuracy and hardware cost for the candidate neural network.

[0114] For a more specific example, in step 320, the neural architecture search method can derive a target neural network in response to a loss function based on cross-entropy loss and hardware cost in the neural architecture search unit.

[0115]

[0116] Ultimately, by using the present invention, it is possible to first find the optimal hardware structure for each operation and define operator-hardware pairs that pair each operation with the optimal hardware block as new basic elements to construct a search space.

[0117] In addition, by using the present invention, the size of the search space and the time and number of resources required for neural network search can be minimized based on operator-hardware pairs.

[0118]

[0119] Although the embodiments described above have been described with limited drawings, those skilled in the art will recognize that various modifications and variations are possible based on the above description. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described devices, structures, apparatuses, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0120] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. A search space configuration unit that configures a plurality of operator templates by matching hardware blocks corresponding to a plurality of operators, and configures a search space based on the plurality of operator templates; and A neural structure search unit that derives a target neural network through a neural search process based on the above search space. A neural structure exploration device including:

2. In paragraph 1, The above multiple operators are, Containing at least one of a skip connection operator, an average pooling operator, a max pooling operator, a standard convolution operator, and a depthwise convolution operator. Neural structure explorer.

3. In paragraph 1, The above search space configuration part is, The hardware blocks corresponding to the above multiple operators are designed to have 100% utilization, thereby configuring the above multiple operator templates. Neural structure explorer.

4. In paragraph 1, The above neural structure exploration unit, The target neural network is derived through the neural search process based on cell-based search and gradient descent. Neural structure explorer.

5. In paragraph 1, The above neural structure exploration unit, A learning process in which a plurality of cells each including a target operator template that connects the plurality of nodes and is composed of at least one operator template among the plurality of operator templates and connects the plurality of nodes is extracted, a weight of each of the at least one operator templates is set, and at least one candidate neural network corresponding to the target neural network is derived as a result of the learning process of updating the weight. Neural structure explorer.

6. In paragraph 5, The above neural structure exploration unit, Deriving the candidate neural network by applying at least one fully connected layer and a softmax activation function to the above plurality of cells. Neural structure explorer.

7. In paragraph 5, The above neural structure exploration unit, The target neural network is derived based on the evaluation process for the above candidate neural network. Neural structure explorer.

8. In paragraph 7, The above neural structure exploration unit, The target neural network is derived based on the evaluation process of evaluating the model accuracy and hardware cost for the candidate neural network. Neural structure explorer.

9. In paragraph 7, The above neural structure exploration unit, The target neural network is derived based on the cross-entropy loss and the loss function based on the hardware cost. Neural structure explorer.

10. In the search space configuration section, a step of configuring a plurality of operator templates by matching hardware blocks corresponding to a plurality of operators, and configuring a search space based on the plurality of operator templates; and In the neural structure exploration section, a step of deriving a target neural network through a neural exploration process based on the search space. A method for exploring neural structures including:

Images