Micro neural architecture searching method based on layering-feedback mechanism and TEG guidance

Abstract

The invention relates to the technical field of neural network architecture search, and discloses a micro neural architecture search method based on a hierarchical-feedback mechanism and TEG guidance, which comprises the following steps: a hierarchical search stage: gradually increasing the depth of a search network, carrying out architecture selection in combination with a TEG index, and carrying out neural architecture search by using a differential architecture search method, the searching step comprises a plurality of searching stages, each searching stage corresponds to one searching network, and the number of units of the corresponding searching network is increased along with the increase of the searching stages and gradually approaches the number of units in the evaluation network used in the evaluation step; and step 2, a feedback search stage: an evaluation network is expanded from the search network of the last search stage, the search network and the evaluation network are integrated, and the search network and the evaluation network of the last search stage are jointly trained in a circulating mode. The search efficiency is improved to a great extent, and meanwhile, the high accuracy of the search model is kept.

Description

Technical Field

[0001] The present invention relates to the technical field of neural network architecture search, and in particular to a differentiable neural architecture search method based on a hierarchical-feedback mechanism and TEG guidance. Background Art

[0002] The development of deep convolutional neural networks (DCNNs) has played a vital role in the success of computer vision tasks. However, manually designing new network architectures is not only time-consuming and labor-intensive, but also requires extensive network training experience and is difficult to scale. In recent years, Neural Architecture Search (NAS) has been used to replace manual design and can automatically discover efficient network architectures in a given search space, thereby reducing manpower and cost investment.

[0003] Although NAS has significant advantages in automation, it still faces the problem of large computing time and resource consumption. Most NAS methods mainly rely on validation sets and perform accuracy-based architecture optimization. Therefore, frequent training and evaluation of sampled architectures become the main bottleneck that hinders search efficiency and interpretability. Even with the use of multiple effective heuristic methods for channel approximation or architecture sampling, the training convergence speed of the hypernetwork is still very slow. Approximate proxy inference techniques, such as truncated training and early stopping, can speed up the search process, but usually introduce serious search biases.

[0004] Differentiable Architecture Search (DARTS) has become one of the most popular NAS methods due to its relatively low computational cost and competitive performance. Unlike traditional methods that search on a set of discrete candidate architectures, DARTS relaxes the search space into a continuous space, so that the architecture can be optimized by gradient descent. This efficiency based on gradient optimization reduces the search cost from thousands of GPU-days to just a few GPU-days. According to a recent NAS survey, due to the simplicity and elegance of the DARTS architecture, related research work is very rich. In addition, the application of gradient optimization in continuous search strategies has become an important development trend in the NAS field.

[0005] However, the existing DARTS method requires the search process to be divided into two steps: search and evaluation. The search step uses a shallow network to discover the optimal unit structure, while the evaluation step stacks these units to build a deep network for final evaluation. This approach results in the optimization of the search process being independent of the target evaluation network. PDARTS attempts to alleviate this depth gap by gradually deepening the search network. EnTranNAS, on the other hand, constructs a search network to narrow this gap by combining the evaluation network module with the search network module. In addition, SNAS and GDAS use Gumbel-Softmax and improved straight-through Gumbel-Softmax, respectively, to alleviate the gap caused by discretization, while AutoHAS enhances GDAS with an entropy term to simultaneously search for hyperparameters and architectures. Although these methods optimize the search and evaluation processes to a certain extent, they still separate the two processes, resulting in limited correlation between the performance of the architecture found in the search network and the actual performance of the evaluation network.

[0006] On the other hand, some recent studies have questioned the effectiveness of DARTS. Li and Talwalkar observed that even a simple random search method can find architectures that outperform the original DARTS. Zela et al. and Liang et al. showed that DARTS easily degenerates into networks full of parameter-free operations (such as skip connections), resulting in poor performance of the searched architectures. To alleviate these problems, Yu et al. proposed a cyclic differentiable architecture search method called CDARTS. CDARTS integrates the search and evaluation networks into a unified architecture and jointly trains the two networks in a cyclic manner, but still suffers from the problem of high search and training costs.

[0007] Recently, people have addressed this issue by proposing training-free NAS. It has been found that even at initialization (i.e., without gradient descent), metrics such as sample Jacobian, Neural Tangent Kernel, and "synflow" are highly correlated with the accuracy of the network. This significantly reduces the search cost. However, these works have only verified some highly customized search methods and exploited limited properties of deep networks in an empirical or ad hoc manner. In addition, these non-training metrics still only pursue the final search performance, providing limited benefits for the interpretation and understanding of search trajectories and different search spaces. To address these issues, Chen et al. proposed a unified, visual, and training-free NAS framework called TEG (Trainability, Expressivity, Generalization), which can improve search time while improving the accuracy of the search model.

[0008] Although PDARTS improves the gap between search and evaluation by gradually increasing the network depth, and CDARTS jointly optimizes the search and evaluation networks by introducing a loop feedback mechanism, these methods still have room for improvement in architecture selection and performance optimization. Summary of the invention

[0009] In order to solve the above technical problems, the present invention provides a differentiable neural architecture search method based on a hierarchical-feedback mechanism and TEG guidance, an innovative method combining the advantages of PDARTS and CDARTS, and further introducing the TEG indicator, aiming to optimize the search efficiency and architecture performance. In the first stage, the staged depth increment of PDARTS is adopted, and the architecture selection is optimized in combination with the TEG indicator, while in the second stage, a loop feedback mechanism is introduced, and the TEG indicator is used to further optimize the performance of the final deep network. Through the integration of this hierarchical and feedback mechanism, the method of the present invention greatly improves the search efficiency while maintaining a high accuracy of the search model.

[0010] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0011] A differentiable neural architecture search method based on a hierarchical-feedback mechanism and TEG guidance includes the following steps:

[0012] Step 1, hierarchical search phase: neural architecture search is performed based on the differentiable architecture search method. The differentiable architecture search method includes a search step and an evaluation step. The search step contains multiple search stages. Each search stage corresponds to a search network. As the number of search stages increases, the number of units in the corresponding search network increases and gradually approaches the number of units in the evaluation network used in the evaluation step.

[0013] At the beginning of each search phase, the TEG index of the search network in the current search phase is calculated, and the TEG index is used to guide the optimization of the search network parameters ω in the current search phase. S and architecture weight α,

[0014] Step 2: Feedback search phase:

[0015] The evaluation network is expanded from the search network of the last search stage. The search network and the evaluation network are integrated and jointly trained in a cyclic manner.

[0016] During the joint training process, the corresponding optional operations for each optional edge are gradually reduced until only two operations with the highest weights remain. Finally, the set of operations with the highest recognition success rate is selected from the training set to form the final network required.

[0017] Furthermore, the search step includes three search phases, and the numbers of units in the search network corresponding to the three search phases are 5, 8, and 11 respectively; the number of units in the evaluation network used in the evaluation step is 20.

[0018] Furthermore, in step 1, the TEG indicator is used to guide the optimization of the search network parameter ω in the current search phase. S When α is the weight of the architecture, the corresponding two-layer optimization problem is:

[0019]

[0020] Among them, r TEG is the TEG index of the search network in the current search phase; ω With λ α is a weight parameter used to balance the weight of loss and TEG indicator in the total loss; and Respectively represent the validation loss function and the training loss function, Represents network parameters.

[0021] Furthermore, the search network and the evaluation network are integrated to jointly train the search network and the evaluation network of the last search stage in a cyclic manner, specifically including:

[0022] The architecture search is modeled as a joint optimization problem of the search network and the evaluation network. The objective function of the joint optimization problem is:

[0023]

[0024]

[0025] where ω E and To evaluate the parameters of the network, ω S and is the parameter for searching the network; and Respectively represent the validation loss function and the training loss function, represents the corresponding weight parameter, r TEG is the TEG index of the search network in the current search phase;

[0026] The objective function of the joint optimization problem is optimized in two stages: separate learning and joint learning. In the separate training stage, a weight sharing strategy is used to update ω. E , the structure of the evaluation network is updated according to the architectural weights of the search network; the weights of the evaluation network are initialized by the parameters of the search network;

[0027] In the joint learning phase, the search algorithm updates the architecture weight α by using the feature feedback of the evaluation network through introspective distillation; the objective function of the joint optimization problem is further expressed as:

[0028]

[0029] in, Indicates the use of fixed weights To optimize the architecture weight α in the search network, Represents using a fixed architecture weight α to optimize the weight ω in the evaluation network E , represents the introspective distillation process for transferring knowledge from the evaluation network to the search network, and using the features obtained from the evaluation network as a supervision signal to guide the update of the architecture weights α in the search network.

[0030] Compared with the prior art, the beneficial technical effects of the present invention are:

[0031] The present invention proposes a differentiable neural architecture search framework based on hierarchical-feedback mechanism and TEG guidance, called DARTS-HF-TEG. The method of the present invention is divided into two stages. The first stage is the hierarchical search stage, which adopts a phased network depth increase and combines the TEG indicator to guide the architecture selection; the second stage is the feedback search stage, which introduces a loop feedback mechanism and uses the TEG indicator to further optimize the performance of the final deep network. The experiments and analysis of the present invention on CIFAR, ImageNet and NATS-Bench demonstrate the effectiveness of the method. Specifically, in the DARTS search space, the present invention achieves an average top-1 accuracy of 97.50% on CIFAR10 (only 0.16 GPU-days are required) and a top-1 accuracy of 75.9% on ImageNet (only 0.8 GPU-days are required). BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A schematic diagram of the flow of the differentiable neural architecture search method proposed in the present invention;

[0033] Figure 2 A schematic diagram of the overall framework of the hierarchical search stage of the present invention;

[0034] Figure 3 Schematic diagram of the feedback search phase of the present invention. DETAILED DESCRIPTION

[0035] A preferred embodiment of the present invention is described in detail below with reference to the accompanying drawings.

[0036] 1. Differentiable Architecture Search (DARTS)

[0037] The goal of differentiable architecture search is to search for a cell motif that can be repeatedly stacked to build a convolutional network. A cell is a directed acyclic graph consisting of an ordered sequence of N nodes. The set of N nodes is represented by Each node x i is a potential representation (e.g., a feature map), and each directed edge (i, j) is associated with an operation o (i,j) This operation transfers information from x i Convert to x j .set up represents the operation space, which consists of a set of candidate operations, such as convolution, pooling, skip connection, etc. Each operation represents a function o(·) and is i To make the search space continuous, DARTS relaxes the choice of a specific operation to a normalized exponential function (softmax) of all possible operations:

[0038]

[0039] in, Represents an optional operation, a pair of nodes (x i ,x j ) is a mixed weight with dimensions Here, the parameter α is the encoding of the architecture to be optimized. The intermediate node of a unit is calculated based on all its predecessor nodes as Output node x N-1 It is obtained by connecting all intermediate nodes in the channel dimension in series.

[0040] With the definition of the cell, the search for the optimal architecture becomes solving the following two-level optimization problem:

[0041]

[0042] Among them, α is the architecture weight optimized on the validation data (val), ω S is the search network parameter learned on the training data (train). This formula represents the optimization of the search network parameters and architecture weights in an alternating manner, that is, ω S After obtaining the optimal structure, the discovered neural units are superimposed to construct a new evaluation network and retrained on the target task.

[0043] In DARTS, the optimization of the evaluation network is separated from the architecture search process. Therefore, the size of the architecture parameters is not strongly correlated with the contribution of the operation to the performance of the search network, resulting in the searched architecture being suboptimal for the final evaluation network.

[0044] 2.TEG

[0045] Trainability, Expressivity, and Generalization are three important, different, and complementary properties for characterizing and understanding neural networks. Specifically, trainability is related to the convergence speed during optimization; expressivity is related to the functional complexity of the network; and generalization indicates the error of the model on unknown data.

[0046] The greater the difference in learning speed between different feature modes, the more difficult it is to optimize the network. Therefore, Yu et al. use the empirical condition number of NTK (neural tangent kernel) to represent trainability:

[0047]

[0048] In the formula, the network parameter θ is taken from the open normal initialization (N l is the width of the lth layer), so it is calculated when the network is initialized It is negatively correlated with both the training and test accuracy of the network. Therefore, in the search process, minimize The discovery of architectures with high performance will be encouraged.

[0049] For a given training example and parameters, the number of linear regions R(x train ,θ) can be approximated by the number of unique activation patterns of all ReLU layer combinations in the entire network. Therefore, Yu et al. use the empirical number of linear regions to represent expressiveness:

[0050]

[0051] It is positively correlated with both the training accuracy and the test accuracy of the network. In addition, The correlation with training accuracy is stronger than that with test accuracy, which confirms that Indicates how well the network fits the training data, but not how well it generalizes.

[0052] The generalization is empirically estimated by computing the test MSE error of the network's NTK kernel regression:

[0053]

[0054] NTK represents the last layer of the deep network. Try to regress x through NTK kernel test With x train Associate and give the given training label x train Passed to test data. If the deep neural network predicts becomes independent of the data, then it will not generalize well and the MSE will become large.

[0055] Combination and MSE, the TEG reward index can be expressed as r TEG =r k +r R +r MSE , with r k For example:

[0056]

[0057]

[0058] in, is the estimated trainability of the architecture sampled at step t. R and r MSE can be calculated in the same way.

[0059] 3. Differentiable Neural Architecture Search Method in the Present Invention

[0060] The proposed method of differentiable neural architecture search based on hierarchical-feedback mechanism and TEG guidance consists of two parts. The first stage is the hierarchical search stage, which uses the network depth to gradually increase and combines the TEG index to select the architecture; the second stage is the feedback search stage, which introduces the loop feedback mechanism and uses the TEG index to further optimize the performance of the final deep network. Figure 1 As shown, the present invention improves the search efficiency to a great extent through the integration of the stratification and feedback mechanism, while maintaining a high accuracy of the search model.

[0061] 3.1 Hierarchical Search Phase

[0062] In DARTS, the architecture search is performed on networks with 8 units, and the discovered architectures are evaluated on networks with 20 units. However, there are large differences in the behavior of shallow and deep networks, which means that the architectures chosen during the search are not necessarily optimal.

[0063] The present invention also adopts the strategy of gradually increasing the network depth during the search process, so that at the end of the search, the depth is as close as possible to the number of units in the evaluation network. The difference is that because the feedback mechanism in the feedback search stage can further bridge the depth gap, in order to save computing costs, the present invention chooses to slow down the speed of increasing the network depth in the hierarchical search stage, such as Figure 2 At the same time, the present invention does not immediately search when the final stage network is obtained, but performs a cyclic feedback search on the final stage search network and the evaluation network. For specific ideas, refer to the specific introduction of the feedback search stage. In addition, at the beginning of each search round (epoch), the present invention calculates the TEG index of the current search network, and uses the TEG index to guide the optimization of the parameters ω of the search network of the current search round (epoch). S and architecture weight α, the two-level optimization problem can be expressed as:

[0064]

[0065] Among them, r TEG is the TEG index of the network in the current epoch, λ ω With λ α is a weight parameter used to balance the weight of loss and TEG index in the total loss. Compared with the PDARTS method, the method of the present invention saves the time required for calculation, and the gap between networks of different depths is smaller.

[0066] Figure 2 Figure 2. Overall framework of the hierarchical search phase. The depth of the search network increases from 5 in the initial phase to 8 and 11 in the middle and final phases, while the number of candidate operations (represented by connections of different colors) is reduced accordingly. The operations with the lowest scores in the previous phase are deleted.

[0067] 3.2 Feedback Search Phase

[0068] In the feedback search phase, the evaluation network is expanded from the search network in the final phase, integrating the search network and the evaluation network into a unified architecture, and jointly training the two networks in a cyclic manner. The architecture search is modeled as a joint optimization problem of the two networks:

[0069]

[0070] where ω ETo evaluate the weight of the network, the present invention also optimizes the objective function in two stages: individual learning and joint learning. For the evaluation network, its internal unit structure is generated by discretizing the learned weight α. The top-k (k=2) operations of each node in the cell are retained by thresholding the learned value in α. In the individual training stage, a weight sharing strategy is used to update ω. E , the architecture of the evaluation network is updated according to the architecture weights of the search network. The weights of the evaluation network are initialized with the parameters of the search network.

[0071] In the joint learning phase, the search algorithm updates the architecture weights α using the feature feedback from the evaluation network through introspective distillation. More specifically, the joint optimization of the two networks is formulated as:

[0072]

[0073] in, With fixed weights To optimize the search network architecture weight α, The weight ω of the evaluation network is optimized with a fixed architecture weight α E . represents the introspective distillation process, which transfers knowledge from the evaluation network to the search network, The features obtained from the evaluation network are used as supervisory signals to guide the update of the architecture hyperparameter α in the search network.

[0074] Feedback search phase Figure 3 As shown in Figure 1, it consists of two branches: the search network and the evaluation network. In order to transfer information, a connection is established between the two branches. Specifically, there is an architecture transfer path that transfers the discovered cell graph from the search branch to the evaluation branch, as shown in Figure 1. Figure 3 As shown by the top bold arrow in . Based on previous work, the present invention retains the top-k (k=2) most effective operations from all candidate operations of all previous nodes. On the other hand, there is also a feedback distillation path that passes the feature feedback of the evaluation branch to the search branch, as shown in Figure 3 As shown by the solid arrows at the bottom, these feedbacks serve as supervisory signals to help the search network find better cell structures.

[0075] Specifically, we use the multi-level features of the evaluation network as feedback signals because they are representative in capturing the semantics of the image. Figure 3The lateral embedding connection shown in the figure combines low-resolution, semantically strong features with high-resolution, semantically weak features. Features are derived from the output of each stage and then the corresponding feature logits are generated through the embedding module. The function of the embedding module is to project the dense feature map into a low-dimensional subspace. The logarithm of the obtained evaluation network is passed through the soft cross entropy layer as a supervision signal for the search network.

[0076] Figure 3 Schematic diagram of the feedback search stage, which includes two networks, the search network (left) and the evaluation network (right). The Embedding module maps each stage feature to a one-dimensional vector. Figure 2 and Figure 3 middle

[0077] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention, and any reference numerals in the claims should not be regarded as limiting the claims involved.

[0078] In addition, it should be understood that although the present specification is described in terms of implementation modes, not every implementation mode contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A differentiable neural architecture search method based on hierarchical-feedback mechanism and TEG guidance, characterized in that: The following steps are involved: Step 1, hierarchical search phase: neural architecture search is performed based on the differentiable architecture search method. The differentiable architecture search method includes a search step and an evaluation step. The search step contains multiple search stages. Each search stage corresponds to a search network. As the number of search stages increases, the number of units in the corresponding search network increases and gradually approaches the number of units in the evaluation network used in the evaluation step. At the beginning of each search phase, the TEG index of the search network in the current search phase is calculated, and the TEG index is used to guide the optimization of the search network parameters ω in the current search phase. S and architecture weight α, Step 2: Feedback search phase: The evaluation network is expanded from the search network of the last search stage. The search network and the evaluation network are integrated and jointly trained in a cyclic manner. During the joint training process, the corresponding optional operations for each optional edge are gradually reduced until only two operations with the highest weights remain. Finally, the set of operations with the highest recognition success rate is selected from the training set to form the final network required.

2. The method for searching differentiable neural architectures based on hierarchical feedback mechanism and TEG guidance according to claim 1, characterized in that: The search step includes three search phases, and the number of units in the search network corresponding to the three search phases is 5, 8, and 11 respectively; the number of units in the evaluation network used in the evaluation step is 20.

3. The method for searching differentiable neural architectures based on hierarchical feedback mechanism and TEG guidance according to claim 1, characterized in that: In step 1, the TEG indicator is used to guide the optimization of the search network parameters ω in the current search phase. S When α is the weight of the architecture, the corresponding two-layer optimization problem is: Among them, r TEG is the TEG index of the search network in the current search phase; ω With λ α is a weight parameter used to balance the weight of loss and TEG indicator in the total loss; and Respectively represent the validation loss function and the training loss function, Represents network parameters.

4. The method for searching differentiable neural architectures based on hierarchical-feedback mechanism and TEG guidance according to claim 1, characterized in that: The search network and the evaluation network are integrated to jointly train the search network and the evaluation network of the last search phase in a cyclic manner, specifically including: The architecture search is modeled as a joint optimization problem of the search network and the evaluation network. The objective function of the joint optimization problem is: where ω E and To evaluate the parameters of the network, ω S and is the parameter for searching the network; and Respectively represent the validation loss function and the training loss function, represents the corresponding weight parameter, r TEG is the TEG index of the search network in the current search phase; The objective function of the joint optimization problem is optimized in two stages: separate learning and joint learning. In the separate training stage, a weight sharing strategy is used to update ω. E , the structure of the evaluation network is updated according to the architectural weights of the search network; the weights of the evaluation network are initialized by the parameters of the search network; In the joint learning phase, the search algorithm updates the architecture weight α by using the feature feedback of the evaluation network through introspective distillation; the objective function of the joint optimization problem is further expressed as: in, Indicates the use of fixed weights To optimize the architecture weight α in the search network, Represents using a fixed architecture weight α to optimize the weight ω in the evaluation network E , represents the introspective distillation process for transferring knowledge from the evaluation network to the search network, and uses the features obtained from the evaluation network as a supervision signal to guide the update of the architecture weights α in the search network.

Images