Evolutionary neural architecture search method and electronic device

Abstract

The present application provides an evolutionary neural architecture search method and an electronic device. The evolutionary neural architecture search method may include the following steps: (a) performing one or more evolutionary operations on an initial population of multiple neural architectures to generate multiple offspring neural architectures; (b) evaluating the performance of each offspring neural architecture to obtain an evaluation value of at least one performance indicator of the offspring neural architecture; (c) adjusting the evaluation value of the offspring neural architecture based on at least one restriction on the evaluation value; (d) selecting at least one of the offspring neural architectures as a new population of neural architectures; and (e) outputting the new population of neural architectures as the final population of neural architectures when the stopping criteria are reached, or (f) treating the new population of neural architectures as the initial population of neural architectures to iterate steps (a) to (d). The evolutionary neural architecture search method and electronic device provided by the present application avoid focusing on invalid optimization directions and quickly obtain the target neural architecture.

Description

Technical Field

[0001] The present invention relates to neural architecture search (NAS), and more particularly to a method for introducing model requirements or preferences into the NAS search phase to produce a better quality Pareto front. Background Art

[0002] The background description provided in this application is intended to generally present the background of the content of this application. The inventors of the inventions described in this section, as well as various aspects of the description that do not otherwise qualify as prior art at the time of filing the application, are not explicitly or implicitly admitted to be prior art of this application.

[0003] In recent years, artificial intelligence (AI) models, such as machine learning (ML) models, deep learning (DL) models, and neural network (NN) models, have become increasingly popular. As the scope of use of AI models expands, the number of different neural architectures that implement AI models has also increased. Considering the high cost of manual implementation, neural architecture search (NAS), which can automatically search for architecture designs, has become a popular method. Summary of the invention

[0004] The present application provides an evolutionary neural architecture search (ENAS) method. The ENAS method includes the following steps: (a) performing one or more evolution operations on an initial population of multiple neural architectures to generate multiple child neural architectures; (b) evaluating the performance of each of the child neural architectures to obtain an evaluation value of at least one performance indicator of the child neural architectures; (c) adjusting the evaluation values ​​of the child neural architectures based on at least one restriction on the evaluation values; (d) selecting at least one of the child neural architectures as a new population of the neural architectures using a selection strategy; (e) when a stopping criterion is reached, outputting the new population of the neural architectures as a final population of the neural architectures; and (f) when the stopping criterion has not been reached, using the new population of the neural architectures as the initial population of the neural architectures, and iterating the steps (a) to (d).

[0005] In one embodiment, adjusting the evaluation values ​​of the neural architectures includes at least one of the following operations: executing a clip algorithm to apply a sufficient boundary to the evaluation values ​​of the child neural architectures; and executing an extinct algorithm to apply a necessary boundary to the evaluation values ​​of the child neural architectures. For example, at least one of the sufficient boundary and the necessary boundary remains fixed at each iteration of the steps (a) to (d). In another example, at least one of the sufficient boundary and the necessary boundary changes at least every two consecutive iterations of the steps (a) to (d). In one embodiment, at least one of the sufficient boundary and the necessary boundary gradually increases as the steps (a) to (d) are iterated. In one example, the necessary boundary gradually increases as the steps (a) to (d) are iterated. In another example, the sufficient boundary gradually increases as the steps (a) to (d) are iterated.

[0006] In one embodiment, the evolution operations include at least one of crossover and mutation. In another embodiment, the selection strategy includes one of non-dominated sorting, elitism, worst-case drop, roulette wheel selection, and tournament selection.

[0007] The present application further provides an electronic device. For example, the electronic device includes a circuit configured to execute an evolutionary neural architecture search (ENAS) method, the ENAS method including the following steps: (a) performing one or more evolution operations on an initial population of multiple neural architectures to generate multiple child neural architectures; (b) evaluating the performance of each of the child neural architectures to obtain an evaluation value of at least one performance indicator of the child neural architectures; (c) adjusting the evaluation values ​​of the child neural architectures based on at least one restriction on the evaluation values; (d) selecting at least one of the child neural architectures as a new population of the neural architectures using a selection strategy; (e) when a stopping criterion is reached, outputting the new population of the neural architectures as a final population of the neural architectures; and (f) when the stopping criterion has not been reached, using the new population of the neural architectures as the initial population of the neural architectures, and iterating the steps (a) to (d).

[0008] The present application further provides a non-transitory computer-readable storage medium. For example, the non-transitory computer-readable storage medium stores a plurality of instructions. When a processor executes the instructions, the processor executes an evolutionary neural architecture search (ENAS) method. The ENAS method includes the following steps: (a) performing one or more evolution operations on an initial population of a plurality of neural architectures to generate a plurality of offspring neural architectures; (b) evaluating the performance of each of the offspring neural architectures to obtain an evaluation value of at least one performance indicator of the offspring neural architectures; (c) adjusting the evaluation values ​​of the offspring neural architectures based on at least one restriction on the evaluation values; (d) selecting at least one of the offspring neural architectures as a new population of the neural architectures using a selection strategy; (e) when a stopping criterion is reached, outputting the new population of the neural architectures as a final population of the neural architectures; and (f) when the stopping criterion has not been reached, using the new population of the neural architectures as the initial population of the neural architectures, and iterating the steps (a) to (d). BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Various embodiments presented as examples in this application will be described in detail with reference to the accompanying drawings, in which like reference numerals represent like elements, and in which:

[0010] Figure 1 A flow chart of an ENAS method.

[0011] Figure 2A Showing the NAS results for two performance metrics, Figure 2B NAS results with respect to three performance metrics are shown.

[0012] Figure 3 A truncation algorithm for adjusting the evaluation value of a candidate neural architecture in certain embodiments of the present application is described.

[0013] Figure 4 An extinction algorithm for adjusting evaluation values ​​of candidate neural architectures in certain embodiments of the present application is described.

[0014] Figure 5 A flow chart describing an ENAS method according to certain embodiments of the present application.

[0015] Figure 6 An example of applying and simulating an ENAS method in certain embodiments of the present application is described. The ENAS method introduces necessary boundaries and / or sufficient boundaries to apply sufficient boundaries and necessary boundaries to candidate neural architectures, respectively.

[0016] Figure 7 A flow chart describing another ENAS method according to certain embodiments of the present application.

[0017] Figure 8A flow chart describing yet another ENAS method according to certain embodiments of the present application.

[0018] Fig. 9 is a functional block diagram of a device according to some embodiments of the present application, wherein the device is configured to perform the ENAS method.

[0019] Description of reference numerals:

[0020] 100:ENAS method

[0021] 110: Initial space (or stage)

[0022] 120: Search space (or stage)

[0023] 121: Evolution step 122: Evaluation step 123: Selection step 124: Stopping criterion step 200A, 200B: NAS

[0024] A, B: Model

[0025] T: Sufficient Boundary / Necessary Boundary

[0026] 500:ENAS method

[0027] 510: Initial space (or stage)

[0028] 520: Search space (or phase)

[0029] 525: Extinction step 526: Truncation step 700: ENAS method

[0030] 710: Initial space (or stage)

[0031] 720: Search Space (or Phase)

[0032] 800:ENAS Method

[0033] 810: Initial space (or stage)

[0034] 820: Search space (or phase)

[0035] 900: Device

[0036] 910: Storage circuit

[0037] 920: Processing circuit DETAILED DESCRIPTION

[0038] In order to make the purpose, technical scheme and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention. It should be noted that, in the absence of conflict, the embodiments in this application and the features in the embodiments can be combined with each other arbitrarily.

[0039] Recent advances in deep learning (DL) (e.g., deep neural networks (DNNs)) have increased the demand for neural architecture design. DNN models (e.g., convolutional neural networks (CNNs)) can be used to solve various computer vision tasks, such as image classification, semantic segmentation, object detection, etc. These efficient CNN models are manually designed by human experts. Due to the high cost of manual design, neural architecture search (NAS), which can automatically search for neural architectures, has become a popular method. NAS aims to automatically design neural architectures with good performance using limited computer resources without requiring too much human involvement.

[0040] Generally speaking, NAS algorithms (or methods) can be divided into three categories based on the optimizer (or search strategy) used: reinforcement learning (RL), gradient, and evolutionary computation (EC). RL-based NAS algorithms use, for example, recurrent neural networks (RNN) to sample representations of neural architectures, and then use reinforcement learning to continuously adjust the neural architectures. RL-based NAS algorithms may require up to thousands of graphics processing units (GPUs) to execute for several days. Gradient-based NAS algorithms continuously relax the previous discrete search space and allow the use of gradient descent strategies to efficiently search for neural architectures. Compared to RL-based NAS algorithms, gradient-based NAS algorithms significantly improve search efficiency, but due to inappropriate relationships used to adapt to gradient-based optimization, they often result in finding poor NAS neural architectures. The EC-based NAS (ENAS) algorithm uses EC technology to solve the NAS problem. The EC technology simulates the evolution of species or the behavior of populations in nature, such as crossover and mutation, to search for the optimal neural architecture of the DNN model.

[0041] Figure 1 A flow chart describing an ENAS method 100, which performs a population updating process. The ENAS method 100 includes an initial space (or stage) 110 and a search space (or stage) 120. In the initial space 110, a population is predefined or sampled, and each individual in the population can be used as a candidate model (i.e., a DNN model) for ENAS. For example, the population in the initial space 110 can be initialized from an arbitrary (trivial) initial condition, or randomly initialized in the encoding space. Optionally, the fitness of the individuals can be evaluated based on some evaluation criteria. Finally, after evaluating each individual in the population, an initial population of the search space 120 is generated, in which each individual is rated as exceeding the evaluation criteria and is defined as a neural architecture that may be discovered by the NAS method. The search space 120 can also be reduced by combining the typical properties of the neural architecture suitable for the task with its related prior knowledge (e.g., multiple initializations) to form a well-designed search space.

[0042] In the search space 120, at least one DNN model may be sampled from the initial population in each evolution step and used as a parent population to generate offspring (offspring) through the evolution operation (e.g., crossover, mutation, etc.) performed in step 121. For example, crossover may include exchanging certain hyperparameters (e.g., the number of channels) of two layers in the DNN model. In another example, mutation may include adding or deleting layers, changing hyperparameters (e.g., the number of channels) of layers, adding skip connections, and changing hyperparameters for DNN model training.

[0043] The ENAS method 100 aims to find a neural architecture that maximizes certain performance indicators, such as accuracy (e.g., based on peak signal-to-noise ratio (PSNR)), latency, power consumption, memory requirements, operating speed, operating cycle, etc. Therefore, in the search space 120, step 122 evaluates the generated offspring to determine their performance. For example, the offspring (e.g., DNN model) can be trained using a training set (training data), and their performance can be evaluated using a validation set (validation data).

[0044] Next, offspring are selected in step 123. For example, the selection strategy used in step 123 may include elitism (retaining individuals with higher fitness), worst-case dropout (removing individuals with lower fitness from the population), roulette wheel selection (giving each individual a chance to survive or be dropped, regardless of whether it is the best, based on its fitness in the population), tournament selection (selecting the best from a sample of equally likely individuals), etc.

[0045] Finally, in step 124, it is determined whether the stopping criterion of the population updating process is reached. For example, if the generation of the generated offspring has not reached the generation limit, steps 121 to 123 may be performed again in sequence; otherwise, the population updating process of the ENAS method 100 ends and the final population is generated.

[0046] Traditionally, NAS is mathematically formulated as a single-objective optimization process (e.g., accuracy), whose goal is to find a neural architecture with the minimum validation error. In order to deploy AI applications on hardware, AI models need to be evaluated not only for their accuracy, but also for latency, power consumption, and other hardware metrics, which makes NAS a multi-objective optimization process. Multi-objective NAS aims not only to minimize validation error, but also to optimize multiple objectives (e.g., accuracy, latency, power consumption, and memory requirements) at the same time. No neural architecture can minimize all objectives at the same time, because these objectives usually compete with each other. Instead, there are multiple Pareto optimal neural architectures close to the true (unknown) Pareto front, which reflect different trade-offs between objectives. For example, a cost function can be customized to convert a multi-objective problem into a single-objective problem. However, such a cost function exhibits poor versatility, and each application may require a dedicated cost function. In addition, when the number of objectives increases, the searched DNN models may become more sparse. Therefore, multi-objective NAS may lead to extremely diverse solutions and result in only a small number of models that meet the pre-defined requirements. Figure 2A , Figure 2BAs shown in the figure, the fourth generation candidate DNN models are more sparse in NAS200B with three objectives (such as PSNR, power consumption and delay) than in NAS200A with two objectives (such as PSNR and delay), and model A and model B have great differences even though they have similar cost function values. Therefore, a large amount of search resources may be wasted on unqualified models, resulting in insufficient number and quality of qualified candidate models.

[0047] The present application provides a method for introducing model requirements or preferences into the NAS search phase to produce a better quality Pareto frontier. Therefore, the NAS search target space (such as search space 120) can be limited and only focus on potentially qualified candidate models. In one embodiment, necessary boundaries (must-bound) and sufficient boundaries (enough-bound) can be introduced into multi-objective NAS. For example, if a model is worse than the necessary boundary, the model is considered unqualified and should not appear in the final candidate model list. In another example, if a model is better than the sufficient boundary, it will not be given additional credit, and it has a predefined evaluation value relative to the sufficient boundary.

[0048] The present application introduces a truncation algorithm (Clip(x)) and an extinction algorithm (Extinct(x)) to apply sufficient boundaries and necessary boundaries to the evolution model, respectively. For example, the truncation algorithm can be adjusted and determine the evaluation value of the model according to the truncation condition. In one embodiment, if x>T (i.e., sufficient boundary, minimum requirement), then Clip(x)=x; otherwise, if x≤T, then Clip(x)=T. For example, if x represents the evaluation value of the delay, then during the selection step (e.g., the selection of step 123 (e.g., Figure 1 As shown in ( ), no additional reward will be given to models that are better than the sufficient margin (T) (e.g., the delay is less than the sufficient margin). Therefore, NAS can avoid focusing on invalid optimization directions. For example, Figure 3 As shown, the sufficient boundary (T) of the delay index of any candidate model is 150, and the evaluation value of model A on delay is truncated to 150 according to Clipatency (x) = 150, even if its delay is 100, which is smaller than the sufficient boundary (T) of 150, because 100 ≤ 150. In another embodiment, if x ≤ T (i.e., sufficient boundary, maximum limit), then Clip (x) = x; otherwise if x> T, then Clip (x) = T, for example, if x represents the evaluation value of PSNR, then during the selection step (e.g., the selection of step 123 (e.g., Figure 1 As shown in (), no additional reward will be given to models that are better than the sufficient margin (e.g., PSNR is greater than the sufficient margin). Similarly, NAS can also avoid focusing on invalid optimization directions. For example, Figure 3 As shown, the sufficient boundary (T) of the PSNR index of any candidate model is 33.5, and the evaluation value of Model B with respect to PSNR is truncated to 33.5 according to ClipPSNR(x) = 33.5, even though its PSNR is 34.0, which is greater than 33.5 of the sufficient boundary (T), because 34 > 33.5.

[0049] For example, the extinction algorithm can adjust and determine the evaluation value of the model based on the extinction condition. In one embodiment, if x < T (i.e., the necessary boundary, the minimum requirement), then Extinct(x) = 0; otherwise, if x ≥ T, then Extinct(x) = x. For example, if x represents the evaluation value of PSNR, the evaluation value of a model worse than the necessary boundary (T) will be reduced to zero (or a low enough value to clearly determine whether it is less than the threshold), and this model will be discarded from the population during the selection step (such as the selection in step 123 (as Figure 1 shown)). Therefore, NAS can avoid focusing on ineffective optimization directions. For example, as Figure 4 shown, the necessary boundary (T) of the PSNR index of any candidate model is 32.5, and the evaluation value of Model A for PSNR is reduced to zero according to ExtinctPSNR(x) = 0, even though the PSNR is non-zero and 32, because 32.0 < 32.5. Model A will be discarded from the population in the selection step because the evaluation value of Model A does not meet the minimum requirement of the necessary boundary. In another embodiment, if x ≤ T, then Extinct(x) = x (i.e., the necessary boundary, the maximum limit); otherwise, if x > T, then Extinct(x) = ∞. For example, if x represents the evaluation value of latency, the evaluation value of a model worse than the necessary boundary (T) will be increased to infinity, and this model will be discarded from the population during the selection step (such as the selection in step 123 (as Figure 1 shown)). Therefore, NAS can avoid focusing on ineffective optimization directions. For example, also as Figure 4 shown, the necessary boundary (T) of the latency metric of any candidate model is 250, and the evaluation value of Model B for latency is increased to infinity according to Extinctlatency(x) = ∞ (or a very large value), even though the latency of 300 is not a too large value, because 300 > 250. Model B will also be discarded from the population in the selection step because the evaluation value of Model B exceeds the maximum limit of the necessary boundary.

[0050] Figure 5Flowchart of the ENAS method 500 describing certain embodiments of the present application. The ENAS method 500 performs a population update process. In one embodiment, the ENAS method 500 may include an initial space (or stage) 510 and a search space (or stage) 520. The ENAS method 500 introduces model requirements or preferences into the NAS search stage to generate a Pareto front of better quality. Therefore, the NAS search target space (e.g., the search space 520) can be restricted (e.g., necessary boundaries and sufficient boundaries), and only potential qualified candidate models are focused on. In one embodiment, the initial space 510 may include the initial space 110. In another embodiment, the search space 520 of the ENAS method 500 is different from the search space 120 of the ENAS method 100 in that the search space 520 further includes an extinction step 525 and a truncation step 526 in addition to the evolution step 121, the evaluation step 122, the selection step 123, and the stopping criterion step 124. In various embodiments, some of the steps of the ENAS method 500 shown may be executed simultaneously or in an order different from the order shown, may be replaced by other method steps, or may be omitted. Other method steps may also be executed as needed.

[0051] In the ENAS method 500, the initial population generated in the initial space 510 is input into the search space 520. In step 121, evolutionary operations (e.g., crossover, mutation, etc.) are performed on the parent population to generate offspring, and the offspring generated in step 121 are evaluated to determine their performance (e.g., the evaluation value of the model). Then, the ENAS method 500 can proceed to step 525.

[0052] In step 525, an extinction algorithm (Extinct(x)) is used to adjust and determine at least one evaluation value of the model based on the extinction condition. For example, if x < T (i.e., the necessary boundary), then Extinct(x) = 0; otherwise, if x ≥ T, then Extinct(x) = x. For example, if x represents the evaluation value of PSNR or running speed, the evaluation value of a model worse than the necessary boundary (T) will be reduced to zero (or a small enough value), and the model will be discarded from the population during the selection step (e.g., the selection in step 123 (as Figure 1 shown)). In another example, if x ≤ T (i.e., the necessary boundary), then Extinct(x) = x; otherwise, if x > T, then Extinct(x) = ∞ (or a very large value). For example, if x represents the evaluation value of latency, power consumption, memory requirement, or running cycle, the evaluation value of a model worse than the necessary boundary (T) will be increased to infinity, and the model will be discarded from the population during the selection step (e.g., the selection in step 123 (as Figure 1 shown)). Then, the ENAS method 500 can proceed to step 526.

[0053] At step 526, a truncation algorithm (Clip(x)) is used to determine at least one evaluation value of the model based on the truncation condition. For example, if x>T (i.e., sufficient margin), then Clip(x)=x; otherwise, if x≤T, then Clip(x)=T. For example, if x represents an evaluation value of latency, power consumption, memory requirements, or operating cycles, then during the selection step (e.g., the selection of step 123 (e.g., Figure 1 As shown)) will not give additional rewards to models that are better than the sufficient margin (T) (e.g., the delay, power consumption, memory requirement, or operation cycle is less than the sufficient margin). In another example, if x≤T (i.e., the sufficient margin), then Clip(x)=x; otherwise if x>T, then Clip(x)=T, for example, if x represents an evaluation value of PSNR or operation speed, then during the selection step (e.g., the selection of step 123 (e.g., Figure 1 As shown in FIG. 1 , no additional reward is given to a model that is better than the sufficient margin (T) (e.g., PSNR or operation period is greater than the sufficient margin). Next, the ENAS method 500 may proceed to step 123.

[0054] In step 123, a child with at least one evaluation value that is truncated (i.e., not greater than or less than the sufficient boundary, depending on the truncation condition) is selected, and none of the selected children has an evaluation value greater than or less than the necessary boundary, depending on the extinction condition. In one embodiment, the selection strategy used in step 123 may include elitism, worst-case discard, roulette wheel selection, tournament selection, non-dominated sorting, etc.

[0055] Next, the ENAS method 500 may proceed to step 124, where it is determined whether the stopping criterion of the population updating process is reached. For example, if the generation of the generated offspring has not reached the generation limit, steps 121 to 123 may be performed again in sequence; otherwise, the population updating process of the ENAS method 500 ends, and the final population is generated.

[0056] Figure 6 An example of applying and simulating an ENAS method in certain embodiments of the present application is described, and the ENAS method introduces necessary margins and / or sufficient margins, such as ENAS method 500. For example, the ENAS method is used for a super-resolution (SR) model to maximize the PSNR (necessary margin) and minimize the delay and power of the SR model (respective necessary margins and sufficient margins). In one embodiment, the necessary margin (e.g., 34.25) is used for PSNR, and the sufficient margin is used for delay and power. Figure 6As shown in the figure, the unrestricted benchmark only found 5 candidate models, and its hypervolume (HV) (a measure of the quality of the Pareto frontier) was only 0.0167. In contrast, the necessary boundary of PSNR found up to 25 candidate models, and its HV reached 0.0226, which is 35.33% larger than 0.0167. In addition, the necessary boundary of PSNR and the sufficient and necessary boundaries of delay and power each found up to 35 candidate models, and their HV reached 0.0220, which is 31.74% larger than 0.0167.

[0057] Figure 7 A flowchart of an ENAS method 700 in some embodiments of the present application is described, and the ENAS method 700 performs a population update process. In one embodiment, the ENAS method 700 may include an initial space (or stage) 710 and a search space (or stage) 720. The ENAS method 700 introduces model requirements or preferences into the NAS search stage to generate a better quality Pareto front. In one embodiment, the initial space 710 may include the initial space 110. In another embodiment, the search space 720 of the ENAS method 700 is different from the search space 520 of the ENAS method 500 in that the search space 720 performs steps 525 and 526 in the opposite order of the search space 520. For example, in the search space 720, a truncation algorithm (Clip(x)) may be used to adjust and determine at least one evaluation value of the model based on the truncation condition of step 526, and then an extinction algorithm (Extinct(x)) may be used to adjust and determine at least one of the remaining evaluation values ​​of the model based on the extinction condition of step 525. In various embodiments, some steps of the illustrated ENAS method 700 may be performed simultaneously or in a different order than shown, may be replaced by other method steps, or may be deleted. Other method steps may also be performed as desired.

[0058] Figure 8A flowchart of an ENAS method 800 in some embodiments of the present application is described, and the ENAS method 800 performs a population update process. In one embodiment, the ENAS method 800 may include an initial space (or stage) 810 and a search space (or stage) 820. The ENAS method 800 introduces model requirements or preferences into the NAS search stage to produce a better quality Pareto front. In one embodiment, the initial space 810 may include the initial space 110. In another embodiment, the search space 820 of the ENAS method 800 is different from the search space 520 of the ENAS method 500 and the search space 720 of the ENAS method 700 in that the search space 820 only includes one of step 525 and step 526. For example, if the search space 820 only includes step 525, the ENAS method 800 uses an extinction algorithm (Extinct(x)) in step 525 to adjust and determine at least one evaluation value of the model based on the extinction condition. In another embodiment, if the search space 820 only includes step 526 , the ENAS method 800 uses a truncation algorithm (Clip(x)) in step 526 to adjust and determine at least one evaluation value of the model based on a truncation condition.

[0059] In one embodiment, the extinction condition of step 525 and the truncation condition of step 526 are static, and the necessary margin and sufficient margin used by the extinction algorithm (Extinct(x)) and the truncation algorithm (Clip) are fixed in each evolution iteration of the ENAS methods 500, 700, and 800. In another embodiment, at least one of the extinction condition and the truncation condition is variable, and in the ENAS methods of certain embodiments of the present application (such as: ENAS methods 500, 700, and 800), the necessary margin and sufficient margin used by the extinction algorithm (Extinct(x)) and the truncation algorithm (Clip(x)) can be changed with the evolution iteration of the ENAS method. In one embodiment, the necessary margin can be gradually increased with each, every other, etc. evolution iteration.

[0060] For example, the necessary margin used during the first evolution iteration of the ENAS method can be increased to the necessary margin × ratio R (e.g., greater than 1) used during the second evolution iteration of the ENAS method, the necessary margin × ratio R used during the second evolution iteration of the ENAS method can be increased to the necessary margin × (ratio R)2 used during the third evolution iteration of the ENAS method, ..., and the necessary margin × (ratio R)(generation limit-2) used during the second-to-last evolution iteration of the ENAS method can be increased to the necessary margin × (ratio R)(generation limit-1) used during the last evolution iteration of the ENAS method. Thus, each evolution iteration of such an ENAS method can retain more candidate models = necessary margin × (ratio R)(generation limit-1) than each evolution iteration of an ENAS method using a constant necessary margin.

[0061] In another example, the sufficient margin used during the first evolution iteration of the ENAS method can be increased to the cutoff margin × ratio R (e.g., greater than 1) used during the second evolution iteration of the ENAS method, the sufficient margin × ratio R used in the second evolution iteration of the ENAS method can be increased to the necessary margin × (ratio R)2 used in the third evolution iteration of the ENAS method, …, and the sufficient margin × (ratio R) (generation limit - 2) used during the second to last evolution iteration of the ENAS method can be increased to the sufficient margin × (ratio R) (generation limit - 1) used during the last evolution iteration of the ENAS method.

[0062] Fig. 9 is a functional block diagram of the device 900. For example, the device 900 may include a circuit (e.g., a storage circuit 910 and a processing circuit 920) configured to execute an ENAS method (e.g., ENAS methods 500, 700, and 800). In one embodiment, the ENAS method may include the following steps: (a) performing one or more evolution operations on an initial population of neural architectures to generate a descendant neural architecture; (b) evaluating the performance of each descendant neural architecture to obtain an evaluation value of at least one performance indicator of the descendant neural architecture; (c) adjusting the evaluation value of the descendant neural architecture based on at least one restriction on the evaluation value; (d) selecting at least one descendant neural architecture as a new neural architecture population using a selection strategy; (e) when a stopping criterion is reached, outputting a new neural architecture population as a final neural architecture population; (f) when the stopping criterion has not yet been reached, using the new neural architecture population as the initial population of the neural architecture and iterating steps (a) to (d).

[0063] In one embodiment, adjusting the evaluation value of the neural architecture may include at least one of the following: executing a truncation algorithm to apply a sufficient boundary to the evaluation value of the offspring neural architecture, and / or executing an extinction algorithm to apply a necessary boundary to the evaluation value of the offspring neural architecture. For example, at least one of the sufficient boundary and the necessary boundary may be fixed in each iteration of steps (a) to (d). In another example, at least one of the sufficient boundary and the necessary boundary may change with at least every two consecutive iterations of steps (a) to (d). In one embodiment, at least one of the sufficient boundary and the necessary boundary may gradually increase with the iteration of steps (a) to (d). For example, the necessary boundary may gradually increase with the iteration of steps (a) to (d). In another example, the sufficient boundary may gradually increase with the iteration of steps (a) to (d).

[0064] In one embodiment, the evolution operation may include at least one of crossover and mutation. In another embodiment, the selection strategy may include one of the following: non-dominated sorting, elitism, worst-case discard, roulette wheel selection, and tournament selection.

[0065] The present application also provides a non-transitory computer-readable storage medium. For example, the non-transitory computer-readable storage medium may store instructions, and the processor executes these instructions to execute the ENAS method, which includes the following steps: (a) performing one or more evolution operations on the initial population of neural architectures to generate offspring neural architectures; (b) evaluating the performance of each offspring neural architecture to obtain an evaluation value of at least one performance indicator of the offspring neural architecture; (c) adjusting the evaluation value of the offspring neural architecture based on at least one restriction on the evaluation value; (d) selecting at least one offspring neural architecture as a new neural architecture population using a selection strategy; (e) when the stopping criterion is reached, outputting the new neural architecture population as the final neural architecture population; (f) when the stopping criterion has not yet been reached, using the new neural architecture population as the initial population of the neural architecture and iterating steps (a) to (d).

[0066] In the context of this application, the term "non-transitory" as used herein is a limitation on the medium itself (i.e., physical, non-signal), rather than a limitation on the persistence of data storage (e.g., random access memory (RAM) and read-only memory (ROM)). In one embodiment, a non-transitory computer-readable storage medium may be any tangible medium that may include or store a program for use by or in conjunction with an instruction execution system, device, or apparatus. The computer-readable storage medium may be a computer-readable signal medium. Computer-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. More specific examples of computer readable storage media include an electrical connection of one or more wires, a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM) or flash memory, optical fibers, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0067] The processes and functions described herein may be implemented as computer programs that are executed by one or more processors to perform the corresponding processes and functions. The computer program may be stored or configured on a suitable medium, such as an optical storage medium or solid-state medium provided with other hardware or as part of other hardware. The computer program may also be sent in other forms, such as via the Internet or other wired or wireless telecommunications systems. For example, a computer program may be obtained and loaded into a device through some means, which may include physical media or distributed systems (including, for example, servers connected to the Internet).

[0068] A computer program can be accessed from a computer-readable medium that provides program instructions used or combined with a computer or any instruction execution system. A computer-readable medium may include any device that stores, communicates, propagates, or transmits a computer program for use or combination with an instruction execution system, device, or equipment. A computer-readable medium may be a magnetic, optical, electronic, electromagnetic, infrared, or semiconductor system (or device or equipment) or a propagation medium. A computer-readable medium may include a computer-readable non-transient storage medium, such as a semiconductor or solid-state memory, a magnetic tape, a removable computer floppy disk, a random access memory (RAM), a read-only memory (ROM), a magnetic disk, and an optical disk. A computer-readable non-transient storage medium may include all types of computer-readable media, including magnetic storage media, optical storage media, flash memory, and solid-state storage media.

[0069] Although the various aspects of the present application are described together with the specific embodiments proposed as examples, the examples can still be replaced, modified and changed. Therefore, the embodiments described in the present application are intended to illustrate rather than limit. Some changes can be made without departing from the scope of the appended claims.

Claims

1. An evolutionary neural architecture search method, characterized in that: It includes the following steps: (a) performing one or more evolution operations on an initial population of multiple neural architectures to generate multiple offspring neural architectures; (b) evaluating the performance of each of the plurality of child neural architectures to obtain an evaluation value of at least one performance indicator of the plurality of child neural architectures; (c) adjusting the plurality of evaluation values ​​of the plurality of descendant neural architectures based on at least one constraint on the plurality of evaluation values; (d) selecting at least one of the plurality of descendant neural architectures as a new population of the plurality of neural architectures using a selection strategy; (e) when a stopping criterion is reached, outputting the new population of the plurality of neural architectures as a final population of the plurality of neural architectures; as well as (f) When the stopping criterion has not been reached, taking the new population of the multiple neural architectures as the initial population of the multiple neural architectures, and iterating the multiple steps (a) to (d).

2. The evolutionary neural architecture search method according to claim 1, wherein: Adjusting the plurality of evaluation values ​​of the plurality of neural architectures comprises at least one of the following operations: executing a truncation algorithm to apply a sufficient margin to the plurality of evaluation values ​​of the plurality of child neural architectures; as well as An extinction algorithm is performed to apply a necessary margin to the plurality of evaluation values ​​of the plurality of child neural structures.

3. The evolutionary neural architecture search method according to claim 2, wherein: At least one of the sufficient boundary and the necessary boundary is maintained fixed at each iteration of the plurality of steps (a) to (d).

4. The evolutionary neural architecture search method according to claim 2, wherein: At least one of the sufficient boundary and the necessary boundary is changed at least every two consecutive iterations of the plurality of steps (a) to (d).

5. The evolutionary neural architecture search method according to claim 4, wherein: At least one of the sufficient margin and the necessary margin is gradually increased as the plurality of steps (a) to (d) are iterated.

6. The evolutionary neural architecture search method according to claim 5, wherein: The necessary margin is gradually increased as the plurality of steps (a) to (d) are iterated.

7. The evolutionary neural architecture search method according to claim 5, wherein: The sufficient margin is gradually increased as the plurality of steps (a) to (d) are iterated.

8. The evolutionary neural architecture search method according to claim 1, wherein: The plurality of evolution operations include at least one of crossover and mutation.

9. The evolutionary neural architecture search method according to claim 1, wherein: The selection strategy includes one of non-dominated sorting, elitism, worst-case discard, roulette wheel selection, and tournament selection.

10. An electronic device, characterized in that: A circuit is included, wherein the circuit is configured to perform an evolutionary neural architecture search method, wherein the evolutionary neural architecture search method comprises the following steps: (a) performing one or more evolution operations on an initial population of multiple neural architectures to generate multiple offspring neural architectures; (b) evaluating the performance of each of the plurality of child neural architectures to obtain an evaluation value of at least one performance indicator of the plurality of child neural architectures; (c) adjusting the plurality of evaluation values ​​of the plurality of descendant neural architectures based on at least one constraint on the plurality of evaluation values; (d) selecting at least one of the plurality of descendant neural architectures as a new population of the plurality of neural architectures using a selection strategy; (e) when a stopping criterion is reached, outputting the new population of the plurality of neural architectures as a final population of the plurality of neural architectures; as well as (f) When the stopping criterion has not been reached, taking the new population of the multiple neural architectures as the initial population of the multiple neural architectures, and iterating the multiple steps (a) to (d).

11. The electronic device according to claim 10, wherein: Adjusting the plurality of evaluation values ​​of the plurality of neural architectures comprises at least one of the following operations: executing a truncation algorithm to apply a sufficient margin to the plurality of evaluation values ​​of the plurality of child neural architectures; as well as An extinction algorithm is performed to apply a necessary margin to the plurality of evaluation values ​​of the plurality of child neural structures.

12. The electronic device according to claim 11, wherein: At least one of the sufficient bound and the necessary bound is maintained fixed at each iteration of the plurality of steps (a) to (d).

13. The electronic device according to claim 11, wherein: At least one of the sufficient boundary and the necessary boundary is changed at least every two consecutive iterations of the plurality of steps (a) to (d).

14. The electronic device according to claim 13, wherein: At least one of the sufficient margin and the necessary margin is gradually increased as the plurality of steps (a) to (d) are iterated.

15. The electronic device according to claim 14, wherein: The necessary margin is gradually increased as the plurality of steps (a) to (d) are iterated.

16. The electronic device as claimed in claim 14, characterized in that: The sufficient margin is gradually increased as the plurality of steps (a) to (d) are iterated.

17. The electronic device as claimed in claim 10, characterized in that: The plurality of evolution operations include at least one of crossover and mutation.

18. The electronic device as claimed in claim 10, wherein: The selection strategy includes one of non-dominated sorting, elitism, worst-case discard, roulette wheel selection, and tournament selection.

19. A non-transitory computer-readable storage medium, characterized in that: A plurality of instructions are stored. When a processor executes the plurality of instructions, the processor executes an evolutionary neural architecture search method. The evolutionary neural architecture search method includes the following steps: (a) performing one or more evolution operations on an initial population of multiple neural architectures to generate multiple offspring neural architectures; (b) evaluating the performance of each of the plurality of child neural architectures to obtain an evaluation value of at least one performance indicator of the plurality of child neural architectures; (c) adjusting the plurality of evaluation values ​​of the plurality of descendant neural architectures based on at least one constraint on the plurality of evaluation values; (d) selecting at least one of the plurality of descendant neural architectures as a new population of the plurality of neural architectures using a selection strategy; (e) when a stopping criterion is reached, outputting the new population of the plurality of neural architectures as a final population of the plurality of neural architectures; as well as (f) When the stopping criterion has not been reached, taking the new population of the multiple neural architectures as the initial population of the multiple neural architectures, and iterating the multiple steps (a) to (d).

20. The non-transitory computer readable storage medium of claim 19, wherein: Adjusting the plurality of evaluation values ​​of the plurality of neural architectures comprises at least one of the following operations: executing a truncation algorithm to apply a sufficient margin to the plurality of evaluation values ​​of the plurality of child neural architectures; as well as An extinction algorithm is performed to apply a necessary margin to the plurality of evaluation values ​​of the plurality of child neural structures.

Images