Method and device for specifying optimum architecture of neural network

Abstract

To provide a method for specifying an optimum architecture of a neural network.SOLUTION: A method includes steps of: defining a search space by using context free grammar; letting a neural network learn with training data by using an architecture candidate and then verifying the neural network made to learn with verification data; initializing a Gaussian process having a Weisfeiler-Lehman graph kernel; adapting the Gaussian process (GP) when the architecture candidate is given so as to predict verification in which the Gaussian process is achieved with the architecture candidate; performing Bayesian optimization so as to find an architecture candidate having attained the best performance; and outputting the architecture candidate having attained the best performance with the verification data.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a method, a training device, a computer program, and a machine-readable storage medium for identifying an optimal architecture of a neural network using a context-free grammar.

Background Art

[0002] Prior Art As is well known, neural architecture search (NAS) for neural networks is the following formula:

Number

[0004] , ,<0OO0027>, , and evaluated on the validation data D val and is a cost function for obtaining the generalization error of the architecture a.

[0003] Liu, Hanxiao et al., "Hierarchical representations for efficient architecture search" (arXiv preprint arXiv:1711.00436 (2017)) discloses an efficient architecture search for neural networks, the approach of which combines a new type of hierarchical genetic representation scheme that mimics a modular design pattern and a hierarchical search space that supports complex topologies. The hierarchical search space for NAS consists in synthesizing relatively high-level motifs from relatively low-level motifs. This is advantageous because the hierarchical search space generalizes the search space for NAS and allows for more flexibility in motif construction.

Prior Art Documents

Non-Patent Documents

[0004] [Non-licensed Document 1] Liu, Hanxiao et al., "Hierarchical representations for efficient architecture search" (arXiv preprint arXiv:1711.00436 (2017)) [Non-licensed Document 2] N. Chomsky, "Three models for the description of language" (IRE Transactions on Information Theory, vol. 2, no. 3, pp. 113-124, September 1956, doi: 10.1109 / TIT.1956.1056813) [Non-licensed Document 3] J. Engelfriet, "Context-free graph grammars" (Handbook of formal languages, Springer, 1997) [Non-licensed Document 4] A. Habel and H.-J. Kreowski, "On context-free graph languages ​​generated by edge replacement" (Graph-Grammars and Their Application to Computer Science, 1983) [Non-licensed Document 5] Ru, Binxin et al., "Interpretable neural architecture search via bayesian optimisation with weisfeiler-lehman kernels" (arXiv preprint arXiv:2006.07556 (2020)) [Non-licensed Document 6] Moss, Henry et al., "Boss: Bayesian optimization over string spaces" (Advances in neural information processing systems 33 (2020): 15476-15486. (Available online: https: / / arxiv.org / abs / 2010.00979 or https: / / henrymoss.github.io / files / BOSS.pdf)) [Non-Patent Document 7] McKay, Robert & Hoai, Nguyen & Whigham, PA & Shan, Yin & O'Neill, Michael (2010), "Grammar-based Genetic Programming" (a survey. Genetic Programming and Evolvable Machines. 11. 365-396. 10.1007 / s10710-010-9109-y) [Overview of the project] [Problems that the invention aims to solve]

[0005] Advantages of the invention The present invention, having the features of independent claim 1, has the advantage of enabling the definition of more general search spaces and ensuring that hierarchically synthesized motifs are permitted, in addition to more efficient searching within these spaces.

[0006] Furthermore, the present invention has the advantage of enabling the discovery of an optimal architecture that was previously undiscoverable when computer resources such as memory, energy consumption, and computing power are limited, through a more general search space.

[0007] Further aspects of the present invention are subject to further independent claims. Advantageous developments are subject to dependent claims. [Means for solving the problem]

[0008] Disclosure of the invention In a first embodiment, the present invention relates to a computer-implemented method for identifying the optimal architecture of a neural network for a given dataset including training data and validation data.

[0009] This method begins with the step of defining a search space that characterizes possible architectures of a neural network using a context-free grammar. Context-free grammars are known from, for example, N. Chomsky, "Three models for the description of language" (IRE Transactions on Information Theory, vol. 2, no. 3, pp. 113-124, September 1956, doi: 10.1109 / TIT.1956.1056813), J. Engelfriet, "Context-free graph grammars" (Handbook of formal languages, Springer, 1997), or A. Habel and H.-J. Kreowski, "On context-free graph languages ​​generated by edge replacement" (Graph-Grammars and Their Application to Computer Science, 1983). It should be noted that, based on a context-free grammar, for example, a given word can be created, and this word defines the architecture.

[0010] A hierarchical search space with multiple levels is described using the production rules of a context-free grammar. The context-free grammar describes a hierarchy of multiple levels, and the lowest level of the hierarchy defines multiple operations. These operations may be, for example, convolution of C channels, depthwise convolution, separable convolution of C channels, maximum pooling, mean pooling, or identity mapping. Each higher level of the hierarchy defines at least one rule (also called a production rule), and the lower levels can be combined with each other according to that rule, or more complex motifs can be synthesized from the lower levels.

[0011] This is followed by a step (e.g., uniform sampling) in which multiple architecture candidates are randomly selected according to a context-free grammar. For this purpose, words, particularly strings, are generated according to the grammar, and these strings can be converted into a syntax tree. Using the syntax trees corresponding to these words, an edge-attributed graph representing the neural architecture candidates is generated.

[0012] Following this, the neural network is trained on training data using candidate architectures, and the trained neural network is validated on validation data. This training can be performed based on a predetermined criterion, such as accuracy.

[0013] This is followed by a step to initialize a Gaussian process that has / uses a Weisfeiler-Lehman graph kernel. The Weisfeiler-Lehman graph kernel is publicly known from the publication "Interpretable neural architecture search via bayesian optimisation with weisfeiler-lehman kernels" by Ru, Binxin et al. (arXiv preprint arXiv:2006.07556 (2020)).

[0014] Next, given an architecture candidate, a step is taken to fit a Gaussian process (GP) to predict the validation quantity that the Gaussian process will achieve. The GP receives the architecture candidate as an input quantity, which is preferably provided as an attributed directed graph.

[0015] Following this, steps i. to iii. below are repeated multiple times (up to 160 times). Repeating these steps has been found to be sufficiently meaningful. i. A step of identifying the next candidate architecture to be evaluated based on an acquisition function that depends on a Gaussian process, wherein the acquisition function is optimized by an evolutionary algorithm, for example, as disclosed in McKay et al., "Grammar-based Genetic Programming: a survey." Preferably, an "expected improvement" acquisition function is used as the acquisition function. It should be noted that the step of identifying the next candidate architecture to be evaluated may alternatively be carried out using random search and / or mutation. ii. On the training data, further train a neural network using the architecture candidates to be evaluated, and on the validation data, verify the further trained neural network. iii. When a previously used architecture candidate is given, adapt the Gaussian process to predict the validation measure achieved by the architecture candidate.

[0016] Finally, the step of outputting the architecture candidate that achieved the best performance on the validation data is performed.

[0017] It is proposed that an evolutionary algorithm uses mutation and crossover, applies mutation and crossover to the respective syntax trees characterizing the architecture candidates, and validates the new syntax trees obtained by mutation or crossover according to a context-free grammar. This has the advantage that the architecture candidates are always kept valid (i.e., the architecture candidates are always kept within the language generated by the grammar), whereby the manipulated architecture is always executable.

[0018] It is further proposed to randomly perform self-crossover instead of crossover and exchange the branches of the same syntax tree in the self-crossover in the syntax tree. This has the advantageous effect of implicit regularization.

[0019] Furthermore, it is proposed that the acquisition function be a grammar-guided acquisition function (see, for example, Moss, Henry et al., "Boss: Bayesian optimization over string spaces" (Advances in neural information processing systems 33 (2020): 15476-15486. (available online: https: / / arxiv.org / abs / 2010.00979 or https: / / henrymoss.github.io / files / BOSS.pdf))), and that the acquisition function be evaluated by a grammar-guided evolutionary algorithm. Grammar-guided evolutionary algorithms are publicly known, for example, from publications such as "Grammar-based Genetic Programming" by McKay, Robert & Hoai, Nguyen & Whigham, PA & Shan, Yin & O'Neill, and Michael (2010) (a survey. Genetic Programming and Evolvable Machines. 11. 365-396. 10.1007 / s10710-010-9109-y).

[0020] It is further proposed that the change in resolution can be modeled using context-free grammars. This allows for exploration across the entire neural architecture. The advantage in this case is that checks regarding dimensional differences are not required. It is further proposed that the context-free grammars have additional conditions that characterize the properties of the architecture. Such conditions could describe, for example, the maximum depth, the maximum number of layers, or the maximum number of convolutional layers, or the number of downsampling operations.

[0021] Furthermore, it is further proposed that when training a neural network, the cost function may have a first function that evaluates the performance of the machine learning system in terms of its performance, for example, in terms of the accuracy of segmentation, object recognition, or similar functions, and optionally a second function that estimates the latency of the machine learning system depending on the path length and edge calculations. Alternatively or additionally, the second function may also estimate the computing resource consumption of the path.

[0022] In a further embodiment of the present invention, a computer-implemented method is proposed for using the neural network output of the first embodiment as a classifier for classifying sensor signals. In addition to the steps of the first embodiment, the following further steps are performed: receiving a sensor signal including data from an image sensor; determining an input signal dependent on the sensor signal; and supplying the input signal to the classifier to obtain an output signal that characterizes the classification of the input signal.

[0023] An image classifier associates an input image with one or more classes of a predetermined classification. For example, images of nominally identical mass-produced products can be used as input images. For instance, an image classifier can be trained to associate input images with one or more classes from at least two possible classes representing the quality of each product.

[0024] An image classifier, such as a neural network, may be provided with a structure that allows it to be trained to identify and distinguish, for example, pedestrians and / or vehicles and / or traffic signs and / or traffic lights and / or road surfaces and / or human faces and / or medical abnormalities within imaging sensor images. Alternatively, the classifier, such as a neural network, may be provided with a structure that allows it to be trained to identify voice commands within audio sensor signals.

[0025] It is further proposed that the output neural network can determine an output amount depending on the sensor's detected sensor quantity, and that this output amount can then be used to determine a control quantity, for example, by a control unit.

[0026] This control quantity can be used to control actuators in a technical system. The technical system may be, for example, a semi-autonomous machine, a semi-autonomous vehicle, a robot, a tool, a machine tool, or a flying object such as a drone. For example, input quantities can be identified based on detected sensor data and provided to a machine learning system. This sensor data may be detectable by sensors in the technical system, such as cameras, or alternatively, received from an external source.

[0027] In a further embodiment, the present invention relates to an apparatus configured to carry out the above method, a computer program configured to carry out the above method, and a machine-readable storage medium in which the computer program is stored.

[0028] Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. [Brief explanation of the drawing]

[0029] [Figure 1] This figure shows one embodiment of the CFG grammar, which includes three levels. [Figure 2] This is a schematic flowchart of one embodiment of the present invention. [Figure 3] This diagram schematically illustrates an embodiment for controlling at least a semi-autonomous robot. [Figure 4] This diagram schematically illustrates an example of controlling a manufacturing system. [Figure 5] This diagram schematically illustrates an example of controlling an access system. [Figure 6]This diagram schematically illustrates an example of controlling a monitoring system. [Figure 7] This diagram schematically illustrates an example of controlling a personal assistant. [Figure 8] This figure schematically illustrates an example of controlling a medical imaging system. [Figure 9] This is a schematic diagram of the training device. [Modes for carrying out the invention]

[0030] A neural architecture is a function composition of multiple operations, such as convolutions or other functions. It is well known that a neural architecture can be represented as a computation graph having an edge-attributed DAG with a single source and a single sink, where edges are associated with operations and nodes are associated with latent representations.

[0031] It has been proposed to use CFGs to represent the (hierarchical) search space for NAS, which has the advantage that CFGs can compactly represent the hierarchical search space. CFGs define the valid space for neural architectures and the rules for selecting and developing them. Neural architectures are efficiently randomly generated, mutated, and represented in the string space, while each string represents the computation graph of the neural architecture, thus implicit processing takes place in the graph space.

[0032] The following sections will explain how hierarchical search spaces can be represented using CFGs, and how string representations can be converted to corresponding computation graphs according to the CFGs of neural architectures.

[0033] Terminal symbols in a CFG are associated with topology or primitive operations, while non-terminal symbols enable the recursive generation of hierarchical structures. The generation rules describe the synthesis process and evolution of neural architectures in the generated search space (i.e., the domain-specific language of neural architectures). This allows for the synthesis of relatively high-level, complex motifs from relatively low-level, simple motifs.

[0034] Figure 1 shows one embodiment of the CFG grammar, which includes three levels. Level 1 defines operations, while higher levels describe possible combinations of the levels below them.

[0035] Figure 2 shows a flowchart 20 of one embodiment of the present invention for identifying the optimal neural network architecture for a given dataset.

[0036] Step (S21) is performed to define a search space that characterizes possible architectures of a neural network using a context-free grammar, wherein the context-free grammar characterizes a hierarchy of multiple levels, the lowest level of the hierarchy defines multiple operations, the upper levels of the hierarchy define at least one rule, and the lower levels can be combined or combined with each other according to that rule.

[0037] This is followed by a step (S22) in which multiple architecture candidates are randomly selected according to a context-free grammar. Furthermore, a neural network is trained on training data using the architecture candidates, and the trained neural network is validated on validation data.

[0038] This is followed by a step (S23) in which a Gaussian process having a Weisfeiler-Lehman graph kernel is initialized. Furthermore, given an architecture candidate, a step is performed to adapt the Gaussian process (GP) so that it predicts the verification that the Gaussian process will achieve with that architecture candidate.

[0039] In step S24, the following substeps are repeated multiple times, namely, A substep that identifies the next candidate architecture to be evaluated based on an acquisition function that depends on a Gaussian process, wherein the acquisition function is optimized by an evolutionary algorithm. The process involves a substep where a further neural network is trained on training data using candidate architectures to be evaluated, and then the further trained neural network is validated on validation data. Given a list of previously used architecture candidates, a substep involves fitting a Gaussian process to predict the verification achieved by that architecture candidate, This is repeated multiple times.

[0040] After the repeated steps in step S24 are completed, the final step (S25) is performed to output the architecture candidate that achieved the best performance on the validation data, in particular the corresponding trained neural network.

[0041] Figure 3 schematically shows an actuator with a control system 40. The environment 20 of the actuator 10 is detected, preferably at regular time intervals, by a sensor 30, in particular an imaging sensor such as a video sensor, which can be provided by multiple sensors, for example, a stereo camera. Other imaging sensors such as radar, ultrasound, or Lidar are also possible. A thermal camera is also possible. The sensor signals S of sensor 30—or, if there are multiple sensors, one sensor signal S from each—are transmitted to the control system 40. Thus, the control system 40 receives a sequence of sensor signals S. The control system 40 identifies a drive signal A from this sequence of sensor signals S, and this drive signal A is transmitted to the actuator 10. The actuator 10 can convert the received control command into mechanical motion or a change in a physical quantity. The actuator 10 can, for example, convert the control command A into electrical, hydraulic, pneumatic, thermal, magnetic, and / or mechanical motion, or cause a change. Examples that are specific but not limited include electric motors, electroactive polymers, hydraulic cylinders, piezoelectric actuators, pneumatic actuators, servo mechanisms, solenoids, and stepping motors.

[0042] The control system 40 receives a sequence of sensor signals S from the sensor 30 in an optional receiving unit 50, and the receiving unit 50 converts the sequence of sensor signals S into a sequence of input images x (alternatively, the sensor signals S can also be received directly as input images x). The input images x may be, for example, a part of the sensor signals S or the result of subsequent processing. The input images x include individual frames of a video recording. In other words, the input images x are determined depending on the sensor signals S. The sequence of input images x is supplied to the neural network 60 output from step S25.

[0043] The output neural network 60 is preferably parameterized by parameters, which are stored in a parameter memory and supplied by the parameter memory.

[0044] The output neural network 60 identifies output quantities y from the input image x. These output quantities y may include, in particular, classification and / or semantic segmentation of the input image x. The output quantities y are supplied to an optional deformation unit 80, which identifies a drive signal A from these output quantities y, and this drive signal A is supplied to the actuator 10 to drive the actuator 10 accordingly. The output quantities y include information about the object detected by the sensor 30.

[0045] Actuator 10 receives drive signal A, is driven accordingly, and performs a corresponding action. In this case, actuator 10 may include drive logic (which is not necessarily structurally integrated), which identifies a second drive signal from drive signal A, and then actuator 10 is driven by this second drive signal.

[0046] In further embodiments, the control system 40 includes a sensor 30. In yet another embodiment, the control system 40 also includes an actuator 10, either alternatively or additionally.

[0047] In a further preferred embodiment, the control system 40 includes one or more processors 45 and at least one machine-readable storage medium 46 on which instructions are stored, such that when executed on the processors 45, the control system 40 is made to carry out the method according to the present invention.

[0048] In an alternative embodiment, a display unit 10a capable of displaying the output amount of the control system 40 is provided instead of or in addition to the actuator 10.

[0049] In the preferred embodiment shown in Figure 3, a control system 40 is used to control the actuators of at least a semi-autonomous robot, here in this case at least a semi-autonomous automobile 100. The sensor 30 may be, for example, a video sensor preferably located inside the automobile 100.

[0050] Preferably, the actuator 10 located within the automobile 100 may be, for example, the brake, drive unit, or steering unit of the automobile 100. In this case, the drive signal A can be identified such that the actuator 10 is driven so that the automobile 100 does not collide with an object that has been reliably identified, for example by an artificial neural network 60, particularly when the object is of a predetermined class, such as a pedestrian.

[0051] Alternatively, the at least semi-autonomous robot may be another mobile robot (not shown), for example, a robot that moves forward by flying, swimming, diving, or walking. The mobile robot may be, for example, an at least semi-autonomous lawnmower or an at least semi-autonomous cleaning robot. In such cases as well, the drive signal A can be identified so that the drive unit and / or steering unit of the mobile robot are driven so that the at least semi-autonomous robot prevents collisions with objects identified, for example, by the artificial neural network 60.

[0052] Figure 4 shows an embodiment in which a control system 40 is used to drive the manufacturing machine 11 of the manufacturing system 200 by driving an actuator 10 that controls the manufacturing machine 11. The manufacturing machine 11 may be, for example, a machine for punching, sawing, drilling, milling, and / or cutting.

[0053] In this case, the sensor 30 may be, for example, an optical sensor that detects the characteristics of manufactured products 12a and 12b. These manufactured products 12a and 12b may be movable. The actuator 10 that controls the manufacturing machine 11 can be driven depending on the correspondence of the detected manufactured products 12a and 12b so that the subsequent processing steps for the correct manufactured product among the multiple manufactured products 12a and 12b are carried out by the manufacturing machine 11. By identifying the correct characteristics of identical manufactured products among the multiple manufactured products 12a and 12b (i.e., without errors in correspondence), the manufacturing machine 11 can also appropriately adapt similar manufacturing steps for the processing of subsequent manufactured products.

[0054] Figure 5 shows an embodiment in which a control system 40 is used to control an access system 300. The access system 300 may include physical access control, such as a door 401. A video sensor 30 is configured to detect people. The detected image can be interpreted using an object recognition system 60. If multiple people are detected simultaneously, their identifiers can be identified with particular certainty by, for example, analyzing the movements of these people (i.e., objects) by associating them with each other. The actuator 10 may be a lock that releases or does not release access control, such as a lock that opens or does not open the door 401, depending on a drive signal A. For this purpose, the drive signal A can be selected depending on the interpretation of the object recognition system 60, for example, depending on the identified identifier of a person. Logical access control can also be provided instead of physical access control.

[0055] Figure 6 shows an embodiment in which a control system 40 is used to control a monitoring system 400. This embodiment differs from the embodiment shown in Figure 5 in that a display unit 10a driven by the control system 40 is provided instead of an actuator 10. For example, an artificial neural network 60 can reliably identify the identifier of an object captured by the video sensor 30, and based on this identifier, it is possible to estimate, for example, which object is suspicious, and then select a drive signal A so that the object is displayed with color emphasis by the display unit 10a.

[0056] Figure 7 shows an embodiment in which the control system 40 is used to control the personal assistant 250. The sensor 30 is preferably an optical sensor that receives images of the user 249's gestures.

[0057] The control system 40, depending on the signal from the sensor 30, identifies a drive signal A for the personal assistant 250, for example, by having a neural network perform gesture recognition. This identified drive signal A is then transmitted to the personal assistant 250, thereby driving the personal assistant 250 accordingly. This identified drive signal A can be selected in particular to correspond to a desired drive inferred by the user 249. This inferred desired drive can be identified depending on a gesture recognized by the artificial neural network 60. The control system 40 can then select a drive signal A to transmit to the personal assistant 250 depending on the inferred desired drive, and / or select a drive signal A to transmit to the personal assistant 250 in accordance with the inferred desired drive.

[0058] Such corresponding actions may include, for example, the personal assistant 250 retrieving information from a database and playing this information back for the user 249 to receive.

[0059] Instead of the personal assistant 250, household appliances (not shown), particularly washing machines, stoves, ovens, microwave ovens, or dishwashers, can be provided and operated accordingly.

[0060] Figure 8 shows an embodiment in which a control system 40 is used to control a medical imaging system 500, such as an MRT device, an X-ray device, or an ultrasound device. The sensor 30 can be provided, for example, by an imaging sensor, and the control system 40 drives a display unit 10a. For example, a neural network 60 can determine whether an area recorded by the imaging sensor is prominent, and then a drive signal A can be selected so that this area is displayed with color emphasis by the display unit 10a.

[0061] Figure 9 schematically shows a training device 500, which includes a supply unit 51 that provides input images from a training dataset. The input images are supplied to a neural network 52 to be trained, and the neural network 52 identifies output quantities from these input images. The output quantities and input images are supplied to a determination unit 53, which identifies updated parameters from these output quantities and input images, and these updated parameters are transmitted to a parameter memory P, where they replace the current parameters. The determination unit 53 is configured to perform steps S23 and / or S24 of the method in Figure 2.

[0062] The method performed by the training device 500 can be implemented as a computer program and stored in the machine-readable storage medium 54, and the computer program can be executed by the processor 55.

[0063] The term "computer" includes any device for processing configurable computational rules. These computational rules may exist in the form of software, in the form of hardware, or in a hybrid form of software and hardware.

Claims

1. 1. A method for identifying an optimal architecture of a neural network for a given data set including training data and validation data, comprising: - a step (S21) of defining a search space characterizing possible architectures of the neural network using a context-free grammar, the context-free grammar characterizing a hierarchy of levels, the lowest levels of the hierarchy defining a number of operations, and higher levels of the hierarchy defining at least one rule according to which the lower levels may be combined with one another; A step (S22) of randomly extracting a plurality of architecture candidates according to the context-free grammar; training a neural network using the candidate architecture on the training data and validating the trained neural network on the validation data; initializing a Gaussian process with a Weisfeiler-Lehman graph kernel (S23); adapting the Gaussian Process (GP) such that, given the candidate architecture, the GP predicts the verification achieved by the candidate architecture; A step (S24) of repeating the following steps i. to iii. multiple times, i. Identifying the next candidate architecture to be evaluated based on an acquisition function that depends on the Gaussian process, the acquisition function being optimized by an evolutionary algorithm; ii. training a further neural network using the candidate architecture to be evaluated on the training data and validating the trained further neural network on the validation data; iii. Fitting the Gaussian process such that it predicts the verification achieved by a candidate architecture given previously used candidate architectures; a step (S24) of repeating the above a plurality of times; a step (S25) of outputting the architecture candidate that achieved the best performance on the verification data; A method comprising:

2. The evolutionary algorithm uses mutation and crossover, said mutation and said crossover are applied to a syntax tree characterizing said candidate architectures; a new syntax tree obtained by the mutation or the crossover is checked according to the context-free grammar; The method of claim 1.

3. Instead of crossover, self-crossover is performed randomly, The self-crossover in the syntax tree involves exchanging edges. The method of claim 2.

4. The acquisition function is a grammar-guided acquisition function, The acquisition function is evaluated by a grammar guided evolutionary algorithm. The method of claim 1.

5. the lowest level of the context-free grammar comprises a downsampling operation; The method of claim 1.

6. the context-free grammar further has conditions that characterize properties of the architecture; The method of claim 1.

7. The input quantity is an image, The machine learning system is an image classifier, The method of claim 1.

8. Apparatus configured to carry out the method of any one of claims 1 to 7.

9. A training device configured to carry out the method of any one of claims 1 to 7.

10. A computer program comprising instructions that, when executed by a computer, cause the computer to carry out the method of claim 1.

11. A machine-readable storage medium having stored thereon the computer program of claim 10.

Images