Method and device for automatically generating artificial neural networks

Abstract

A method for automatically generating an artificial neural network (60), the artificial neural network comprising modules and connections, the connections linking the modules, wherein individual modules and / or connections are added to a current initial network (60), wherein the modules and / or connections to be added are randomly selected from a plurality of predeterminable possible addable modules and connections, and wherein a plurality of possible extensions (61) of the current initial network (60) are generated by adding the modules and / or connections to be added to the initial network, and wherein one of the extensions (61) from the plurality of possible extensions (61) is subsequently selected to be used as the current initial network (60) in the next run through the method.

Description

Technical Field

[0001] The present invention relates to a method for automatically generating an artificial neural network, a computer program, a machine-readable storage medium and a system; the computer program comprises instructions, the instructions are configured to implement the method when the computer program is executed on a computer; the computer program is stored on the machine-readable storage medium; the system in particular comprises one or more computers, the system being configured to implement the method. Background Art

[0002] From “Reinforcement Learning for Architecture Search by Network Transformation” (arXivpreprint arXiv:1707.04873vl, 2017, Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, Jun Wang) a method for automatically generating the architecture of an artificial neural network with the aid of reinforcement learning is known, wherein the network is deepened or widened while retaining its functionality. Summary of the invention

[0003] Advantages of the present invention

[0004] In contrast, the method having the features of the independent claim has the advantage that it enables a very efficient and automated creation of a powerful, enlarged artificial neural network.

[0005] Advantageous developments are the subject matter of the independent claims.

[0006] Disclosure of the Invention

[0007] In a first aspect, the invention relates to a method for automatically generating an artificial neural network comprising modules and connections which link these modules, wherein individual modules and / or connections are added to a current initial network.

[0008] wherein the module and / or connection to be added is randomly selected from a plurality of possible addable modules and connections that can be predetermined, and wherein a plurality of possible extensions of the current initial network are generated by respectively adding the module and / or connection to be added to the initial network (that is, each of the extensions comprises the current initial network and the added module and / or connection),

[0009] And in this case one of the expansions is subsequently selected from a plurality of possible expansions in order to be used as the current starting network in the next run through the method.

[0010] Modules are units of a neural network which determine (possibly vector-valued) output variables as a function of (possibly vector-valued) input variables. Modules can be layers of an artificial neural network or artificial neurons, for example.

[0011] That is, “one by one” means that the method comprises a plurality of rounds, wherein in each round possible extensions are generated starting from the current starting network and the (preferably best) extension is subsequently selected from these to be used as the current starting network in the next round.

[0012] That is, a plurality of possible expansions are repeatedly formed and only the (preferably best) expansion is selected from these possible expansions, which is then used as the basis for further expansions. In this way, the artificial neural network is gradually expanded. It has been found that this greedy behavior surprisingly leads to very powerful artificial neural networks in a short time, without, for example, introducing obstructive subminima.

[0013] In an improved solution, it can be provided that, when adding corresponding modules and / or connections, the pre-given parameters characterizing the corresponding modules and / or connections to be added are selected so that: for each possible input variable, the initial network to which the corresponding modules and / or connections to be added are added provides unchanged output variables before and after adding the corresponding modules and / or connections to be added, respectively.

[0014] That is, the corresponding parameters of the modules and / or connections to be added are selected so that: the initial network and the network extended by adding the modules and / or connections to be added are functionally equivalent. Through this, possible extensions become much more trainable quickly, which speeds up the method.

[0015] Alternatively or additionally, provision can be made that, before the first addition of modules and / or connections, the current initial network is trained with the initial training data set: the current initial network serves as the starting point for the first addition of modules and / or connections. This leads to a further acceleration of the method.

[0016] Alternatively or additionally, provision can be made for each possible expansion of the current initial network to be trained with a training data set before one of the expansions from a plurality of possible expansions is selected for use as the current initial network in the next run through the method.

[0017] The selection of one of the plurality of possible extensions may be performed based on a variable that characterizes a respective property of each of the plurality of possible extensions. The variable may preferably be selected based on an output variable of a respective one of the plurality of extensions.

[0018] In a development of this aspect, it can be provided that the extensions of the initial network are respectively trained using a gradient descent method, wherein a parameter characterizing the learning rate of the gradient descent method is selected to decrease with increasing number of performed training epochs.

[0019] That is, the variable characterizing the learning rate is selected such that: the variable decreases as the learning duration increases. This has the advantage that the gradient descent method converges particularly quickly. It has been recognized that fluctuations in the estimated gradient can be particularly reliably offset in this way.

[0020] In this case, the course of the decrease of the parameter characterizing the learning rate can be characterized by means of trigonometric functions.

[0021] This is a particularly simple way to characterize a reduced learning rate with a preferably smooth course. For example, one can choose a variable characterizing the learning rate , where t denotes the training time (measured, for example, in a training session) and T denotes the predeterminable total time.

[0022] In a further aspect of the invention, it can be provided that, according to one of the above claims, the predeterminable plurality of possible modules and / or connections comprises a normalization module and / or a skip connection and / or a module containing a nonlinear function, wherein the nonlinear function is idempotent for at least one value of a parameter characterizing its properties. By means of the condition of the nonlinear function, the characteristic parameter can be selected such that the nonlinear function does not change the output variable if an idempotent function already exists in the initial network immediately before or after the idempotent function.

[0023] Normalization modules can be, for example, layer normalization or batch normalization. It has been recognized that these modules can be initialized efficiently and that by assimilating these modules into a predeterminable plurality of possible modules, efficient artificial neural networks can be generated particularly efficiently.

[0024] In particular, it can be provided that the predefined multiple possible modules and / or connections include normalization modules and skip connections and modules including nonlinear functions and modules including linear functions, and include modules for widening existing linear layers by adding units or channels, and include modules for widening the linear layers by duplicating the linear layers.

[0025] It has been shown that artificial neural networks can be generated particularly efficiently using a predefinable plurality of possible modules and / or connections.

[0026] In other aspects, the present invention relates to a computer program configured to implement the method according to the present invention. That is, the computer program comprises instructions which, when a computer runs the computer program, cause the computer to implement the method according to the present invention.

[0027] In other aspects, the invention relates to a machine-readable storage medium on which the computer program is stored; and to a computer configured to implement the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Subsequently, the embodiments of the present invention will be described in more detail with reference to the accompanying drawings. In the drawings:

[0029] Figure 1 Schematically illustrates possible applications of the artificial neural network generated according to the invention;

[0030] Figure 2 Schematically illustrates an exemplary learning system for generating an artificial neural network;

[0031] Figure 3 A possible sequence of steps for the method for generating an artificial neural network is shown in a flow chart. DETAILED DESCRIPTION

[0032] Figure 1 An actuator 10 is shown by way of example in its environment 20 interacting with an actuator control system 40. The actuator 10 and the environment 20 are also collectively referred to below as an actuator system. The state of the actuator system is detected by means of a sensor 30, which can also be given by a plurality of sensors. An output signal S of the sensor 30 is transmitted to the actuator control system 40. The actuator control system 40 determines therefrom an actuation signal A, which the actuator 10 receives.

[0033] Actuator 10 may be, for example, a (partially) autonomous robot, for example a (partially) autonomous motor vehicle. Sensor 30 may be, for example, one or more video sensors and / or one or more radar sensors and / or one or more ultrasonic sensors and / or one or more position sensors (e.g. GPS). Alternatively or in addition, sensor 30 may also include an information system that determines information about the state of the actuator system, such as, for example, a weather information system that determines the current or future state of the weather in environment 20.

[0034] In another embodiment, the actuator 10 may be a manufacturing robot, and then the sensor 30 is, for example, an optical sensor that detects properties of a manufacturing product of the manufacturing robot.

[0035] In other embodiments, the actuator 10 may be a release system configured to release or not release the activity of the device. The sensor 30 may be, for example, an optical sensor (e.g. for detecting images or video data) configured to detect a face. The actuator 10 determines a release signal based on the control signal A, which can be used to release the device based on the value of the release signal. The device may be, for example, a physical or logical access control device. Based on the value of the control signal A, the access control device may then be set to allow access or not allow access.

[0036] It is also possible to control, for example, a medical diagnosis system instead of the actuator 10, which outputs a medical diagnosis as a function of the control signal A. If the sensor 30 is an optical sensor that detects medical images, the control signal A can be, for example, a semantic segmentation of the images, optionally with annotations, in which regions that are particularly important for the medical diagnosis are highlighted. In this case, reference numeral 40 designates an agent control system.

[0037] The actuator control system 40 receives the output signal S of the sensor in an optional receiving unit 50, which converts the output signal S into a data signal x (alternatively, the output signal S can also be directly adopted as the data signal x). The data signal x can be, for example, a fragment of the output signal S or a further processing. The data signal x is supplied to the artificial neural network 60 as an input signal. The structure of the artificial neural network is characterized by a structural parameter q, which is stored in a structural parameter memory Q. The artificial neural network 60 is parameterized by a predeterminable parameter p, which is stored in a parameter memory P.

[0038] In the preferred embodiment described below, the data signal x is a two-dimensional image signal, the pixels of which are characterized by, for example, one value (brightness coding) or three values ​​(RGB coding). However, the data signal x may also be another one-dimensional or multi-dimensional data signal.

[0039] The artificial neural network 60 determines an associated output signal y from an input signal x (here, an image signal x), for example a pixel-by-pixel semantic segmentation, wherein a semantic value is assigned to each region of the image signal x.

[0040] The output signal y (here the semantic segmentation y) is transmitted to the output unit 80, which determines therefrom the control signal A. That is to say, the control signal A is determined as a function of the semantic segmentation y. In other embodiments, it is also possible for the output unit 80 to adopt the output signal y directly as the control signal A.

[0041] In one embodiment, the actuator control system 40 includes a computer and a machine-readable storage medium (not shown) on which a computer program is stored, which, when executed by the computer, causes the computer to implement the described functionality of the actuator control system 40 .

[0042] Figure 2 An exemplary embodiment of a system 140 for generating an artificial neural network 60 is schematically shown. A training data unit 150 determines suitable input variables x, which are supplied to the artificial neural network. For example, the training data unit 150 accesses a computer-implemented database in which a set of training data is stored and selects the input variable x, for example randomly, from the set of training data. Optionally, the training data unit 150 also determines the desired output variable y assigned to the input variable x. s , the output variable y s is supplied to an evaluation unit 180 .

[0043] The artificial neural network x is configured to determine an associated output variable y from the input variable x supplied to it. The output variable y is supplied to an evaluation unit 180 .

[0044] The modification unit 160 uses, for example, Figure 31 , a new structural parameter q' and a new parameter p' are determined by the method illustrated in FIG. 1 , and the new structural parameter q' and the new parameter p' are supplied to the structural parameter memory Q and the parameter memory P, where the new structural parameter q' and the new parameter p' replace the structural parameter q and the parameter p. Possible variations of the structural parameter q are stored in the modification unit 160. These possible variations of the structural parameter q correspond to possible combinations of modules and connections that can be added to the artificial neural network 60.

[0045] According to one aspect of the invention, starting from an initial version of an artificial neural network 60, i.e. an initial network, a possible expansion 61 of the artificial neural network 60 is thus completed each time by supplying structural parameters q and parameters p to the artificial neural network 60. As with the artificial neural network 60, input variables x can be supplied to each of the expansions 61 accordingly, so that the expansion then determines the corresponding output variables y.

[0046] The evaluation unit 180 can, for example, use the output variable y and the desired output variable y s The relevant cost function (loss function) is used to measure the feature variables , the characteristic variable characterizes the performance of the artificial neural network 60 or the extension 61.

[0047] When the structural parameter q is fixed, the artificial neural network 60 or the extension 61 can be optimized by changing the parameter p. , the most efficient extension 61 can be selected.

[0048] The system 140 may include, for example, one computer or multiple computers, and in an embodiment, the system 140 includes a machine-readable storage medium 200 on which a computer program 210 is stored, wherein the computer program 210 is configured to implement the method according to the present invention.

[0049] Figure 3 A possible sequence of steps of a method for generating an artificial neural network 60 is shown in a flow chart.

[0050] First ( 1000 ), an initial network 60 is determined (for example by reading out characteristic structural parameters q from a memory area). Optionally, parameters p assigned to the initial network are trained.

[0051] Next (1100), possible variants of the structural parameter q are selected by means of a (pseudo) random number generator, for example 8 or 16 different variants (of course, this number can be freely selected, but it has been shown that numbers in the range of 8 to 16 lead to particularly good results). In other words, a plurality (for example 8 or 16) of possible extensions 61 are generated in parallel, which are each obtained from the initial network 60 by adding a combination of modules and / or connections, which modules and / or connections include one or more of the following combinations:

[0052] A first possible combination is to add a linear layer before a predeterminable (and preferably randomly selected) reference layer of the initial network 60. The predeterminable reference layer can also be the output of the output network 60. That is, if the input variable z e is fed to a reference layer, which generates an output variable from it, and is preceded by a linear layer, which receives an input variable z e , from which the transformed input variable z is determined e '=a*z e +b (where parameters a and b are freely selectable) and replace the input variable z e Feed the transformed input variable z' to the reference layer e Preferably, b=0 and a=1 are selected.

[0053] A second possible combination is to add a nonlinear layer before a predeterminable reference layer. In this case, an idempotent function h of the reference layer is identified (for example, the input variable z e is fed to the function relu(x)=max(0,x)). The parameters characterizing the idempotent function h to be added are preferably selected to be equal to the corresponding parameters of the idempotent function present in the reference layer, and the output variable of the idempotent function h to be added is fed to the idempotent function present in the reference layer. Alternatively, nonlinear functions can also be embedded without necessarily identifying the idempotent function in the reference layer. Then, the idempotent function to be added can be selected (for example, as a convex linear combination (konvexe Lineararkombination)) so that: the idempotent function to be added is the same for the determined parameters. That is, instead of the input variable z e , feeding the transformed input variable z to the reference layer e ' =a*h(z e ) + (la)*z e Preferably, a=0 is initially selected.

[0054] A third possible combination is to add individual neurons to a predeterminable reference layer. In this case, the parameters characterizing the functional correlation of the input variables into the output variables of the reference layer are preferably selected such that the functional correlation of the output variables of the reference layer with the input variables remains unchanged.

[0055] A fourth possible combination is to add individual connections to a predeterminable reference layer. For example, connections can be added in parallel with already existing connections. In this case, the parameters characterizing the functional association of the input variables into the output variables of the reference layer are preferably selected such that the functional correlation of the output variables of the reference layer with the input variables remains unchanged.

[0056] A fifth possible combination is a jump connection added between two (preferably randomly selected) predeterminable reference layers, i.e. a jump connection from a predeterminable output layer to a predeterminable target layer. Such a jump connection can or can be selected such that its output variable is added to another output variable in the target layer (in this case, the parameters of the jump connection are preferably selected such that the weight of the jump connection is selected to be equal to 0, while the weight of the added output variable is selected to be equal to 1).

[0057] The sixth possibility is similar to the fifth possibility, except that the skip connection is chosen such that its output variable is connected to a further output variable in the target layer (in this case, the weights weighting the connected components are chosen to be preferably equal to 0).

[0058] A seventh possibility is to add a normalization module after a predeterminable (and preferably randomly selected) reference layer of the initial network 60. r , determine the statistical parameters μ, σ, which can correspond, for example, to the mean value of the selected parameters or to the estimated standard distribution. For example, these parameters can be determined by averaging the rows and / or columns of a layer, or by averaging the output variables of the reference layer over a batch of input variables x of the extension 61. The output variable y of the normalization module r Then the choices can be:

[0059] .

[0060] The parameter ε can in this case be set to a fixed chosen small value for reasons of numerical stability, for example ε=10 -10 The parameters γ, β are preferably chosen as:

[0061]

[0062] .

[0063] Next (1200), the generated possible extensions 61 are trained. For this purpose, during a training phase, input variables x are supplied to each of the generated possible extensions 61, and characteristic variables characterizing the performance are determined from the output variables y determined from the input variables x. The parameter p is now varied iteratively in steps where p0=p, In this case, η t is a parameter that characterizes the learning rate. Preferably, the parameter is selected to decrease with increasing iteration index t, for example with a total time T that can be predetermined. The parameter is selected proportionally. This iteration is performed for a long time until the parameter p is at index t c Next, choose a new parameter p'=p tc .

[0064] Next ( 1300 ), predeterminable input variables are supplied to the generated and trained possible expansions 61 , and the resulting characteristic variable characterizing the performance is determined for each of the possible expansions 61 . It is also conceivable to take into account the characteristic variables determined in step (1200) during the training process. .

[0065] Finally (1400), based on the measured characteristic variables Select the following extension 61: the characteristic variable of the extension 61 is the best, e.g. the largest.

[0066] Now it can be checked whether the selected extension 61 has achieved a characteristic variable that is significantly better than the initial network 60 If this is not the case, the method ends and the selected expansion 61 (alternatively the current initial network 60 ) results in the generated artificial neural network.

[0067] Otherwise, the selected extension 61 now replaces the current initial network 60 , ie the structural parameter q′ assigned to the selected extension 61 and the determined parameter p′ assigned to the extension 61 replace the structural parameter q and the parameter p. The method then branches back to step 1100 .

[0068] It should be noted that with the generation of the artificial neural network 60, a corresponding actuator or proxy control system 40 can also be generated in which the artificial neural network 60 is used. For example, all other components can be predefined unchanged and only the included artificial neural network 60 can be generated.

[0069] The method can be implemented as a computer program, ie, software, or in hardware or in a hybrid form consisting of hardware and software.

Claims

1. A computer-implemented method for automatically generating an artificial neural network (60), the artificial neural network (60) comprising modules and connections linking the modules, wherein modules and / or connections are added one by one to the current initial network (60), wherein the module and / or connection to be added is randomly selected from a plurality of possible addable modules and connections which can be predetermined, wherein a plurality of possible extensions (61) of the current initial network (60) are generated by respectively adding the modules and / or connections to be added to the initial network, in, When adding a corresponding module and / or connection, the predeterminable parameters (p) characterizing the corresponding module and / or connection to be added are selected so that, for each possible input variable (x), the initial network (60) to which the corresponding module and / or connection to be added is added provides an unchanged output variable (y) before and after adding the corresponding module and / or connection to be added, respectively. wherein one of the extensions (61) from the plurality of possible extensions (61) is then selected to be used as the current initial network (60) in the next run through the method, wherein before the first addition of modules and / or connections, the initial training data set is used to train the current initial network (60) that serves as the starting point for the first addition of modules and / or connections, wherein an output signal (S) of a sensor (30) of an actuator system is received by an actuator control system (40) including the artificial neural network (60), and The actuator control system (40) uses the artificial neural network (60) to determine a control signal (A) for the actuator (10).

2. The method according to claim 1, wherein: The possible expansions (61) of the current initial network (60) are then respectively trained with a training data set before selecting the one of the expansions from the plurality of possible expansions for use as the current initial network (60) in the next run through the method.

3. The method according to claim 2, wherein: The extensions of the initial network (60) are respectively trained using a gradient descent method, wherein a parameter (η) characterizing the learning rate of the gradient descent method is selected to decrease with increasing number of training epochs performed.

4. The method according to claim 3, wherein: The curve of the decrease of the parameter (η) characterizing the learning rate is characterized by means of a trigonometric function.

5. The method according to any one of claims 1 to 4, wherein: The predeterminable plurality of possible modules and / or connections comprises normalization modules and / or skip connections and / or modules containing non-linear functions, wherein the non-linear function is idempotent for at least one value of a parameter (p) characterizing the non-linear function.

6. The method according to claim 5, wherein: The plurality of possible predeterminable modules and / or connections include the normalization module and the skip connection and the module including the nonlinear function and the module including the linear function, and include a module for widening an existing linear layer by adding units and / or channels, and include a module for widening the linear layer by duplicating the linear layer.

7. A computer program product comprising a computer program (210), the computer program (210) being configured to implement the method according to any one of claims 1 to 6 when being executed on a processor.

8. A machine-readable storage medium (200) on which a computer program is stored, the computer program being configured to implement the method according to any one of claims 1 to 6 when being executed on a processor.

9. A system (140) comprising the machine-readable storage medium (200) according to claim 8.

Images