METHOD AND DEVICE FOR AUTOMATICALLY GENERATING AN ARTIFICIAL NEURAL NETWORK

Description

[0001] The invention relates to a method for automatically generating an artificial neural network, a computer program comprising instructions configured to carry out the method when executed on a computer, a machine-readable storage medium on which the computer program is stored, and a system, in particular comprising one or more computers, configured to carry out the method. State of the art

[0002] From "Reinforcement Learning for Architecture Search by Network Transformation", arXiv preprint arXiv:1707.04873v1, 2017, Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, Jun Wang is a method for automatically generating an architecture of an artificial neural network using reinforcement learning (English: reinforcement learning ), whereby the network is deepened or widened while maintaining its function.

[0003] RISTO MIIKKULAINEN ET AL: "Evolving Deep Neural Networks", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLINE LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, March 2, 2017 (2017-03-02), available online: htts: / / arxiv.org / pdf / 1703.00548.pdf, proposes an automated method, CoDeepNEAT, for optimizing deep learning architectures through evolution. By extending existing neuroevolution methods to include topology, components, and hyperparameters, this method achieves results comparable to human best designs on standard benchmarks in object recognition and language modeling. Ilya Loshchilov ET AL: "SGDR: Stochastic Gradient Descent with Warm Restarts", May 3, 2017 (2017-05-03), pages 1-16, arXiv.org, available online at https: / / arxiv.org / abs / 1608.03983, discloses a warm start technique for stochastic gradient descent to improve its performance at any time when training deep neural networks.

[0004] DE 20 2017 102238 U1 discloses an actuator control system for controlling an actuator. Advantage of the invention

[0005] The method according to claim 1 has the advantage that it enables a very efficient automated construction of a powerful enlarged artificial neural network.

[0006] Advantageous further training is the subject of dependent claims. Disclosure of the invention

[0007] In a first aspect, the invention relates to a method according to claim 1.

[0008] Modules are units of the neural network that determine a (possibly vector-valued) output variable based on a (possibly vector-valued) input variable. A module can be, for example, a layer of the artificial neural network or artificial neurons.

[0009] That is, "successive" means that the method comprises a plurality of passes, whereby in each pass possible further developments are generated starting from the current initial network and from this one, preferably the best, further development is then selected to serve as the current initial network in the next pass.

[0010] This means that a plurality of possible extensions are repeatedly generated, and from these, only one, preferably the best, extension is selected, which then serves as the basis for further extensions. In this way, the artificial neural network is gradually expanded. It has been found that such a greedy approach surprisingly leads to a very powerful artificial neural network in a short time, and does not lead to inhibiting secondary minima.

[0011] When adding the respective modules and / or connections, predeterminable parameters that characterize the respective modules and / or connections to be added are selected such that the output network to which the respective modules and / or connections to be added are added provides unchanged output variables for each possible input variable before and after the addition of the respective modules and / or connections to be added.

[0012] This means that the corresponding parameters of the modules and / or connections to be added are selected so that the initial network and the network expanded by adding the modules and / or connections are functionally equivalent. This allows the possible extensions to be trained much more quickly, accelerating the process.

[0013] In addition, it can be provided that the possible further developments of the current initial network are each trained with a training data set before one of the further developments is selected from the plurality of possible further developments to serve as the current initial network in the next run of the method.

[0014] The selection of one training course from the majority of possible courses can be based on a parameter that characterizes the respective performance capability of each of the majority of possible courses. This parameter can preferably be selected based on the initial parameters of the respective one of the majority of possible courses.

[0015] In a further development of this aspect, it can be provided that the further developments of the output network are each trained with a gradient descent method, wherein a parameter that characterizes a learning rate of the gradient descent method is chosen to decrease with increasing number of training epochs carried out.

[0016] This means that the variable characterizing the learning rate is chosen so that it decreases with increasing learning time. This has the advantage that the gradient descent method converges particularly quickly. It has been recognized that this method can be used to counteract fluctuations in an estimated gradient particularly reliably.

[0017] Here, the falling course of the parameter that characterizes the learning rate can be characterized by means of a trigonometric function.

[0018] This is a particularly simple way to characterize a decreasing learning rate that exhibits a preferred smooth progression. For example, the quantity characterizing the learning rate can be ∝ 1 + cos t T π choose, where t a training time (e.g. measured in training epochs) and T denotes a specifiable total time.

[0019] It can be provided that according to one of the preceding claims, wherein the predeterminable plurality of possible modules and / or connections comprises a standardization module and / or a jump connection (English: skip connection) and / or a module containing a non-linear function, wherein the non-linear function is idempotent for at least one value of a parameter characterizing its behavior. This condition of the non-linear function allows the characterizing parameter to be selected such that the non-linear function does not change the output variable if the idempotent function is already present in the output network immediately before or after the idempotent function.

[0020] The normalization module can, for example, be a layer normalization (English: layer normalization ) or a batch normalization (English: batch normalization ). It was recognized that these modules can be initialized efficiently and that by including them in the predefined majority of possible modules, particularly efficient, powerful artificial neural networks can be generated.

[0021] In particular, it can be provided that the predeterminable plurality of possible modules and / or connections includes the normalization module and the jump connection and the module containing the non-linear function and a module containing a linear function and a module that widens an existing linear layer by adding units or channels and a module that widens a linear layer by duplicating this linear layer.

[0022] It has been shown that artificial neural networks can be generated particularly efficiently using this predefined plurality of possible modules and / or connections.

[0023] In further aspects, the invention relates to a computer program according to claim 7.

[0024] In still further aspects, the invention relates to a machine-readable storage medium according to claim 8 and a system according to claim 9.

[0025] Embodiments of the invention are explained in more detail below with reference to the accompanying drawings. In the drawings: Figure 1 schematically illustrates a possible application of an artificial neural network generated according to the invention; Figure 2 schematically illustrates an exemplary learning system for generating an artificial neural network; Figure 3 shows a flowchart illustrating a possible sequence of a method for generating an artificial neural network. Description of the embodiments

[0026] Figure 1shows, by way of example, an actuator 10 in its environment 20 interacting with an actuator control system 40. Actuator 10 and environment 20 are collectively referred to below as the actuator system. A state of the actuator system is detected by a sensor 30, which can also be provided by a plurality of sensors. An output signal S from sensor 30 is transmitted to actuator control system 40. From this, actuator control system 40 determines a control signal A, which actuator 10 receives.

[0027] The actuator 10 can, for example, be a (semi-)autonomous robot, such as a (semi-)autonomous motor vehicle. The sensor 30 can, for example, be one or more video sensors and / or one or more radar sensors and / or one or more ultrasonic sensors and / or one or more position sensors (e.g., GPS). Alternatively or additionally, the sensor 30 can also comprise an information system that determines information about a state of the actuator system, such as a weather information system that determines a current or future state of the weather in the environment 20.

[0028] In another embodiment, the actuator 10 may be a manufacturing robot, and the sensor 30 may then be, for example, an optical sensor that detects properties of manufactured products of the manufacturing robot.

[0029] In a further embodiment, the actuator 10 can be an enabling system configured to enable or disable the activity of a device. The sensor 30 can, for example, be an optical sensor (e.g., for capturing image or video data) configured to detect a face. Depending on the control signal A, the actuator 10 determines an enabling signal, which can be used to enable the device depending on the value of the enabling signal. The device can, for example, be a physical or logical access control. Depending on the value of the control signal A, the access control can then provide for access to be granted or denied.

[0030] It is also possible that, instead of actuator 10, a medical diagnostic system is controlled, for example, which outputs a medical diagnosis depending on the control signal A. If the sensor 30 is an optical sensor that captures medical images, the control signal A can, for example, be a semantic segmentation of the images, possibly with annotations, in which areas that appear particularly relevant for a medical diagnosis are highlighted. In this case, reference numeral 40 denotes an agent control system.

[0031] The actuator control system 40 receives the sensor output signal S in an optional receiving unit 50, which converts the output signal S into a data signal x (alternatively, the output signal S can also be adopted directly as the data signal x). The data signal x can, for example, be an excerpt or further processing of the output signal S. The data signal x is fed to an artificial neural network 60 as an input signal. The structure of the artificial neural network is characterized by structure parameters q, which are stored in a structure parameter memory Q. The artificial neural network 60 is parameterized by predefinable parameters p, which are stored in a parameter memory P.

[0032] The data signal x is a two-dimensional image signal. Its pixels can be characterized, for example, by one (brightness coding) or three numerical values ​​(RGB coding).

[0033] The artificial neural network 60 determines a pixel-by-pixel semantic segmentation y as the output signal from the input signal x, here the image signal x. A semantic value is assigned to each region of the image signal x.

[0034] The output signal y, here: the semantic segmentation y, is transmitted to an output unit 80, which determines the control signal A therefrom. This means that the control signal A is determined depending on the semantic segmentation y. In other embodiments, it is also possible for the output unit 80 to accept the output signal y directly as the control signal A.

[0035] In one embodiment, the actuator control system 40 comprises a computer and a machine-readable storage medium (not shown) on which a computer program is stored that, when executed by the computer, causes the computer to execute the described functionalities of the actuator control system 40.

[0036] Figure 2 schematically shows an embodiment of a system 140 for generating the artificial neural network 60. A training data unit 150 determines suitable input variables x, which are fed to the artificial neural network. For example, the training data unit 150 accesses a computer-implemented database in which a set of training data is stored and, for example, randomly selects input variables x from the set of training data. Optionally, the training data unit 150 also determines desired output variables ys associated with the input variables x, which are fed to an evaluation unit 180.

[0037] The artificial neural network x is configured to determine corresponding output variables y from the input variables x supplied to it. These output variables y are fed to the evaluation unit 180.

[0038] A modification unit 160 determines new structure parameters q` and new parameters p', for example, with the Figure 3 illustrated method and feeds them to the structural parameter memory Q and the parameter memory P, where they replace the structural parameters q and the parameters p. Possible variations of the structural parameters q are stored in the modification unit 160. Each such possible variation of the structural parameters q corresponds to a possible combination of addable modules and connections to the artificial neural network 60.

[0039] According to one aspect of the invention, starting from an initial version of the artificial neural network 60, the output network, a possible further development 61 of the artificial neural network 60 is created by supplying the structural parameters q and the parameters p to the artificial neural network 60. Each of these further developments 61 can be supplied with corresponding input variables x to the artificial neural network 60, so that it then determines corresponding output variables y.

[0040] The evaluation unit 180 can, for example, use a cost function (loss function ) a parameter determine which characterizes the performance of the artificial neural network 60 or the further training 61.

[0041] With fixed structural parameters q, the artificial neural network 60 or the further developments 61 can be optimized by varying the parameters p. The resulting parameters characterizing the performance the training course 61 that is most effective can be selected.

[0042] The system 140 may, for example, comprise one computer or a plurality of computers and, in the exemplary embodiment, comprises a machine-readable storage medium 200 on which the computer program 210 is stored, which is configured to carry out the method according to the invention.

[0043] Figure 3 shows a flowchart of a possible procedure for generating an artificial neural network 60.

[0044] First, (1000) an output network 60 is determined (for example, by reading the characterizing structural parameters q from a memory area). Optionally, the parameters p assigned to it are trained.

[0045] Then (1100) a possible variation of the structural parameters q is selected using a (pseudo-)random number generator, for example, 8 or 16 different ones (of course, this number is freely selectable, but it has been shown that a number in the range 8 to 16 leads to particularly good results). This means that a plurality, for example, 8 or 16, of possible further developments 61 are generated in parallel, each resulting from the initial network 60 by adding a combination of modules and / or connections that include one or more of the following combinations:

[0046] A first possible combination is the addition of a linear layer before a predeterminable (and preferably randomly selected) reference layer of the output network 60. The predeterminable reference layer can also be the output of the output network 60. This means that if input variables ze are fed to the reference layer, which generates output variables therefrom, a linear layer is inserted before this layer, which receives the input variables ze, determines transformed input variables ze ' = a*ze +b (with freely selectable parameters a, b) therefrom, and feeds these transformed input variables z' e to the reference layer instead of the input variables ze. Preferably, b=0 and a=1 are selected.

[0047] A second possible combination is the addition of a nonlinear layer before the predeterminable reference layer. Here, an idempotent function h of the reference layer is identified (for example, it can be identified that the input variable ze is fed to the function relu(x) = max(0,x)). Parameters that characterize the idempotent function h to be added are preferably chosen to be equal to the corresponding parameters of the idempotent function present in the reference layer, to which the output variable of the idempotent function h to be added is fed. Alternatively, a nonlinear function can also be inserted without necessarily identifying an idempotent function in the reference layer. In this case, the function to be added can be chosen in such a way (e.g., as a convex linear combination) that it yields the identity for a specific parameter. That is,Instead of the input variables ze, transformed input variables ze ' = a*h(ze ) + (1-a)*ze are fed to the reference layer. Preferably, a=0 is initially chosen.

[0048] A third possible combination is the addition of individual neurons to the predefined reference layer. Here, the parameters that characterize the functional relationship between the input variables and an output variable of the reference layer are preferably chosen such that the functional dependence of the output variable of the reference layer on the input variable remains unchanged.

[0049] A fourth possible combination is the addition of individual connections to the predefined reference layer. For example, a connection can be added parallel to an existing connection. In this case, the parameters that characterize the functional relationship between the input variables and an output variable of the reference layer are preferably selected such that the functional dependence of the output variable of the reference layer on the input variable remains unchanged.

[0050] A fifth possible combination is the addition of a jump connection between two (preferably randomly chosen) predefined reference layers, i.e., from a predefined source layer to a predefined target layer. Such a jump connection can be chosen such that its output variable in the target layer is added to another output variable (in this case, the parameters of the jump connection are preferably chosen such that the weight of the jump connection is 0 and the weight of the added output variable is 1).

[0051] A sixth possibility is analogous to the fifth possibility with the difference that the jump connection is chosen such that its output variable is concatenated to another output variable in the target layer (in this case, weights that weight the concatenated components are preferably chosen equal to 0).

[0052] A seventh possibility is to add a normalization module behind a predeterminable (and preferably randomly chosen) reference layer of the output network 60. Depending on a statistical distribution of output variables zr of the reference layer, statistical parameters µ , s which can, for example, correspond to a mean value or an estimated standard distribution of the selected parameters. These parameters can, for example, be averaged over rows and / or columns of a layer, or also over the output variables of the reference layer over a stack of input variables x of the further development 61. An output variable yr of the normalization module can then be selected as y r = γ ⋅ z r − μ σ 2 + ε + β .

[0053] The parameter e can be set to a fixed small value for reasons of numerical stability, e.g. e = 10 -10< . The parameters c,bare preferably chosen as γ = σ 2 + ε β = μ .

[0054] Subsequently (1200) the generated possible further trainings 61 are trained. For this purpose, each of the generated possible further trainings 61 is supplied with input variables x in training epochs, and depending on the output variables y determined from the input variables x, the parameter characterizing the performance The parameters p are then successively varied iteratively, with p 0 = p , p t +1 = pt - the t ∇ p . Here, the t the parameter that characterizes the learning rate. It is preferably chosen to decrease with increasing iteration index t, e.g. proportional to ( 1 + cos t T π ) with a specified total time T. This iteration is repeated until the parameters p converge at an index tc. Then, the new parameter p' = p tc is chosen.

[0055] Then (1300) 61 predeterminable input variables are fed to the generated and trained possible further trainings, and the resulting parameter characterizing the performance for each of the possible training courses 61. It is also conceivable that a parameter determined during the training in step (1200) is used.

[0056] Finally (1400) is determined depending on the parameters the training 61 selected whose parameter is best, e.g. largest.

[0057] It can now be checked whether the selected training 61 provides a sufficiently significantly better indicator than the initial network 60. If this is not the case, the process is terminated and the selected further development 61 (alternatively the current initial network 60) results in the generated artificial neural network.

[0058] Otherwise, the selected further development 61 now replaces the current output network 60, i.e., the structural parameters q' assigned to the selected further development 61 and the determined parameters p' assigned to it replace the structural parameters q and the parameters p. The method then branches back to step 1100.

[0059] It should be noted that by generating the artificial neural network 60, a corresponding actuator or agent control system 40 can also be generated, in which the artificial neural network 60 is used. For example, all other components can be specified in an unchangeable manner, and only the contained artificial neural network 60 is generated.

[0060] The method can be implemented as a computer program, i.e. software, or in hardware or a mixture of hardware and software.

Claims

1. Computer-implemented method for automatically generating an artificial neural network (60) for the pixel-wise semantic segmentation of an image signal, comprising modules and connections linking these modules, wherein the artificial neural network generated in this way is configured to determine an associated output signal from an input signal, wherein the input signal is a two-dimensional image signal and wherein the output signal is a pixel-wise semantic segmentation of the input signal, wherein the method comprises the following steps: wherein successive modules and / or connections are added to a current starting network (60), wherein modules and / or connections to be added are randomly selected from a predefinable plurality of possible addable modules and connections, and wherein a plurality of possible developments (61) of the current starting network (60) are each generated by adding modules and / or connections to be added to the starting network, and wherein, when adding the respective modules and / or connections, predefinable parameters (p) that characterize the respective modules and / or connections to be added are selected such that the starting network (60) to which the respective modules and / or connections to be added are added delivers unchanged output variables (y) for each possible input variable (x) in each case before and after the addition of the respective modules and / or connections to be added, and wherein one of the developments (61) is then selected from the plurality of possible developments (61) to serve as the current starting network (60) in a next run of the method, wherein the current starting network (60) that serves as the starting point for a first-time addition of modules and / or connections is trained with an initial training dataset before the first-time addition of modules and / or connections, and wherein the artificial neural network is trained to control an actuator based on sensor data.

2. Method according to Claim 1, wherein the possible developments (61) of the current starting network (60) are each trained with a training dataset before one of the developments is then selected from the plurality of possible developments to serve as the current starting network (60) in the next run of the method.

3. Method according to Claim 2, wherein the developments of the starting network (60) are each trained with a gradient descent method, wherein a parameter (η), which characterizes a learning rate of the gradient descent method, is selected so as to fall as the number of training epochs that have been performed increases.

4. Method according to Claim 3, wherein the falling characteristic of the parameter (η), which characterizes the learning rate, is characterized by way of a trigonometric function.

5. Method according to one of the preceding claims, wherein the predefinable plurality of possible modules and / or connections comprises a normalization module and / or a skip connection and / or a module containing a non-linear function, wherein the non-linear function is idempotent for at least one value of a parameter (p) that characterizes its behavior.

6. Method according to Claim 5, wherein the predefinable plurality of possible modules and / or connections includes the normalization module and the skip connection and the module containing the non-linear function and a module containing a linear function and a module that broadens an existing linear layer by adding units or channels and a module that broadens a linear layer by duplicating this linear layer.

7. Computer program comprising instructions that, when executed by a computer, cause said computer to carry out the method according to one of Claims 1 to 6.

8. Machine-readable storage medium (200) on which the computer program according to Claim 7 is stored.

9. System (140) configured to carry out the method according to one of Claims 1 to 6.