Method for specifying optimal architecture of artificial neural network
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- ROBERT BOSCH GMBH
- Filing Date
- 2023-07-10
- Publication Date
- 2026-05-22
AI Technical Summary
Identifying the optimal architecture for artificial neural networks is resource-intensive and challenging due to the complexity of determining the number and type of layers and neurons, often requiring high computational resources and being susceptible to noise.
A method using a directed graph to represent possible architectures, where nodes represent subsets of architectures, edges represent links, and rewards are evaluated to identify optimal architectures based on flow probabilities, optimizing the architecture search without gradient estimation.
This approach reduces resource consumption and susceptibility to noise, increasing the probability of finding an optimal architecture with fewer iterations, thus saving memory and processor capacity.
Smart Images
Figure 00000000_0000_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a method for identifying an optimal architecture of an artificial neural network, which can save resources in identifying the optimal architecture and at the same time increase the accuracy in identifying the optimal architecture. [Background technology]
[0002] Machine learning algorithms are based on using statistical methods to train a data processing system so that it can perform a specific task without originally programming the data processing system explicitly for that task. The goal of machine learning is to build algorithms that can learn from data and make predictions. Such algorithms create, for example, mathematical models that can classify data.
[0003] An example of such an algorithm of machine learning is an artificial neural network. Such an artificial neural network makes it possible to determine a strategy based on biological neurons, learn unknown system characteristics from existing training data, and subsequently apply the learned system characteristics to unknown input variables. A neural network is composed of layers containing idealized neurons, which are connected to each other in different ways according to the topology of the network. The first layer, also called the input layer, detects and transmits the input values, the number of neurons in the input layer corresponding to the number of input signals to be processed. The last layer, also called the output layer, contains a number of neurons equal to the number of output values to be provided. Furthermore, between the input layer and the output layer, at least one intermediate layer, often also called a hidden layer, is provided, the number of intermediate layers as well as the number and / or type of neurons in these layers depending on the concrete task to be solved by the neural network.
[0004] However, developing the architecture of an artificial neural network, i.e. determining the appearance of the network or the number of layers in the network, and the number and / or type of neurons in each layer, is usually very complex, especially in terms of resource consumption. In order to optimize the development of the architecture, Neural Architecture Search (NAS) has been developed, which automatically develops the optimal architecture for a particular problem. The NAS algorithm first constructs an architecture for an artificial neural network from different modules and configurations, which is then trained with a set of training data, and the results obtained are then evaluated in terms of performance. Based on this evaluation, a new architecture that is expected to be more optimal in terms of performance can then be identified, which is then again trained on the training data, and the results obtained are then again evaluated in terms of performance. These steps can be repeated until it is no longer possible to achieve improvements by changing the architecture, and usually gradient-based methods are used to identify the more optimal architecture.
[0005] The performance of an artificial neural network in particular depends in particular on the chosen architecture, but in this case it proves to be a drawback that it is usually difficult to identify an actual optimal architecture for an artificial neural network, which is nevertheless usually associated with high resource consumption.
[0006] From DE 10 2019 214 625 A1 a method for creating an artificial neural network is known, which method comprises providing a number of different data sets, initializing a number of hyperparameters, training an artificial neural network, evaluating the trained artificial neural network, optimizing the hyperparameters depending on the evaluation and retraining the artificial neural network using the optimized hyperparameters. [Prior art documents] [Patent documents]
[0007] [Patent Document 1] DE 102019214625 A1 Summary of the Invention [Problem to be solved by the invention]
[0008] The problem underlying the present invention is therefore to provide an improved method for identifying optimal architectures for artificial neural networks. [Means for solving the problem]
[0009] The above problem is solved by a method for identifying an optimal architecture of an artificial neural network according to the features of claim 1.
[0010] Furthermore, the above problem is also solved by a system for identifying an optimal architecture of an artificial neural network according to the features of claim 7.
[0011] Disclosure of the Invention According to one embodiment of the present invention, the above problem is solved by a method for identifying an optimal architecture of an artificial neural network, the method comprising the steps of providing a set of possible architectures of the artificial neural network, representing the set of possible architectures of the artificial neural network in a directed graph, each of which symbolically represents a subset of one of the possible architectures, each of which symbolically represents an input layer, each of which symbolically represents a subset including an output layer, and each of which symbolically represents a possible link between the subsets, associating for each edge of the directed graph a flow to the corresponding edge, and a step of defining a strategy for identifying an optimal architecture based on the directed graph. and identifying an optimal architecture for the artificial neural network, the optimal architecture for the artificial neural network being identified by repeating the steps of identifying a trajectory from a start node to a terminal node based on a prescribed strategy, determining a reward for the identified trajectory, determining a cost function for the identified trajectory based on the reward determined for the trajectory and on the flows associated with the edges along the trajectory, and updating each time the flows associated with the edges along the trajectory based on the cost function, wherein the steps of identifying a trajectory, determining the reward, determining the cost function, and updating the flows are repeated until the identified trajectory satisfies a stop criterion for the architecture search, and the trajectory that satisfies the stop criterion represents the optimal architecture.
[0012] A set of possible architectures in this document is understood to be a plurality of possible architectures of the artificial neural network or a corresponding search space.
[0013] Furthermore, a directed graph is a graph that includes nodes and edges connecting the individual nodes, where the edges are directed edges, i.e. edges that can be traversed in only one direction.
[0014] That each node of the directed graph symbolically represents a respective one subset of one of the multiple possible architectures means that each node symbolically represents a subset of at least one possible architecture of the multiple possible architectures of the artificial neural network, and each node can symbolically represent a respective one other subset, and these subsets can be distributed among the individual nodes of the directed graph such that all possible architectures of the artificial neural network as a whole are contained or represented in the directed graph, the subsets comprising or characterizing respectively at least one layer of the corresponding possible architecture.
[0015] Furthermore, a strategy for identifying an optimal architecture based on a directed graph is understood to be a plan or setting based on which individual nodes of the directed graph are selected to obtain a trajectory.
[0016] A trajectory specifically refers to a consistent path between a starting node and one of the terminal nodes.
[0017] Further, a reward is understood to be a value that can be determined by evaluating the architecture represented by the corresponding trajectory and the value of the improvement that can be achieved by the corresponding architecture.
[0018] Further, a cost function or loss is understood to be the loss or error between the reward for a specified trajectory that is predicted based on the flow associated with the edges along the trajectory and the actual reward determined for the trajectory.
[0019] Furthermore, a stopping criterion for the architecture search refers to a predefined criterion, whereby the identification of the optimal architecture is terminated if the identified architecture, or the architecture represented by the identified trajectory, meets this stopping criterion.
[0020] That the architecture is represented by a specified trajectory means that the architecture is formed by correspondingly linking subsets that are symbolically represented by nodes along the specified trajectory.
[0021] The method according to the invention therefore differs from known conventional methods for identifying optimal architectures of artificial neural networks in that the rewards themselves are not optimized, but rather each potential architecture is checked or tested on the basis of the rewards associated with them.Furthermore, the method according to the invention differs from known conventional methods for identifying optimal architectures of artificial neural networks in that, for example, gradients are not estimated to identify a more optimal architecture, but flows or values associated with individual edges of a directed graph or with correspondences between a subset of possible architectures of an artificial neural network are optimized and adapted to the actual situation.
[0022] The fact that the rewards themselves are not optimized, but rather each potential architecture is checked or tested based on the rewards associated with them, has the advantage that it can also increase the accuracy in identifying the optimal architecture and, in particular, the probability that an optimal architecture can actually be found.
[0023] Furthermore, the fact that gradients are not estimated but rather flows or values associated with individual edges of a directed graph or with correspondences between a subset of possible architectures of an artificial neural network are optimized and adapted to the actual situation has the advantage that it is e.g. less susceptible to noise and that fewer iterations overall are needed to identify the optimal architecture, thereby making it possible to save resources, e.g. memory capacity and / or processor capacity, required to identify the optimal architecture.
[0024] Overall, therefore, an improved method for identifying optimal architectures for artificial neural networks is presented.
[0025] A strategy for identifying optimal architectures based on directed graphs can be defined in the following way: for each node of the directed graph, a probability is indicated that the trajectory to be identified passes through the corresponding node of the directed graph, the probabilities being respectively proportional to the flows associated with the edges of the directed graph that extend to the corresponding node, and a trajectory is identified by selecting in each case the edge with the highest probability and / or the edge whose probability is proportional to the probability.
[0026] The probabilities are respectively proportional to the flow associated with the edge of the directed graph leading to the corresponding node, meaning that the higher the probability, the greater the flow associated with the edge of the directed graph leading to the corresponding node.
[0027] Furthermore, the edge with the highest probability is selected each time, which means that a trajectory is identified by selecting as part of the trajectory the edge with the highest probability value, or the edge with the corresponding highest flow, which starts or emanates from the node, respectively, along the trajectory.
[0028] Alternatively, for example, edges may be selected that are proportional to the probability.
[0029] Thus, the strategies may reproduce or be based on probability distributions, which allows the identification of optimal architectures and in particular the adaptation of flows to be easily performed by known functions in the context of artificial neural networks, without the need for resource-intensive and laborious adaptations.
[0030] It is only one preferred embodiment that the strategy indicates for each node of the directed graph the probability that the trajectory to be determined passes through the corresponding node of the directed graph, the probability being respectively proportional to the flow associated with the edge of the directed graph that leads to the corresponding node, and the trajectory is determined by selecting each time the edge with the highest probability. Thus, the strategy can additionally set, for example, that at certain times other edges can also be followed with deviations from a certain probability, which allows the method to converge more quickly, especially when the initial association of edges with flows is performed randomly.
[0031] In one embodiment, the reward for a trajectory is determined based on the hardware status of at least one goal component.
[0032] A target component is understood to be a server or a client to which a correspondingly trained artificial neural network is subsequently applied.
[0033] Furthermore, the hardware context of the at least one target component is understood to be information relating to the resources of the at least one target component, such as memory capacity and / or processor capacity, which are provided in particular for using the artificial neural network.
[0034] Therefore, the context of the data processing system to which the appropriately trained artificial neural network will subsequently be applied is taken into account when identifying the optimal architecture of the artificial neural network.
[0035] According to a further embodiment of the invention, there is also presented a method for training an artificial neural network, comprising the steps of providing training data for training the artificial neural network, providing an optimal architecture for the artificial neural network, the optimal architecture being identified by the above-mentioned method for identifying an optimal architecture for an artificial neural network, and training the artificial neural network on the basis of the training data and the optimal architecture.
[0036] Thus, a method for training an artificial neural network is presented, which is based on an optimal architecture identified by the improved method for identifying optimal architectures for artificial neural networks. The fact that the rewards themselves are not optimized, but that each potential architecture is checked or tested based on the rewards associated with them, has the advantage that the accuracy of identifying the optimal architecture and in particular the probability of actually being able to find an optimal architecture can also be increased. Furthermore, the fact that the flows or values associated with the individual edges of the directed graph or with the correspondence between a subset of possible architectures of the artificial neural network are optimized and adapted to the actual situation, rather than the gradients being estimated, has the advantage that it is for example less susceptible to noise and that fewer iterations are required overall to identify the optimal architecture, which allows to save resources, such as memory capacity and / or processor capacity, required to identify the optimal architecture.
[0037] The training data may include sensor data.
[0038] A sensor, also called detector, (measurand or measurement) recorder or (measurement) sensor, is a technological component capable of detecting certain physical or chemical properties and / or material characteristics of its surroundings, either qualitatively or quantitatively as a measurand.
[0039] Thus, conditions external to the native data processing system in which the method is implemented can be easily detected and taken into account when training the artificial neural network.
[0040] Furthermore, according to a further embodiment of the present invention, there is also presented a method for controlling a controllable system based on an artificial neural network, comprising the steps of providing an artificial neural network trained to control the controllable system, the artificial neural network being trained by the above-mentioned method for training an artificial neural network, and controlling the controllable system based on the provided artificial neural network.
[0041] The controllable system may in particular be a robotic system, which may for example be an embedded system and / or a function of a motor vehicle.
[0042] Thus, a method for controlling a controllable system based on an artificial neural network is presented, the artificial neural network being based on an optimal architecture identified by an improved method for identifying optimal architectures for artificial neural networks. The fact that the rewards themselves are not optimized, but that each potential architecture is checked or tested based on the rewards associated with them, has the advantage that the accuracy of identifying the optimal architecture and in particular the probability of actually being able to find an optimal architecture can also be increased. Furthermore, the fact that the flows or values associated with the individual edges of the directed graph or with the correspondence between a subset of possible architectures of the artificial neural network are optimized and adapted to the actual situation, rather than the gradients being estimated, has the advantage that it is for example less susceptible to noise and that fewer iterations are required overall to identify the optimal architecture, which allows to save resources, for example memory capacity and / or processor capacity, required to identify the optimal architecture.
[0043] Furthermore, according to a further embodiment of the invention, there is provided a system for identifying an optimal architecture of an artificial neural network, comprising: a providing unit configured to provide a set of a plurality of possible architectures of the artificial neural network; a mapping unit configured to map the set of a plurality of possible architectures of the artificial neural network onto a directed graph, the nodes of the directed graph symbolically representing a respective subset of one of the plurality of possible architectures, the start nodes symbolically representing the input layer, the end nodes of the directed graph symbolically representing a respective subset including the output layer, and the edges of the directed graph symbolically representing possible links between the respective subsets; a mapping unit configured to map, for each edge of the directed graph, a respective flow to the corresponding edge; and a mapping unit configured to identify an optimal architecture based on the directed graph. A system is also presented, comprising: a definition unit configured to define a strategy for the identification; and an identification unit configured to identify an optimal architecture of the artificial neural network by repeating the steps of: identifying a trajectory from a start node to a terminal node based on the defined strategy; determining a reward for the identified trajectory; determining a cost function for the identified trajectory based on the reward determined for the trajectory and the flows associated with the edges along the trajectory; and updating each time the flows associated with the edges along the trajectory based on the cost function, wherein the steps of identifying a trajectory, determining the reward, determining the cost function and updating the flows are repeated until the identified trajectory satisfies a stop criterion for the architecture search, and a trajectory satisfying the stop criterion represents the optimal architecture.
[0044] Thus, an improved system for identifying optimal architectures for artificial neural networks is presented. The fact that the rewards themselves are not optimized, but that each potential architecture is checked or tested on the basis of the rewards associated with them, has the advantage that the accuracy of identifying the optimal architecture and in particular the probability of actually being able to find the optimal architecture can also be increased. Furthermore, the fact that the flows or values associated with the individual edges of the directed graph or with the correspondence between a subset of possible architectures of the artificial neural network are optimized and adapted to the actual situation, rather than the gradients being estimated, has the advantage that it is for example less susceptible to noise and that fewer iterations are required overall to identify the optimal architecture, which allows to save resources, for example memory capacity and / or processor capacity, required to identify the optimal architecture.
[0045] The strategy for identifying an optimal architecture based on a directed graph may indicate for each node of the directed graph a probability that the trajectory to be identified passes through the corresponding node of the directed graph, the probability being respectively proportional to the flow associated with the edge of the directed graph that extends to the corresponding node, the identification unit being configured to identify the trajectory by selecting each time the edge with the highest probability. The strategy may thus reproduce or be based on a probability distribution, which allows the identification of the optimal architecture, and in particular the adaptation of the flows, to be easily performed by known functions in the context of artificial neural networks, without the need for a resource-intensive and laborious adaptation.
[0046] It is only one preferred embodiment that the strategy indicates for each node of the directed graph a probability that the trajectory to be identified passes through the corresponding node of the directed graph, the probabilities being respectively proportional to the flows associated with the edges of the directed graph that lead to the corresponding node, and the identification unit is configured to identify the trajectory by selecting each time the edge with the highest probability. Thus, the strategy can additionally set, for example, that at a given time, other edges can also be followed, deviating from a given probability, which allows the method to converge more quickly, especially when the initial association of edges with flows is performed randomly.
[0047] Moreover, in one embodiment, the identification unit is configured to determine the reward for the trajectory based on the hardware status of at least one target component, so that the status of the data processing system to which the correspondingly trained artificial neural network is subsequently applied is taken into account when identifying the optimal architecture of the artificial neural network.
[0048] Furthermore, according to a further embodiment of the invention, there is also presented a system for training an artificial neural network, comprising a first providing unit configured to provide training data for training the artificial neural network, a second providing unit configured to provide an optimal architecture for the artificial neural network, the optimal architecture being identified by the above-mentioned system for identifying an optimal architecture for an artificial neural network, and a training unit configured to train the artificial neural network on the basis of the training data and the optimal architecture.
[0049] Thus, a system for training an artificial neural network is presented, which is based on the optimal architecture identified by the improved system for identifying optimal architectures for artificial neural networks. The fact that the rewards themselves are not optimized, but that each potential architecture is checked or tested based on the rewards associated with them, has the advantage that the accuracy of identifying the optimal architecture and in particular the probability of actually being able to find the optimal architecture can also be increased. Furthermore, the fact that the flows or values associated with the individual edges of the directed graph or with the correspondence between a subset of possible architectures of the artificial neural network are optimized and adapted to the actual situation, rather than the gradients being estimated, has the advantage that it is for example less susceptible to noise and that fewer iterations are required overall to identify the optimal architecture, which allows to save resources, such as memory capacity and / or processor capacity, required to identify the optimal architecture.
[0050] Again, the training data may include sensor data, so that conditions external to the original data processing system in which the method is implemented can be easily detected and taken into account when training the artificial neural network.
[0051] Furthermore, according to a further embodiment of the present invention, there is also presented a system for controlling a controllable system based on an artificial neural network, comprising a providing unit configured to provide an artificial neural network trained to control the controllable system, the artificial neural network being trained by the above-mentioned system for training an artificial neural network, and a control unit configured to control the controllable system based on the provided artificial neural network.
[0052] Thus, a system for controlling a controllable system based on an artificial neural network is presented, the artificial neural network being based on an optimal architecture identified by an improved system for identifying optimal architectures for artificial neural networks. The fact that the rewards themselves are not optimized, but each potential architecture is checked or tested based on the rewards associated with them, has the advantage that the accuracy of identifying the optimal architecture and in particular the probability of actually being able to find an optimal architecture can also be increased. Furthermore, the fact that the flows or values associated with the individual edges of the directed graph or with the correspondence between a subset of possible architectures of the artificial neural network are optimized and adapted to the actual situation, rather than the gradients being estimated, has the advantage that it is for example less susceptible to noise and that fewer iterations are required overall to identify the optimal architecture, which allows to save resources, for example memory capacity and / or processor capacity, required to identify the optimal architecture.
[0053] Furthermore, according to a further embodiment of the invention, there is also provided a computer program comprising a program code for implementing the above-described method for identifying the optimal architecture of an artificial neural network, when the computer program is executed on a computer.
[0054] Furthermore, according to a further embodiment of the invention, there is also provided a computer readable data carrier comprising a program code of a computer program for implementing the above-mentioned method for identifying an optimal architecture of an artificial neural network, when the program code is executed on a computer.
[0055] The computer program and the computer readable data carrier each have the advantage that they are arranged to implement an improved method for identifying optimal architectures for artificial neural networks. The fact that the rewards themselves are not optimized, but that each potential architecture is checked or tested on the basis of the rewards associated with them, has the advantage that the accuracy of identifying the optimal architecture and in particular also the probability of being able to find an optimal architecture in practice can be increased. Furthermore, the fact that the flows or values associated with the individual edges of the directed graph or with the correspondences between a subset of possible architectures of the artificial neural network are optimized and adapted to the actual situation, rather than the gradients being estimated, has the advantage that it is for example less susceptible to noise and that fewer iterations are required overall to identify the optimal architecture, which allows to save resources, for example memory capacity and / or processor capacity, required to identify the optimal architecture.
[0056] In summary, it is observed that, according to the present invention, a method for identifying an optimal architecture of an artificial neural network is presented, which can save resources in identifying the optimal architecture, while at the same time increasing the accuracy in identifying the optimal architecture.
[0057] The described embodiments and developments can be combined with each other in any desired manner.
[0058] Further possible embodiments, developments and implementations of the invention also include combinations, not explicitly mentioned, of the features of the invention that are described above or below with reference to the examples.
[0059] The accompanying drawings are intended to provide a further understanding of embodiments of the present invention, which illustrate embodiments and, in conjunction with the specification, are used to explain the principles and concepts of the invention.
[0060] Other embodiments and many of the above advantages will become apparent with reference to the drawings, in which elements are not necessarily drawn to scale relative to one another. [Brief description of the drawings]
[0061] [Figure 1] 1 is a flowchart of a method for identifying an optimal architecture of an artificial neural network according to an embodiment of the present invention. [Diagram 2] FIG. 1 is a schematic block diagram of a system for identifying optimal architectures of artificial neural networks according to an embodiment of the present invention.
[0062] The same reference numbers in the various figures of the drawings, unless stated otherwise, refer to the same or functionally equivalent elements, components or parts. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0063] FIG. 1 shows a flow chart of a method for identifying the optimal architecture of an artificial neural network 1 according to an embodiment of the invention.
[0064] Neural Architecture Search (NAS) is generally understood to be a method for automatically developing optimal architectures of artificial neural networks for a given problem. NAS eliminates the need for complex manual design of artificial neural networks and is a sub-area of automated machine learning.
[0065] The scalable neural architecture search method is a gradient-based method. From all possible architectures for the artificial neural network contained in the search space, a supergraph is formed, each possible architecture being a subgraph of the supergraph. The nodes of the supergraph symbolically represent a respective subset of one possible architecture of the multiple possible architectures, each node can symbolically represent exactly one possible layer of the artificial neural network in particular, the start node symbolically represents the input layer of the artificial neural network, the terminal nodes of the directed graph symbolically represent a respective subset of one possible architecture of the multiple possible architectures, including the output layer, and the edges symbolically represent possible links between the subsets, each edge being associated with a parameter based on a strategy for selecting the nodes. Furthermore, an attempt is made to find an architecture based on the supergraph with the maximum reward or profit, and a gradient descent method is used to identify the optimal architecture for the artificial neural network.
[0066] FIG. 1 shows a method 1, which comprises a step 2 of providing a set of possible architectures of an artificial neural network or a corresponding search space; a step 3 of displaying the set of possible architectures of the artificial neural network in a directed graph, where the nodes of the directed graph symbolically represent a respective subset of one of the possible architectures, the start node symbolically represents the input layer, the end node of the directed graph symbolically represents a respective subset including the output layer, and the edges of the directed graph symbolically represent possible links between the subsets; a step 4 of associating for each edge of the directed graph a flow to the corresponding edge; a step 5 of defining a strategy for identifying an optimal architecture based on the directed graph; and a step 6 of identifying an optimal architecture of the artificial neural network, where the optimal architecture of the artificial neural network is identified by a trajectory from the start node to the end node based on the defined strategy in a step 7. and (c) a step 6 of identifying an optimal architecture by repeating the steps 7 of identifying a trajectory, 8 of determining a reward for the identified trajectory, 9 of determining a cost function for the identified trajectory based on the reward determined for the trajectory and the flows associated with the edges along the trajectory, and 10 of updating the flows associated with the edges along the trajectory each time based on the cost function, in which in step 11 it is checked whether the trajectory thus identified satisfies an interruption criterion for the architecture search, and if the trajectory thus identified does not satisfy the interruption criterion, the steps 7 of identifying a trajectory, 8 of determining a reward, 9 of determining a cost function and the steps of updating the flows are repeated until the identified trajectory satisfies the interruption criterion, and if in step 11 it is determined that the trajectory thus identified satisfies the interruption criterion, the trajectory satisfying this interruption criterion represents an optimal architecture, and in step 12 this optimal architecture is output and provided for training an artificial neural network.
[0067] The fact that the rewards themselves are not optimized, but rather each potential architecture is checked or tested based on the rewards associated with them, has the advantage that it can also increase the accuracy in identifying the optimal architecture and, in particular, the probability that an optimal architecture can actually be found.
[0068] Furthermore, the fact that gradients are not estimated but rather flows or values associated with individual edges of a directed graph or with correspondences between a subset of possible architectures of an artificial neural network are optimized and adapted to the actual situation has the advantage that it is e.g. less susceptible to noise and that fewer iterations overall are needed to identify the optimal architecture, thereby making it possible to save resources, e.g. memory capacity and / or processor capacity, required to identify the optimal architecture.
[0069] Overall, therefore, an improved method for identifying optimal architectures for artificial neural networks 1 is presented.
[0070] In particular, FIG. 1 shows Method 1, which is based on the use of flow methods instead of a gradient-based approach.
[0071] The collection of possible architectures, and thus also the directed graph or supergraph, can be based on labeled training data, e.g., labeled sensor data for training an artificial neural network.
[0072] Moreover, according to the embodiment of Fig. 1, each node in the directed graph symbolically represents exactly one possible layer of the artificial neural network. Based on the illustrated method 1, the architecture can be built in a particularly sequential manner, i.e. each layer can be selected individually, or each one can decide individually which layer should be inserted at what point in time. For this purpose, in particular, the links of the directed graph can be set for a predefined set of actions that correspond to the selection of each edge of the directed graph.
[0073] Further, step 8 of determining a reward for the identified trajectory can be performed, for example, again such that the architecture represented by the identified trajectory is trained based on the labeled training data, and the obtained results are subsequently validated or evaluated in terms of performance.
[0074] Further, the cost function in step 9 can be determined, for example, by determining a Flow Matching Objective. However, the cost function may also be determined, for example, by determining a Detailed Balance Objective and a Backward Policy or a Trajectory Balance Objective.
[0075] Furthermore, the step 10 of updating each time the flows associated with the edges along the trajectory based on the cost function may include using a backtracking algorithm.
[0076] Additionally, the interruption criteria may be selected such that method 1 proceeds to step 11 as soon as the reward determined for the identified trajectory lies within a predefined target range for the reward.
[0077] Additionally, the initial flow value can be selected randomly.
[0078] Furthermore, the strategy for identifying an optimal architecture based on a directed graph can be based on flow values.
[0079] According to the embodiment of FIG. 1, the strategy for identifying an optimal architecture based on a directed graph indicates for each node of the directed graph a probability that the trajectory to be identified passes through the corresponding node of the directed graph, this probability being respectively proportional to the flow associated with an edge of the directed graph extending from a preselected node to the corresponding node, and a trajectory is identified by selecting in each case the edge with the highest probability and / or the edge being proportional to the probability.
[0080] Additionally, the strategy provides that at a given time, other edges may be followed with a certain probability.
[0081] Furthermore, according to the embodiment of Fig. 1, the reward for a trajectory is also determined based on the hardware status of at least one target component. For example, the hardware requirements can be taken into account together when determining the performance of an artificial neural network trained on the architecture represented by each trajectory and on the training data, and the hardware characteristics can be given a weighting factor, the larger the weighting factor is selected, the more weight is placed on the hardware requirements.
[0082] The optimal architecture identified by Method 1 can then be used to train a corresponding artificial neural network based on the corresponding labeled training data.
[0083] In particular, artificial neural networks can be trained and subsequently used to control a controllable system, which may be, for example, an embedded system in an automobile or a function of an autonomous automobile.
[0084] However, artificial neural networks can also be trained to classify image data, in particular digital image data based on low-level features, such as edges or pixel attributes, and image processing algorithms can be used to analyze the classification results focusing on the corresponding low-level features.
[0085] FIG. 2 shows a schematic block diagram of a system for identifying an optimal architecture for an artificial neural network 20 according to an embodiment of the present invention.
[0086] According to the embodiment of FIG. 2, the system 20 comprises a providing unit 21 configured to provide a set of possible architectures of the artificial neural network, a mapping unit 22 configured to map the set of possible architectures of the artificial neural network onto a directed graph, the nodes of the directed graph symbolically representing a respective subset of one of the possible architectures, the start nodes symbolically representing the input layer, the end nodes of the directed graph symbolically representing a respective subset including the output layer, and the edges of the directed graph symbolically representing possible links between the subsets, a matching unit 23 configured to match, for each edge of the directed graph, a respective flow to the corresponding edge, and defining a strategy for identifying an optimal architecture based on the directed graph. and an identification unit 25 configured to identify an optimal architecture for the artificial neural network by repeating the steps of identifying a trajectory from a start node to a terminal node based on a prescribed strategy, determining a reward for the identified trajectory, determining a cost function for the identified trajectory based on the reward determined for the trajectory and the flows associated with the edges along the trajectory, and updating each time the flows associated with the edges along the trajectory based on the cost function, wherein the steps of identifying a trajectory, determining the reward, determining the cost function, and updating the flows are repeated until the identified trajectory satisfies a stop criterion for the architecture search, and the trajectory satisfying the stop criterion represents the optimal architecture.
[0087] The providing unit may in particular be a receiver configured to receive the corresponding data. Furthermore, the mapping unit, the matching unit, the defining unit and the identifying unit may each be realized based on a code executable by a processor, for example stored in a memory.
[0088] Again, the strategy for identifying an optimal architecture based on a directed graph indicates, for each node of the directed graph, a probability that the trajectory to be identified passes through the corresponding node of the directed graph, this probability being proportional to the flow associated with an edge of the directed graph that extends to the corresponding node, and the identification unit is configured to identify the trajectory by selecting in each case the edge with the highest probability.
[0089] According to the embodiment of FIG. 2, again the identification unit 25 is adapted to determine a reward for a trajectory based on a hardware condition of at least one target component.
[0090] Moreover, the system 20 can be configured to implement, among other things, the above-described method for identifying the optimal architecture of an artificial neural network.
Claims
1. A method for identifying the optimal architecture for an artificial neural network, Method (1) is, - Step (2) of providing a set of multiple possible architectures for the artificial neural network, Step (3) is to map the set of multiple possible architectures of the artificial neural network to a directed graph, wherein the nodes of the directed graph symbolically represent one subset of one of the multiple possible architectures, the start node symbolically represents the input layer, the end nodes of the directed graph symbolically represent one subset including the output layer, and the edges of the directed graph symbolically represent possible links between the subsets. - Step (4) of associating one flow to each edge of the directed graph with the corresponding edge, - Step (5) defines a strategy for identifying the optimal architecture based on the directed graph, - Step (6) to identify the optimal architecture of the artificial neural network, wherein the optimal architecture of the artificial neural network is identified by repeatedly performing the steps of: (7) identifying a trajectory from the start node to the end node based on a defined strategy; (8) determining a reward for the identified trajectory; (9) determining a cost function for the identified trajectory based on the reward determined for the trajectory and the flow associated with the edges along the trajectory; and (10) updating the flow associated with the edges along the trajectory each time based on the cost function, Includes, The steps of identifying the trajectory (7), determining the reward (8), determining the cost function (9), and updating the flow (10) are repeated until the identified trajectory satisfies the interruption criteria for architecture exploration. A method that satisfies the aforementioned interruption criteria represents the optimal architecture.
2. The strategy for identifying the optimal architecture based on the directed graph indicates, for each node of the directed graph, the probability that the trajectory to be identified passes through the corresponding node of the directed graph. The aforementioned probabilities are proportional to the flows associated with the edges in the directed graph that extend to the corresponding node, The trajectory is determined by selecting the edge with the highest probability and / or an edge proportional to that probability each time. The method according to claim 1.
3. The reward for the identified trajectory is determined based on the hardware status of at least one target component. The method according to claim 1.
4. A method for training artificial neural networks, - A step of providing training data for training the artificial neural network, - A step of providing an optimal architecture for the artificial neural network, wherein the optimal architecture is identified by a method for identifying an optimal architecture for an artificial neural network as described in any one of claims 1 to 3. - A step of training the artificial neural network based on the training data and the optimal architecture, A method that includes this.
5. The training data includes sensor data, The method according to claim 4.
6. A method for controlling a system that can be controlled based on an artificial neural network, - A step of providing an artificial neural network trained to control the controllable system, wherein the artificial neural network is trained by the method for training an artificial neural network described in claim 4. - The steps of controlling the controllable system based on the provided artificial neural network, A method that includes this.
7. A system for identifying the optimal architecture for artificial neural networks, A providing unit (21) configured to provide a set of multiple possible architectures of the artificial neural network, A mapping unit (22) configured to map the set of multiple possible architectures of the artificial neural network to a directed graph, wherein the nodes of the directed graph symbolically represent one subset of one of the multiple possible architectures, the start node symbolically represents the input layer, the end nodes of the directed graph symbolically represent one subset including the output layer, and the edges of the directed graph symbolically represent possible links between the subsets, A mapping unit (23) is configured to associate one flow with each corresponding edge of the directed graph, A defining unit (24) configured to define a strategy for identifying the optimal architecture based on the directed graph, A specific unit (25) is configured to identify the optimal architecture of the artificial neural network by repeatedly performing the steps of: identifying a trajectory from the start node to the end node based on a defined strategy; determining a reward for the identified trajectory; determining a cost function for the identified trajectory based on the reward determined for the trajectory and the flow associated with the edges along the trajectory; and updating the flow associated with the edges along the trajectory each time based on the cost function, wherein the steps of identifying the trajectory, determining the reward, determining the cost function, and updating the flow are repeated until the identified trajectory satisfies the interruption criteria for architecture search, and the trajectory that satisfies the interruption criteria represents the optimal architecture, and the specific unit (25) is configured to identify the optimal architecture, A system that includes this.
8. The strategy for identifying the optimal architecture based on the directed graph indicates, for each node of the directed graph, the probability that the trajectory to be identified passes through the corresponding node of the directed graph. The aforementioned probabilities are proportional to the flows associated with the edges in the directed graph that extend to the corresponding node, The specified unit (25) is configured to identify the trajectory by selecting the edge with the highest probability each time. The system according to claim 7.
9. The specified unit (25) is configured to determine the reward for the trajectory based on the hardware status of at least one target component. The system according to claim 7.
10. A system for training artificial neural networks, A first supply unit configured to provide training data for training the artificial neural network, A second providing unit configured to provide an optimal architecture for the artificial neural network, wherein the optimal architecture is identified by a system for identifying an optimal architecture for an artificial neural network as described in any one of claims 7 to 9, the second providing unit, A training unit configured to train the artificial neural network based on the training data and the optimal architecture, A system that includes this.
11. The training data includes sensor data, The system according to claim 10.
12. A system for controlling a system that is controllable based on an artificial neural network, A providing unit configured to provide an artificial neural network trained to control the controllable system, wherein the artificial neural network is trained by a system for training an artificial neural network as described in claim 10, A control unit configured to control the controllable system based on the provided artificial neural network, A system that includes this.
13. A computer program, when executed on a computer, comprising program code for carrying out a method for identifying the optimal architecture of an artificial neural network according to any one of claims 1 to 3.
14. A computer-readable data carrier comprising program code for a computer program that, when executed on a computer, carries out a method for identifying the optimal architecture of an artificial neural network according to any one of claims 1 to 3.