Method for determining optimal architecture of artificial neural network
The method uses a directed graph to efficiently determine optimal neural network architectures by evaluating potential architectures based on assigned rewards, reducing resource consumption and noise susceptibility, and enabling adjustments without full re-computation.
Patent Information
- Application Number
- JP2024076790
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-10
- Filing Date
- 2024-05-09
- Publication Date
- 2024-11-21
AI Technical Summary
The development of optimal architectures for artificial neural networks is resource-intensive and time-consuming, particularly when multiple criteria need to be considered, and existing methods often require costly re-determination if settings change.
A method that utilizes a directed graph to represent possible architectures, assigns flows to edges, determines rewards and cost functions, and iteratively updates these to find an optimal architecture based on settings, without optimizing rewards directly, thus reducing resource consumption and susceptibility to noise.
This approach increases the probability of finding an actual optimal architecture, reduces resource requirements, and saves memory and processor capacity by minimizing iterations, while allowing for adjustments to new criteria without complete re-determination.
Smart Images

Figure 2024163091000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a method for determining the optimal architecture of an artificial neural network, in particular for determining the optimal architecture with respect to multiple criteria of an artificial neural network, which method makes it possible to save resources in determining the optimal architecture and at the same time to improve the accuracy of the determination of the optimal architecture. [Background technology]
[0002] Machine learning algorithms are based on the use of statistical methods to train data processing equipment to perform specific tasks without originally being explicitly programmed for this purpose. The aim of machine learning here is to build algorithms that can learn from data and make predictions. These algorithms create mathematical models that can, for example, classify data.
[0003] An example for this kind of machine learning algorithm is the artificial neural network. This kind of artificial neural network is directed to biological neurons, which learn the unknown system behavior from existing training data and subsequently make it possible to apply the learned system behavior to unknown input variables. The neural network here consists of layers with idealized neurons, which are connected to each other in different ways according to the topology of the network. The first layer, also called input layer, here captures and transmits the input values, the number of neurons in the input layer corresponding to the number of input signals to be processed. The last layer, also called output layer, has a number of neurons exactly the same as the number of output values to be provided. Between the input layer and the output layer, there is also at least one intermediate layer, often called hidden layer, where the number of intermediate layers and the number and / or type of neurons in these layers depend on the concrete task to be solved by the neural network.
[0004] However, the development of the architecture of an artificial neural network, i.e. the determination of the network's appearance or the number of layers in the network, as well as the number and / or type of neurons in each layer, is here usually very laborious and problematic, especially with regard to resource consumption. In order to optimize the development of the architecture, a Neural Architecture Search (NAS) has been developed, which automatically develops the optimal architecture for a particular problem problem. This NAS algorithm here first compiles an architecture for an artificial neural network from various modules and configurations, which is then trained on a training data set and the results obtained here are then evaluated with respect to performance. Based on this evaluation, a new architecture that seems more optimal in terms of performance is then found, which is then trained again on the training data and the results obtained here are then evaluated with respect to performance. These steps can be repeated here until no further improvements are obtained by changing the architecture, whereby gradient-based methods are usually used to find the more optimal architecture.
[0005] In particular, the performance of the artificial neural network here depends in particular on the selected architecture. However, a drawback here is that it is usually difficult to find a practically optimal architecture for an artificial neural network, and the determination of the optimal architecture here nevertheless usually involves a high expenditure of resources. Moreover, when multiple settings or conditions or criteria are to be taken into account when determining the optimal architecture, if the settings are changed or other criteria are to be taken into account, a completely new determination of the optimal architecture is usually required.
[0006] From DE 10 2019214 625 A1 a method for creating an artificial neural network is known, which method here comprises the steps of providing a number of different data sets, initializing a number of hyperparameters, training an artificial neural network, evaluating the trained artificial neural network, optimizing the hyperparameters depending on the evaluation and training anew an artificial neural network using the optimized hyperparameters. [Prior art documents] [Patent documents]
[0007] [Patent Document 1] DE 102019214625 A1 Summary of the Invention [Problem to be solved by the invention]
[0008] The problem on which the invention is based is therefore to provide an improved method for determining the optimal architecture for an artificial neural network. [Means for solving the problem]
[0009] This problem is solved by a method for determining an optimal architecture of an artificial neural network according to the features of claim 1.
[0010] The problem is also solved by a system for determining an optimal architecture of an artificial neural network by the features of claim 8.
[0011] Disclosure of the Invention According to one embodiment of the present invention, this problem is solved by a method for determining optimal architectures of an artificial neural network, which method comprises the following steps: providing a set of possible architectures of the artificial neural network; representing the set of possible architectures of the artificial neural network in a directed graph, the nodes of which represent a respective subset of one of the possible architectures, the start nodes of which represent the input layer, the end nodes of which represent a respective subset having an output layer, and the edges of the directed graph represent possible connections between the subsets; and assigning, for each of at least two configurations for determining an architecture, respectively, a flow to each edge of the directed graph and, based on the directed graph, a corresponding determining a strategy for finding an optimal architecture for a corresponding configuration, determining a corresponding optimal architecture by iteratively finding trajectories from a start node to a destination node based on the determined strategy, determining a reward for the found trajectory, determining a cost function for the found trajectory based on the reward found for the trajectory and the flows assigned to the edges along the trajectory, and finding an optimal architecture for the corresponding configuration by updating the flows assigned to the edges along the trajectory, respectively, based on the cost function, wherein the steps of finding the trajectory, determining the reward, determining the cost function, and updating the flows are repeated until the found trajectory satisfies a stopping criterion for the architecture search, the trajectory that satisfies the stopping criterion represents the optimal architecture.Furthermore, the method comprises the steps of finding an optimal architecture for the artificial neural network based on a respective optimal architecture for each of the at least two settings or on previously determined optimal architectures for all of the at least two settings, and providing an optimal architecture for the artificial neural network.
[0012] A set of possible architectures is understood here to mean a plurality of possible architectures of the artificial neural network or a corresponding search space.
[0013] A directed graph is also a graph with nodes and edges connecting each other, the edges being directed, i.e. edges that can only extend in one direction.
[0014] By each node of the directed graph representing a subset of the possible architectures, we mean that each node represents at least a subset of the possible architectures of the artificial neural network, and each node can represent other subsets, which subsets may be distributed on the individual nodes of the directed graph such that, taken as a whole, all possible architectures of the artificial neural network are included or represented in the directed graph, where each of these subsets comprises or characterizes at least one layer of the corresponding possible architecture.
[0015] A strategy for finding an optimal architecture based on a directed graph is further understood to mean a plan in which, or based on them, the individual nodes of the directed graph are selected based on corresponding settings or corresponding criteria in order to obtain a trajectory.
[0016] The trajectory is here particularly represented as a path connecting a start node and an end node.
[0017] Reward or payoff is further understood to mean the value of the improvement that can be determined by evaluation of the architecture represented by the corresponding trajectory and that is achievable by the corresponding architecture.
[0018] Cost function or loss is further understood to mean the loss or error between the expected reward for a given trajectory based on the flows assigned to the edges along the trajectory and the reward actually determined for the trajectory.
[0019] Additionally, a termination criterion for architecture search refers to a predefined criterion, where if the determined architecture or the architecture represented by the determined trajectory satisfies the termination criterion for the corresponding setting or the corresponding criterion, the step of searching for the corresponding optimal architecture is terminated.
[0020] The architecture is represented here by a determined trajectory, meaning that the architecture is formed by the corresponding combination of the subsets symbolized by the edges along the determined trajectory.
[0021] The method according to the invention therefore differs from known methods for determining optimal architectures of conventional artificial neural networks in that the rewards themselves are not optimized, but rather each potential architecture is examined or queried based on the rewards assigned to them. The method according to the invention also differs from known methods for determining optimal architectures of conventional artificial neural networks in that, for example, instead of gradients being estimated to determine a more optimal architecture, the flows or values assigned to the individual edges of a directed graph or the assignment between a subset of possible architectures of an artificial neural network are optimized and adapted to the actual given situation.
[0022] The fact that the rewards themselves are not optimized, but rather that potential architectures are each examined or queried based on the rewards assigned to them, has the advantage here that it allows for greater precision in finding the optimal architecture, and in particular also increases the probability of finding the actual optimal architecture.
[0023] Furthermore, the fact that gradients are not estimated, but rather flows or values assigned to the individual edges of a directed graph or an assignment between a subset of possible architectures of an artificial neural network, are optimized and adapted to the given actual situation has the advantage, for example, of being less susceptible to noise and, overall, of requiring fewer iterations to find the optimal architecture, thereby saving resources, e.g. memory capacity and / or processor capacity, required to find the optimal architecture.
[0024] The fact that the optimal architecture of the artificial neural network is also determined based on a pre-determined architecture that is optimal for the settings to be taken into account when determining the optimal settings, respectively, also has the advantage that a costly and resource-intensive complete re-determination of the optimal architecture of the artificial neural network is not required if the settings are to be weighted differently or if other criteria are to be taken into account.
[0025] Overall, there is thus presented an improved method for finding optimal architectures for artificial neural networks.
[0026] In one embodiment, determining an optimal architecture for the artificial neural network based on the respective optimal architectures for each of the at least two settings uses a weighted sum of the optimal architectures for the at least two settings.
[0027] The optimal architectures are added together, which means here that the rewards corresponding to the corresponding optimal architectures and the optimal flows corresponding to the corresponding optimal architectures are added together, and the optimal architecture of the artificial neural network is found based on the sum of the individual optimal flows.
[0028] Weighting is further understood to mean that the individual influencing variables of the mathematical model are evaluated, for example, with respect to their value. Weighted sum means in this case that the individual sums, i.e. the individual optimal flows, are weighted according to their value or importance.
[0029] Thus, the optimal architecture of the artificial neural network can be determined based on the individual optimal architectures for at least two settings in a straightforward manner that does not require costly and resource-intensive adaptation.
[0030] Here, the corresponding weighting may be based on the current hardware givens of the at least one target component.
[0031] The hardware given conditions of the at least one target component are furthermore understood to mean in particular information about the resources of the at least one target component available for the application of the artificial neural network, for example memory capacity and / or processor capacity.
[0032] Thus, the given conditions of the data processing installation to which the correspondingly trained artificial neural network will subsequently be applied are taken into account when determining the optimal architecture of the artificial neural network.
[0033] In one embodiment, the reward for the trajectory is further determined based on a hardware givens of at least one goal component.
[0034] Thereby, the given conditions of the data processing installation to which the correspondingly trained artificial neural network will subsequently be applied are likewise taken into account when determining the optimal architecture of the artificial neural network.
[0035] According to a further embodiment of the invention, there is also provided a method for training an artificial neural network, comprising the following steps: providing training data for training the artificial neural network; providing an optimal architecture for the artificial neural network, the optimal architecture of the artificial neural network being determined by the method for determining an optimal architecture of an artificial neural network as described above; and training the artificial neural network on the basis of the training data and the optimal architecture.
[0036] Thus, a method for training an artificial neural network based on an optimal architecture determined by an improved method for determining optimal architectures for artificial neural networks is presented. The fact that the rewards themselves are not optimized, but rather potential architectures are checked or queried based on the rewards assigned to them, respectively, has the advantage that it is possible here to increase the accuracy of the determination of the optimal architecture and in particular also to increase the probability of finding the actual optimal architecture. The fact that the flows or values assigned to the individual edges of the directed graph or the assignment between a subset of possible architectures of the artificial neural network, but rather than gradients are estimated, is also advantageous in that it is less susceptible to noise and, overall, fewer iterations are required to determine the optimal architecture, thereby saving resources, such as memory capacity and / or processor capacity, required to determine the optimal architecture. The fact that the optimal architecture of the artificial neural network is also determined based on previously determined architectures, which are optimal for the settings to be considered when determining the optimal settings, respectively, has the advantage that it is not necessary to completely re-determine the optimal architecture of the artificial neural network, which is costly and resource-intensive, if the settings are to be weighted differently or other criteria are to be taken into account.
[0037] The training data here may include sensor data.
[0038] A sensor, also called detector, (measured variable or measurement) transducer or (measurement) probe, is a technological component capable of capturing certain physical or chemical properties and / or material properties of its environment either qualitatively or quantitatively as a measured variable.
[0039] Thereby, a simple technique can be used to capture given situations outside the actual data processing facility in which the method is implemented and to take them into account when training the artificial neural network.
[0040] According to a further embodiment of the present invention, there is also provided a method for controlling a controllable system based on an artificial neural network, the method comprising the following steps: providing an artificial neural network trained to control a controllable system, the artificial neural network being trained by the method for training an artificial neural network as described above; and controlling the controllable system based on the provided artificial neural network.
[0041] The controllable system here may in particular be a robotic system, which may for example be an embedded system and / or a vehicle function of a motor vehicle.
[0042] Thus, a method for controlling a controllable system based on an artificial neural network is presented, where the artificial neural network is based on an optimal architecture determined by an improved method for determining optimal architectures for artificial neural networks. The fact that the rewards themselves are not optimized, but rather the potential architectures are checked or queried, respectively, based on the rewards assigned to them, has the advantage here that it is possible to increase the accuracy of the determination of the optimal architecture and in particular also to increase the probability of finding the actual optimal architecture. The fact that the flows or values assigned to the individual edges of the directed graph or the assignment between a subset of possible architectures of the artificial neural network are optimized and adapted to the actual given situation, instead of the gradients being estimated, also has the advantage that it is, for example, less sensitive to noise and, overall, fewer iterations are required to determine the optimal architecture, which makes it possible to save resources, for example memory capacity and / or processor capacity, required for determining the optimal architecture. The fact that the optimal architecture of the artificial neural network is also determined based on a pre-determined architecture that is optimal for the settings to be taken into account when determining the optimal settings, respectively, also has the advantage that the optimal architecture of the artificial neural network does not have to be completely re-determined, which would be costly and resource intensive, if the settings are to be weighted differently or other criteria are to be taken into account.
[0043] According to a further embodiment of the invention, a system for determining optimal architectures of an artificial neural network is also presented, the system comprising: a first providing unit configured to provide a set of possible architectures of the artificial neural network; and an imaging unit configured to map the set of possible architectures of the artificial neural network into a directed graph, the nodes of the directed graph representing a respective one of the possible architectures, a start node representing an input layer, a finish node of the directed graph representing a respective one of the subsets having an output layer, and the edges of the directed graph representing respective possible connections between the subsets; and, for each of at least two configurations for determining an architecture, respectively, assigning a flow to each edge of the directed graph, respectively, and based on the directed graph, mapping the corresponding architectures of the artificial neural network to the corresponding architectures of the artificial neural network. and at least one computing unit configured to determine a strategy for determining an optimal architecture for a corresponding configuration, to determine a corresponding optimal architecture by iteratively determining trajectories from a start node to an end node based on the determined strategy, to determine a reward for the determined trajectory, to determine a cost function for the determined trajectory based on the determined reward for the trajectory and the flows assigned to the edges along the trajectory, and to determine an optimal architecture for the corresponding configuration by respectively updating the flows assigned to the edges along the trajectory based on the cost function, wherein determining the trajectory, determining the reward, determining the cost function, and updating the flows are repeated until the determined trajectory satisfies a stopping criterion for the architecture search, and the trajectory that satisfies the stopping criterion represents the optimal architecture.Furthermore, the system comprises a second computing unit configured to determine an optimal architecture for the artificial neural network based on the respectively determined optimal architectures for each of the at least two settings, and a second providing unit configured to provide the optimal architecture for the artificial neural network.
[0044] Thus, an improved system for determining optimal architectures for artificial neural networks is presented. The fact that the rewards themselves are not optimized, but rather potential architectures are checked or queried, respectively, on the basis of the rewards assigned to these architectures, has the advantage here that it is possible to increase the accuracy of the determination of the optimal architecture and in particular also to increase the probability of finding the actual optimal architecture. The fact that the flows or values assigned to the individual edges of the directed graph or the assignment between a subset of possible architectures of the artificial neural network, rather than gradients being estimated, are optimized and adapted to the actual given situation, has the advantage, for example, that it is less susceptible to noise and, overall, fewer iterations are required to determine the optimal architecture, thereby saving resources, for example memory capacity and / or processor capacity, required for determining the optimal architecture. The fact that the optimal architecture of the artificial neural network is also determined on the basis of a pre-determined architecture, which is optimal for the settings to be taken into account when determining the optimal settings, has the advantage that it is not necessary to completely re-determine the optimal architecture of the artificial neural network, which is costly and resource-intensive, if the settings are to be weighted differently or other criteria are to be taken into account.
[0045] In one embodiment, the second calculation unit is configured to determine the optimal architecture of the artificial neural network by a weighted sum of the respective optimal architectures for each of the at least two settings. Thus, the optimal architecture of the artificial neural network can be determined based on the individual optimal architectures for the at least two settings in a simple manner that does not require costly and resource-intensive adaptation.
[0046] Here, the corresponding weighting can be based on the current hardware givens of at least one target component, so that the givens of the data processing equipment to which the correspondingly trained artificial neural network will subsequently be applied are taken into account when determining the optimal architecture of the artificial neural network.
[0047] In one embodiment, the first calculation unit is also configured to determine the reward for the trajectory based on hardware givens of the at least one target component, whereby givens of the data processing equipment to which the correspondingly trained artificial neural network is subsequently applied are also taken into account when determining the optimal architecture of the artificial neural network.
[0048] Furthermore, according to a further embodiment of the invention, there is also provided a system for training an artificial neural network, comprising a first providing unit configured to provide training data for training the artificial neural network, a second providing unit configured to provide an optimal architecture for the artificial neural network, the optimal architecture having been determined by the system for determining an optimal architecture for an artificial neural network as described above, and a training unit configured to train the artificial neural network on the basis of the training data and the optimal architecture.
[0049] Thus, a system for training an artificial neural network is presented, based on an optimal architecture determined by the improved system for determining optimal architectures for artificial neural networks. The fact that the rewards themselves are not optimized, but rather the potential architectures are checked or queried, respectively, based on the rewards assigned to them, has the advantage here of increasing the accuracy of the determination of the optimal architecture, and in particular also increasing the probability of finding the actual optimal architecture. The fact that the flows or values assigned to the individual edges of the directed graph or the assignment between a subset of possible architectures of the artificial neural network, rather than the gradients being estimated, are optimized and adapted to the actual given situation, also has the advantage, for example, of being less susceptible to noise and, overall, of requiring fewer iterations to determine the optimal architecture, which allows saving resources, for example memory capacity and / or processor capacity, required for determining the optimal architecture. The fact that the optimal architecture of the artificial neural network is also determined based on a pre-determined architecture that is optimal for the settings to be taken into account when determining the optimal settings, respectively, also has the advantage that a costly and resource-intensive complete re-determination of the optimal architecture of the artificial neural network is not required if the settings are to be weighted differently or if other criteria are to be taken into account.
[0050] The training data may again comprise sensor data, which allows a simple approach to capture given situations outside the actual data processing facility in which the method is performed and to take them into account when training the artificial neural network.
[0051] Furthermore, according to a further embodiment of the present invention, a system for controlling a controllable system based on an artificial neural network is also provided, comprising a providing unit configured to provide an artificial neural network trained to control the controllable system, the artificial neural network being trained by the system for training an artificial neural network as described above, and a control unit configured to control the controllable system based on the provided artificial neural network.
[0052] Thus, a system for controlling a controllable system based on an artificial neural network is presented, where the artificial neural network is based on an optimal architecture determined by an improved system for determining optimal architectures for artificial neural networks. The fact that the rewards themselves are not optimized, but rather the potential architectures are checked or queried, respectively, based on the rewards assigned to them, has the advantage here that it is possible to increase the accuracy of the determination of the optimal architecture and in particular also to increase the probability of finding the actual optimal architecture. The fact that the flows or values assigned to the individual edges of the directed graph or the assignment between a subset of possible architectures of the artificial neural network are optimized and adapted to the actual given situation, instead of the gradients being estimated, also has the advantage that it is less susceptible to noise and, overall, fewer iterations are required to determine the optimal architecture, which makes it possible to save resources, such as memory capacity and / or processor capacity, required for determining the optimal architecture. The fact that the optimal architecture of the artificial neural network is also determined based on a pre-determined architecture that is optimal for the settings to be taken into account when determining the optimal settings, respectively, also has the advantage that a costly and resource-intensive complete re-determination of the optimal architecture of the artificial neural network is not required if the settings are to be weighted differently or if other criteria are to be taken into account.
[0053] Furthermore, according to a further embodiment of the invention there is also provided a computer program comprising a program code for implementing the above-described method for determining an optimal architecture of an artificial neural network, when the computer program is run on a computer.
[0054] This computer program has the advantage here that it is adapted to execute an improved method for determining an optimal architecture for an artificial neural network. The fact that the rewards themselves are not optimized, but rather the potential architectures are checked or queried on the basis of the rewards assigned to them, respectively, has the advantage here that it is possible to increase the accuracy of the determination of the optimal architecture and in particular also to increase the probability of finding the actual optimal architecture. The fact that the flows or values assigned to the individual edges of the directed graph or the assignment between a subset of possible architectures of the artificial neural network, but rather than the gradients being estimated, are optimized and adapted to the actual given situation also has the advantage that it is for example less sensitive to noise and that, overall, fewer iterations are required to determine the optimal architecture, thereby making it possible to save resources, for example memory capacity and / or processor capacity, required for determining the optimal architecture. The fact that the optimal architecture of the artificial neural network is also determined on the basis of a previously determined architecture that is optimal for the settings to be taken into account when determining the optimal settings, respectively, also has the advantage that it is not necessary to completely re-determine the optimal architecture of the artificial neural network, which is costly and resource-intensive, if the settings are to be weighted differently or other criteria are to be taken into account.
[0055] In summary, according to the present invention, a method for determining an optimal architecture of an artificial neural network, in particular an optimal architecture of an artificial neural network with respect to multiple criteria, is presented, and it is determined that this method makes it possible to save resources in determining the optimal architecture, and at the same time to improve the accuracy of the determination of the optimal architecture.
[0056] The described embodiments and further developments can be combined with one another in any desired manner.
[0057] Further possible embodiments, developments and implementations of the invention also include combinations not explicitly mentioned of the features of the invention described above or below with reference to the examples.
[0058] The accompanying drawings are intended to facilitate a better understanding of embodiments of the present invention. These drawings illustrate embodiments and, together with the description, serve to explain the principles and concepts of the present invention.
[0059] Other embodiments and many of the noted advantages will become apparent from a consideration of the drawings, in which elements are not necessarily drawn to scale relative to one another. [Brief description of the drawings]
[0060] [Figure 1] 1 is a flow chart illustrating a method for determining an optimal architecture for an artificial neural network according to an embodiment of the present invention. [Diagram 2] FIG. 1 is a schematic block diagram illustrating a system for determining an optimal architecture of an artificial neural network according to an embodiment of the present invention.
[0061] In the drawing figures, like reference numbers, unless otherwise stated, indicate identical or functionally equivalent elements, components or parts. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0062] FIG. 1 shows a flow chart of a method 1 for finding the optimal architecture of an artificial neural network according to an embodiment of the invention.
[0063] Neural architecture search (NAS) is generally understood to mean methods for automatically developing optimal architectures of artificial neural networks for a predefined problem statement. It is a sub-area of automated machine learning that makes the costly manual design of artificial neural networks unnecessary.
[0064] The scalable neural architecture search method is here a gradient-based method, where a supergraph is formed from all possible architectures included in the search space for the artificial neural network, where each possible architecture is a subgraph of the supergraph. The nodes of the supergraph represent a respective subset of one of the possible architectures, where each node can in particular represent exactly one possible layer of the artificial neural network, where the start node represents the input layer of the artificial neural network, and the end node of the directed graph represents a respective subset of one of the possible architectures with the output layer, where the edges represent possible connections between the subsets, where each edge is assigned a parameter based on a strategy for the selection of the node. Furthermore, based on the supergraph, an attempt is made to find an architecture such that the reward or profit is maximized, where a gradient descent method is used to determine the optimal architecture for the artificial neural network. Also, in the case of multi-criteria tasks or neural networks where several tasks are to be satisfied, it is usually attempted to find the optimal architecture based on a common strategy in which all criteria are to be guaranteed.
[0065] FIG. 1 shows here a method 1, which comprises a step 2 of providing a set of possible architectures of an artificial neural network or a corresponding search space, a step 3 of representing the set of possible architectures of the artificial neural network in a directed graph, the nodes of the directed graph representing a respective one of a subset of the possible architectures, the start nodes of the directed graph representing the input layer, the end nodes of the directed graph representing a respective one of the subsets having an output layer, and the edges of the directed graph representing the possible connections between the subsets, and a step 4 of assigning, for each of at least two configurations for searching for architectures, respectively, a flow to each edge of the directed graph and determining, based on the directed graph, a strategy for searching for the corresponding optimal architecture. and 4. determining an optimal architecture for the corresponding configuration by iteratively determining trajectories from a start node to a destination node based on the established strategy, determining a reward for the determined trajectory, determining a cost function for the determined trajectory based on the determined reward for the trajectory and the flows assigned to the edges along the trajectory, and determining an optimal architecture for the corresponding configuration by updating the flows assigned to the edges along the trajectory, respectively, based on the cost function, wherein the steps of determining the trajectory, determining the reward, determining the cost function and updating the flows are repeated until the determined trajectory satisfies a stopping criterion for the architecture search, the trajectory that satisfies the stopping criterion represents the optimal architecture.Furthermore, the method 1 comprises a step 5 of determining an optimal architecture for the artificial neural network based on the respective optimal architecture for each of the at least two settings, and a step 6 of providing an optimal architecture for the artificial neural network.
[0066] The fact that the rewards themselves are not optimized, but rather that potential architectures are each examined or queried based on the rewards assigned to them, has the advantage here that it allows for greater precision in finding the optimal architecture, and in particular also increases the probability of finding the actual optimal architecture.
[0067] Furthermore, the fact that gradients are not estimated, but rather flows or values assigned to the individual edges of a directed graph or an assignment between a subset of possible architectures of an artificial neural network are optimized and adapted to the given actual situation has the advantage, for example, of being less susceptible to noise and, overall, of requiring fewer iterations to find the optimal architecture, thereby saving resources, e.g. memory capacity and / or processor capacity, required to find the optimal architecture.
[0068] The fact that the optimal architecture of the artificial neural network is also determined based on a pre-determined architecture that is optimal for the settings to be taken into account when determining the optimal settings, respectively, also has the advantage that a costly and resource-intensive complete re-determination of the optimal architecture of the artificial neural network is not required if the settings are to be weighted differently or if other criteria are to be taken into account.
[0069] Overall, there is thus presented an improved method 1 for finding optimal architectures for artificial neural networks.
[0070] In particular, Fig. 1 shows here a method 1 based on the application of a flow method instead of a gradient-based approach, in particular to find an optimal architecture for a neural network performing multiple tasks simultaneously, where a desired architecture can be found with respect to its performance with respect to the multiple tasks to be performed by the corresponding neural network.
[0071] According to the invention, the at least two settings include hardware metrics, in particular desired latency, energy consumption, and / or currently available resources. Additionally, the at least two settings may include a receiving task, for example detection accuracy or detection type.
[0072] The set of possible architectures, and thus the directed graph or supergraph, can here be based on labeled training data, e.g. labeled sensor data for training an artificial neural network.
[0073] According to the embodiment of Fig. 1, moreover, each node in the directed graph represents exactly one possible layer of the artificial neural network. Based on the illustrated method 1, for each of the at least two configurations, the respectively corresponding architecture can here in particular be built sequentially, i.e. each layer can be selected individually or each layer can be determined individually which one should be inserted at which point in time. For this purpose, the connections of the directed graph can in particular be predefined on a predefined set of actions corresponding to the selection of the individual edges of the directed graph.
[0074] The step of determining a reward for the determined trajectory can also be performed, for example, by training an architecture represented by each determined trajectory on labeled training data, in which case the obtained results are subsequently validated or evaluated with respect to their performance with respect to the corresponding settings.
[0075] The cost functions can also be determined, for example, by determining a flow matching objective, respectively. The cost functions can even be determined, for example, by determining a detailed balance objective and a backward policy or trajectory balance objective, respectively.
[0076] The step of updating the flows assigned to each edge along the trajectory based on the cost function may further include applying a backtracking algorithm.
[0077] Furthermore, the interruption criterion can be selected such that the determination of the corresponding optimal architecture is interrupted as soon as the reward determined for the determined trajectory lies within a corresponding pre-set target area for the reward, where the target area may be different for each of the at least two settings, for example depending on the importance of the corresponding setting.
[0078] The initial flow value may now also be chosen randomly.
[0079] Furthermore, a corresponding strategy for finding an optimal architecture based on a directed graph can be based on flow values.
[0080] According to the embodiment of FIG. 1, for each of the at least two settings, the strategy for determining a corresponding optimal architecture based on the directed graph for each node of the directed graph here indicates in particular the probability for the trajectory to be determined to pass through the corresponding node of the directed graph, where the probability is proportional to the flow assigned to the edges of the directed graph leading from a preselected node, respectively, to the corresponding node, where the trajectory is determined by selecting, respectively, the edge having the maximum probability and / or is determined proportional to the probability.
[0081] This strategy also provides that at any given time, there is an additional possibility that other edges may be followed, outside of a preset probability.
[0082] According to the embodiment of FIG. 1, the step 5 of determining an optimal architecture for the artificial neural network based on the respective optimal architecture for each of the at least two settings uses a weighted sum of the respective optimal architectures for each of the at least two settings or the determined optimal architectures for all of the at least two settings.
[0083] In particular, the optimal architecture of the artificial neural network is found here by the weighted sum of the corresponding individual rewards and the weighted sum of the corresponding individual flows.
[0084] The weighting here, according to the embodiment of FIG. 1, is further based on the current hardware givens of at least one target component.
[0085] For example, the weightings can be selected such that energy consumption is minimized as soon as the filling level of the energy store of at least one target component falls below a threshold value for the filling level.
[0086] In particular, it is possible here to vary the importance or desired impact of a setting by appropriate selection of the weightings.
[0087] According to the embodiment of Fig. 1, the reward for a trajectory is further determined based on the hardware givens of at least one target component, for example, the hardware requirements can be included in the determination of the performance of an artificial neural network trained on the architecture and training data represented by the trajectory, respectively, where the hardware characteristics can comprise a weighting factor, where the larger the weighting factor selected, the tighter the focus on the hardware requirements.
[0088] The optimal architecture determined by method 1 can subsequently be used to train a corresponding artificial neural network based on the corresponding labeled training data.
[0089] In particular, here an artificial neural network can be trained and subsequently used to control a controllable system, which can be, for example, an embedded system of a vehicle or a functional part of an autonomously driving vehicle, For example, the method can be used to determine an optimal architecture of a neural network that performs multiple tasks within the context of partially automated driving.
[0090] Moreover, here, artificial neural networks can even be trained to classify image data, in particular digital image data, based on low-level features, such as edges or pixel attributes, etc. Here, image processing algorithms can further be used to analyze the classification results centered on the corresponding low-level features.
[0091] FIG. 2 shows a schematic block diagram of a system 10 for determining an optimal architecture of an artificial neural network according to an embodiment of the present invention.
[0092] According to the embodiment of FIG. 2, the system 10 here comprises a first providing unit 11 configured to provide a set of possible architectures of the artificial neural network, an imaging unit 12 configured to map the set of possible architectures of the artificial neural network into a directed graph, the nodes of the directed graph representing a respective one of the possible architectures, a start node representing an input layer, a finish node of the directed graph representing a respective one of the subsets with an output layer and the edges of the directed graph representing possible connections between the subsets, and a unit for assigning, for each of at least two configurations for determining an architecture, respectively, a flow to each edge of the directed graph, determining, based on the directed graph, a strategy for determining the corresponding optimal architecture, and determining, based on the determined strategy, a trajectory from the start node to the finish node, thereby determining the corresponding optimal architecture. the computing unit 13 configured to determine an optimal architecture for a corresponding configuration by determining a trajectory, determining a reward for the determined trajectory, determining a cost function for the determined trajectory based on the determined reward for the trajectory and the flows assigned to the edges along the trajectory, and updating the flows assigned to the edges along the trajectory based on the cost function, respectively, wherein the determining of the trajectory, determining the reward, determining the cost function and updating the flows are repeated until the determined trajectory satisfies a stopping criterion for the architecture search, the trajectory satisfying the stopping criterion representing the optimal architecture; a second computing unit 14 configured to determine an optimal architecture for the artificial neural network based on the respective optimal architecture for each of the at least two settings; and a second providing unit 15 configured to provide the optimal architecture for the artificial neural network.
[0093] The first providing unit here may in particular be a receiver configured to receive corresponding data. The second providing unit may in particular be a transmitter configured to transmit corresponding data or information. The first providing unit and the second providing unit here may be integrated in a common transceiver.
[0094] The imaging unit, the first calculation unit and the second calculation unit may further each be implemented based on, for example, code stored in a memory and executable by a processor.
[0095] According to the embodiment of FIG. 2, the second calculation unit 14 is again configured to determine the optimal architecture of the artificial neural network by a weighted sum of the optimal architectures for the at least two settings.
[0096] The weighting may again be based on the current hardware givens of at least one component of interest.
[0097] According to the embodiment of FIG. 2, the first determination unit 13 is also configured to determine a reward for the trajectory based on a hardware given condition of the respective at least one goal component.
[0098] Moreover, the system 10 may be particularly adapted to execute the above-mentioned method for determining the optimal architecture of an artificial neural network.
Claims
1. 1. A method for determining an optimal architecture for an artificial neural network, comprising the steps of: The method (1) comprises the following steps: - a step (2) of providing a set of possible architectures of said artificial neural network; - a step (3) of mapping the set of possible architectures of said artificial neural network into a directed graph, the nodes of said directed graph representing a respective subset of one of the possible architectures, a start node of said directed graph representing an input layer, a finish node of said directed graph representing a respective subset having an output layer, and edges of said directed graph representing each possible connection between said subsets; - for each of at least two settings for which an architecture is to be found, respectively, assigning flows to each edge of the directed graph, determining a strategy for finding a corresponding optimal architecture based on the directed graph, determining a corresponding optimal architecture by iteratively finding trajectories from the start node to the end node based on the determined strategy, determining a reward for the found trajectory, determining a cost function for the found trajectory based on the reward found for the trajectory and the flows assigned to the edges along the trajectory, respectively, and updating the flows assigned to the edges along the trajectory based on the cost function (4), thereby finding an optimal architecture for the corresponding setting, wherein the steps of finding a trajectory, determining the reward, determining the cost function, and updating the flows are repeated until the found trajectory satisfies a stopping criterion for architecture search, and a trajectory that satisfies the stopping criterion represents an optimal architecture; - determining (5) an optimal architecture for said artificial neural network based on the respective optimal architecture for each of said at least two settings; - a step (6) of providing said optimal architecture of said artificial neural network; The method (1),
2. 2. The method of claim 1, wherein the step of determining (5) the optimal architecture of the artificial neural network based on a respective optimal architecture for each of the at least two settings uses a weighted sum of the respective optimal architectures for each of the at least two settings.
3. The method (1) of claim 2, wherein the corresponding weighting is based on a current hardware given condition of the at least one target component.
4. The method (1) according to any one of claims 1 to 3, wherein the rewards for the determined trajectories are determined based on hardware givens of at least one target component, respectively.
5. 1. A method for training an artificial neural network, comprising: The method comprises the following steps: - providing training data for training said artificial neural network; - providing an optimal architecture for said artificial neural network, said optimal architecture having been determined by a method for determining an optimal architecture of an artificial neural network according to any one of claims 1 to 4; - training said artificial neural network based on said training data and on said optimal architecture; A method comprising:
6. The method of claim 4 , wherein the training data comprises sensor data.
7. 1. A method for controlling a controllable system based on an artificial neural network, comprising: The method comprises the steps of: - providing an artificial neural network trained to control said controllable system, said artificial neural network having been trained by a method for training an artificial neural network according to claim 5 or 6; - controlling said controllable system based on said provided artificial neural network; A method comprising:
8. A system for determining an optimal architecture for an artificial neural network, the system (10) comprising: a first providing unit (11) configured to provide a set of possible architectures of said artificial neural network; an imaging unit (12) configured to map a set of possible architectures of said artificial neural network into a directed graph, the nodes of said directed graph representing a respective subset of one of the possible architectures, a start node representing an input layer, a finish node of said directed graph representing a respective subset having an output layer, and edges of said directed graph representing each possible connection between said subsets; at least one first calculation unit (13) configured to: for each of at least two settings for which the architecture is to be found, respectively, assign a flow to each edge of the directed graph, determine a strategy for finding a corresponding optimal architecture based on the directed graph, find a corresponding optimal architecture by iteratively finding a trajectory from the start node to the end node based on the determined strategy, determine a reward for the found trajectory, determine a cost function for the found trajectory based on the reward found for the trajectory and the flows assigned to the edges along the trajectory, respectively, and update the flows assigned to the edges along the trajectory based on the cost function, wherein the steps of finding a trajectory, determining a reward, determining a cost function, and updating the flows are repeated until the found trajectory satisfies an interruption criterion for architecture search, and a trajectory that satisfies the interruption criterion represents an optimal architecture; a second calculation unit (14) configured to determine an optimal architecture for the artificial neural network based on a respective optimal architecture for each of the at least two settings; a second providing unit (15) configured to provide said optimal architecture of said artificial neural network; A system (10) comprising:
9. 9. The system (10) according to claim 8, wherein the second calculation unit (14) is configured to determine the optimal architecture of the artificial neural network by a weighted sum of respective optimal architectures for each of the at least two settings.
10. 10. The system (10) of claim 9, wherein the corresponding weighting is based on a current hardware given condition of the at least one target component.
11. 11. The system (10) of claim 8, wherein each of the at least one first calculation unit (13) is configured to determine a reward for the trajectory based on a hardware given condition of at least one target component.
12. 1. A system for training an artificial neural network, comprising: The system is a first providing unit configured to provide training data for training said artificial neural network; a second providing unit adapted to provide an optimal architecture for said artificial neural network, said optimal architecture having been determined by a system for determining an optimal architecture for an artificial neural network according to any one of claims 8 to 11; and a training unit configured to train the artificial neural network based on the training data and the optimal architecture; A system comprising:
13. The system of claim 12 , wherein the training data includes sensor data.
14. 1. A system for controlling a controllable system based on an artificial neural network, comprising: The system is a providing unit configured to provide an artificial neural network trained to control the controllable system, the artificial neural network having been trained by a system for training an artificial neural network according to claim 13 or 14; and a control unit configured to control the controllable system based on the provided artificial neural network; A system comprising:
15. 5. A computer program comprising a program code for carrying out the method for finding the optimal architecture of an artificial neural network according to any one of claims 1 to 4, when the computer program is run on a computer.
Citation Information
Patent Citations
Method, apparatus and computer program for creating an artificial neural network
DE102019214625A1
