Implementation of large neural network architectures on embedded hardware

Abstract

Method (100) for operating a neural network architecture (2) on a hardware platform (1), wherein the neural network architecture (2) is organized into layers (21-23) and at least one layer (21-23) has multiple processing units (31-34) that receive the input (21a-23a) of layer (21-23) and contribute (31b-34b) to the output (21b-23b) of layer (21-23), wherein the method (100) comprises the following steps: • the input (21a-23a) of at least one layer (21-23), and / or the input (2a) of the neural network architecture (2) as a whole, is fed to a trained selection network (4) (110); • Using the output (4b) of the selection network (4), a selection (31*-34*) of processing units (31-34) is determined for processing the input (21a-23a) of layer (21-23) (120); • the selected processing units (31*-34*) of layer (21-23) are executed on the hardware platform (1) (130), so that the input (21a-23a) of layer (21-23) is processed into contributions (31b-34b) to the output (21b-23b) of layer (21-23); • from these contributions (31b-34b) the output (21b-23b) of layer (21-23) is formed (140).

Description

[0001] The present invention relates to the implementation of large neural network architectures, such as those used for radar data processing, on embedded hardware used, for example, in vehicles or other mobile applications. State of the art

[0002] The at least partially automated driving of vehicles or robots on company premises or even in public transport requires constant monitoring of the vehicle's or robot's surroundings. As part of this monitoring, image data, radar data, and other sensor data are recorded and evaluated using neural networks to determine the resulting sensor data for the vehicle's or robot's future behavior planning.

[0003] The neural networks used exhibit great generalization power. This power stems from the fact that, during training with training examples, a large number of parameters are optimized so that these training examples are processed well. The neural network is then able to process even measurement data from situations previously unseen during training to produce a meaningful result for the intended application.

[0004] The larger the neural network architecture, i.e., the more parameters characterize its behavior, the greater the generalization power. As the number of parameters increases, so does the demand for hardware resources to run the neural network architecture. Disclosure of the invention

[0005] The invention provides a method for operating a neural network architecture on a hardware platform. This neural network architecture is organized into layers. In particular, for example, a first layer can receive the input of the neural network architecture as a whole as its input. As processing continues, the outputs of one layer are then fed to the respective subsequent layer as inputs, until finally the output of the last layer is also the output of the neural network architecture as a whole.

[0006] Here, at least one layer has several processing units, each of which receives the layer's input and contributes to the layer's output. This means that the layer's input is equally available to all of these processing units. The processing units are also independent of each other in the sense that no processing unit is dependent on a precursor product from another processing unit.

[0007] Within the method, the input of at least one layer and / or the input of the neural network architecture as a whole is fed to a trained selection network. Using the output of this selection network, a selection of processing units is determined for processing the input of the layer.

[0008] The selected processing units of the layer are executed on the hardware platform, so that the layer's input is processed into contributions to the layer's output. These contributions form the layer's output.

[0009] It was recognized that in this way, the benefits of a multi-parameter neural network architecture can be decoupled from the need for hardware resources. That is, the neural network architecture retains its ability to generalize but can be executed with fewer hardware resources.

[0010] In particular, for given hardware resources, the computational time requirement can be kept constant regardless of the number of parameters. This allows us to know in advance how long it will take for the neural network architecture to process an input into an output. This is particularly important in vehicles, robots, and other mobile applications where a guaranteed response time must be guaranteed and, at the same time, embedded systems with very limited resources are used as hardware platforms.

[0011] The neural network architecture can solve any task with respect to the input. In particular, it can, for example, assign classification scores to the input with respect to one or more classes of a given classification. If the input comprises measurement data, the classes can, for example, represent types of objects whose presence the measurement data indicates. The neural network architecture can, in particular, be designed as an image classifier.

[0012] In a particularly advantageous embodiment, a convolutional layer with processing units that generate a contribution to the layer's output from the layer's input by applying one or more filters is selected as the layer. Each filter is used to recognize specific types of features in the layer's input. For example, if types of objects in images are to be classified, the more different filters are available, the greater the variety of features and ultimately also object types that can be recognized. However, when processing a given image, only a small portion of the total number of filters will be important, comparable to the human brain, in which certain areas that make up only a small portion of the total number of neurons are preferentially activated for a specific task.

[0013] This is somewhat comparable to a craftsman who, to complete a specific task, doesn't immediately dump all the tools in the workshop onto the table, requiring a correspondingly large table for this, but instead purposefully reaches for the box or drawer containing the tools that prima facie seem to be suitable for the task. If, for example, the task involves assembly with screws that all have a Phillips head, then neither Allen keys nor hexagon wrenches have any place on the table.

[0014] For example, if • the creation of a two-dimensional feature map between two convolutional layers of a convolutional neural network (CNN) requires 1000 floating point operations (FLOPS), • applying a filter requires the creation of 256 feature maps and • If there are a total of 512 filters, the total computational effort required is 256 * 512 * 1000 = 131,072,000 FLOPS, or 130 MFLOPS. However, if only the 100 most important feature maps for the 100 most important filters are evaluated, the effort shrinks to 10 MFLOPS. This is less than a tenth of the original effort. At the same time, the memory requirement is also reduced.

[0015] In another particularly advantageous embodiment, the selection network selects a representative portion of its input and processes it further to produce its output. This significantly reduces the additional effort required to process the selection network. Selecting the processing units requires only a cursory analysis of the input to the selection network to determine roughly what the input is about.

[0016] For example, the selection network can sample its input on a grid that ignores parts of that input. For example, if that input includes an image or a two-dimensional feature map, the selection network can evaluate the information from small patches arranged in a regular grid on the image or feature map.

[0017] In another particularly advantageous embodiment, the selection network assigns each processing unit a score value that represents the importance of this processing unit for generating the layer's output. A predetermined number N of processing units with the best score values ​​are then selected for processing the layer's input. In this way, the computing time requirement can be specified, and the neural network architecture can deliver an output with exactly the level of detail that is achievable with the given hardware resources and in the given time.

[0018] In particular, the specified number N of processing units can be varied depending on the load on the hardware platform and / or the position of the layer in the neural network architecture. For example, a control unit for a vehicle or robot can run additional applications in addition to the neural network architecture, so that depending on the operating situation, more or fewer resources are available for the neural network architecture. By adjusting the number N of selected processing units accordingly, it is possible to respond more effectively to such fluctuations.

[0019] Furthermore, the processing units are not equally important in all layers. For example, in a CNN, the feature maps become increasingly specialized the further back they are located in the neural network architecture, i.e., the closer they are to the output layer of that neural network architecture. Many of these feature maps contain only very small values ​​or even only zeros, or they are a copy of another feature map due to known redundancies. No additional computational effort needs to be invested in such feature maps.

[0020] In another particularly advantageous embodiment, measurement data recorded with at least one sensor is fed as input to the neural network architecture as a whole. A control signal is determined from the output of the neural network architecture as a whole. A vehicle, a driver assistance system, a robot, a quality control system, an area monitoring system, and / or a medical imaging system is controlled with the control signal. With the method proposed here, even with limited hardware resources on the hardware platform used, a control signal can ultimately be determined that causes the respective controlled technical system to react appropriately to the situation embodied in the measurement data.

[0021] In another particularly advantageous embodiment, a hardware platform is selected whose resources are insufficient to execute all processing units of the layer. Since only a small portion of the processing units need to be active simultaneously, a neural network architecture with a very large number of parameters and correspondingly many processing units can still run on this hardware platform. Especially in mobile applications, hardware platforms containing digital signal processors (DSPs) and central processing units (CPUs) from the microcontroller domain are often used. Such hardware platforms would normally be overwhelmed by the processing of radar data using a neural network architecture, for example.Common optimizations for the use of neural network architectures on such hardware platforms do not focus on the total number of instructions required to execute the neural network architecture, but instead focus on reducing the number of parameters in the neural network architecture and thus simplifying it. The method proposed here, in contrast, first considers the full complexity of the neural network architecture and calculates an approximation of the output that disregards precisely what is dispensable in the current situation.

[0022] In this respect, the method proposed here differs fundamentally from structural "pruning" methods, which, by whatever means, identify less important processing units in the neural network architecture once and then remove or deactivate these processing units once and for all. Instead, according to the method proposed here, one and the same processing unit can be unimportant for processing a specific input of the neural network architecture, while simultaneously being very important for processing another input. In the previously described analogy of the craftsman, the craftsman keeps the tool he doesn't need at the moment in reserve, instead of immediately throwing it in the trash, believing he no longer needs it.

[0023] In another particularly advantageous embodiment, a maximum of 20%, preferably a maximum of 15%, and most preferably a maximum of 10%, of the processing units are selected for processing the layer's input. For many applications, these proportions correspond to the proportion of processing units whose simultaneous execution prima facie makes sense.

[0024] The selection network is trainable, just like the actual neural network architecture whose execution it supports. The invention therefore further provides a method for training a combination of a neural network architecture and a selection network.

[0025] Within this process, a set of training examples is provided, which are to be evaluated with the neural network architecture with regard to a given task. The neural network architecture is operated using the method described above. The training examples are each processed into outputs with regard to the given task.

[0026] Using a given cost function, we evaluate how well the outputs perform with regard to the given task. If the training examples are labeled with target outputs related to the given task, a deviation of the actually delivered outputs from the target outputs can be evaluated during supervised training. If the training examples are at least partially unlabeled, the agreement of the actually delivered outputs with arbitrary consistency conditions can be evaluated during unsupervised or self-supervised training. For example, an autoencoder with an encoder-decoder arrangement is measured by whether a representation of the training example generated with the encoder in a latent space can be transformed back to the original training example using the decoder.

[0027] Second, the cost function also evaluates how many processing units process the input of at least one layer to contribute to the output of that layer. The fewer processing units required on average, the better the evaluation by the cost function.

[0028] For example, both contributions to the cost function can be added together, with one contribution then being weighted relative to the other by multiplying it by a factor. This factor can be optimized, for example, as a hyperparameter.

[0029] Both parameters that characterize the behavior of the neural network architecture and parameters that characterize the behavior of the selection network are now optimized with the goal of improving the evaluation by the cost function. The two sets of parameters can be trained simultaneously or alternately.

[0030] Through this combined training, the training of the neural network architecture, in particular, can adapt to the constraint that only a limited number (N) of processing units can be executed simultaneously. It's pointless to refine the interaction of, say, 15 processing units if, in the end, only 10 of them can run simultaneously. In this case, refinement only wastes time and may even ultimately lead to poorer results if, for example, after optimizing for 15 processing units, five are simply omitted to reach the required 10. Conversely, the selection network can learn to recognize and select the best and most relevant processing units for the given task from those in training.

[0031] In a particularly advantageous embodiment, parameter optimization involves backpropagating the evaluation by the cost function through the neural network architecture, or through the selection network, to changes in the respective parameters. In this way, those parameters are determined whose change is likely to result in the greatest improvement in the cost function. The processing units not selected according to the output of the selection network are now also involved in the backpropagation through the neural network architecture. This means that omitting the processing units not selected only has an effect when processing training examples in the forward direction. The neural network architecture is therefore operated in the forward direction in the same way as in later operational mode, while all processing units are used during backpropagation.In this way, training always takes place for those processing units that are not relevant to the current training example. After all, every training example contains knowledge that can be generalized.

[0032] The selection of those processing units of a layer of the neural network architecture that actually execute on the hardware platform can advantageously be integrated into the neural network architecture itself. This neural network architecture can then also be operated on a hardware platform that does not have sufficient resources to execute all processing units.

[0033] The invention therefore also relates to a neural network architecture with a plurality of neurons organized in multiple layers. At least one of these layers has multiple processing units that receive the input of the layer and contribute to the output of the layer. The neural network architecture further comprises at least one selection network. This selection network receives the input of the at least one layer and / or the input of the neural network architecture as a whole as input and is designed to activate or deactivate processing units of the at least one layer for execution on a hardware platform. In particular, it can be provided that only a maximum of a fixed number of processing units of the layer are activated simultaneously.

[0034] The methods can, in particular, be fully or partially computer-implemented. Therefore, the invention also relates to a computer program with machine-readable instructions that, when executed on one or more computers and / or compute instances, cause the computer(s) and / or compute instances to execute one of the described methods. In this sense, control units for vehicles and embedded systems for technical devices that are also capable of executing machine-readable instructions are also to be regarded as computers. Compute instances can, for example, be virtual machines, containers, or serverless execution environments, which can be provided in a cloud, in particular.

[0035] The invention also relates to a machine-readable data carrier and / or a downloadable product containing the computer program. A downloadable product is a digital product that can be transmitted over a data network, i.e., downloaded by a user of the data network, and which can be offered for immediate download, for example, in an online shop.

[0036] Furthermore, one or more computers and / or compute instances may be equipped with the computer program, the machine-readable data carrier or the download product.

[0037] Further measures improving the invention are presented in more detail below together with the description of the preferred embodiments of the invention with reference to figures. Examples of implementation

[0038] It shows: Fig. 1 embodiment of the method 100 for operating a neural network architecture 2 on a hardware platform 1; Fig. 2 Visualization of the processing process during the execution of the method 100; Fig. 3 Visualization of the execution of a selection 31*, 33* of processing units 31-34 on a hardware platform 1 with limited resources; Fig. 4 Embodiment of the method 200 for training a combination of a neural network architecture 2 and a selection network 4.

[0039] Fig. 1 is a schematic flow diagram of an embodiment of the method 100 for operating a combination of a neural network architecture 2 and a selection network 4 on a hardware platform 1. The neural network architecture 2 is organized in layers 21-23. At least one layer 21-23 has several processing units 31-34 that receive the input 21a-23a of layer 21-23 and provide a contribution 31b-34b to the output 21b-23b of layer 21-23. This is shown in Fig. 2 is shown in more detail.

[0040] According to block 105, a convolutional layer with processing units 31-34 that generate a contribution 31b-34b to the output 21b-23b of layer 21-23 from the input 21a-23a of layer 21-23 by applying one or more filters may be selected as layer 21-23.

[0041] According to block 106, measurement data recorded with at least one sensor 5 can be supplied to the neural network architecture 2 as a whole as input 2a.

[0042] According to block 107, a hardware platform 1 may be selected whose resources are insufficient to execute all processing units 31-34 of layers 21-23.

[0043] In step 110, the input 21a-23a of the at least one layer 21-23, and / or the input 2a of the neural network architecture 2 as a whole, is fed to a trained selection network 4.

[0044] According to block 111, the selection network 4 can select a representative portion 4a* from its input 4a. This representative portion 4a* can then be processed according to block 112 to produce the output 4b of the selection network 4.

[0045] According to block 111a, the selection network 4 may sample its input 4a in a grid that ignores parts of this input 4a.

[0046] According to block 113, the selection network 4 can assign a score value 31a-34a to each processing unit 31-34, which represents the importance of this processing unit 31-34 for the formation of the output 21b-23b of the layer 21-23.

[0047] In step 120, using the output 4b of the selection network 4, a selection 31*-34* of processing units 31-34 is determined for processing the input 21a-23a of the layer 21-23.

[0048] According to block 121, if score values ​​31a-34a for processing units 31-34 have been determined according to block 113, a predetermined number N of processing units 31-34 with the best score values ​​31a-34a can be selected for processing the input 21a-23a of layer 21-23.

[0049] According to block 122, at most 20%, preferably at most 15%, and most preferably at most 10%, of the processing units 31-34 can be selected for processing the input 21a-23a of layer 21-23.

[0050] In step 130, the selected processing units 31*-34* of layer 21-23 are executed on the hardware platform 1. This processes the input 21a-23a of layer 21-23 into contributions 31b-34b to the output 21b-23b of layer 21-23, respectively.

[0051] In step 140, the output 21b-23b of layer 21-23 is formed from the contributions 31b-34b provided by the selected processing units 31*-34*.

[0052] In the Fig. In the example shown in Figure 1, a control signal 6 is determined in step 150 from the output 2b of the neural network architecture 2 as a whole. In step 160, a vehicle 50, a driver assistance system 51, a robot 60, a quality control system 70, an area monitoring system 80, and / or a medical imaging system 90 is controlled with the control signal 6.

[0053] The processing of the input 2a of the neural network architecture 2 as a whole to the output 2b of this network architecture 2 as a whole is in Fig. 2 illustrates.

[0054] In the Fig. In the example shown in Figure 2, the neural network architecture 2 comprises three layers 21-23. The input 2a of the neural network architecture 2 as a whole is initially fed to the first layer 21. The output 21a of the first layer 21 is fed as input 22a to the second layer 22. The output 22a of the second layer 22 is fed as input 23a to the third layer 23. The output 23b of the third layer 23 is simultaneously the output 2b of the neural network architecture 2 as a whole.

[0055] For reasons of clarity, Fig. 2 shows the division into processing units 31-34 only for the first layer 21. Each of these processing units 31-34, when executed on the hardware platform 1, provides a contribution 31b-34b to the output 21b of layer 21. The selection network 4, which receives the input 2a of the neural network architecture 2 as input 4a, determines score values ​​31a-34a for the processing units 31-34 as outputs 4b. Based on these score values ​​31a-34a, the processing units that are actually to be executed on the hardware platform 1 are selected.

[0056] In the Fig. In the example shown in Figure 2, the processing units 31 and 33 are selected as processing units 31* and 33* to be executed and accordingly provide contributions 31b and 33b to the output 21b of layer 21. This output 21b is then further processed in layers 22 and 23 until finally the output 2b of the neural network architecture 2 as a whole is created.

[0057] In the Fig. In the example shown in Figure 2, the selection network 4 is drawn separately from the neural network architecture 2. As previously explained, the selection network 4 can also be integrated into the network architecture 2.

[0058] Fig. Figure 3 illustrates how hardware for executing processing units 31-34 can be saved through the previously described selection. Hardware platform 1 is illustrated here as a machine that has exactly two slots for executing processing units 31-34. However, layer 21 of neural network architecture 2 comprises four processing units 31-34. Before processing a specific input 21a with layer 21, a selection must be made of those processing units 31-34 that will be inserted as selected processing units (here: 31* and 33*) into the two available slots of hardware platform 1. Hardware platform 1 is then started and processes inputs 21a of layer 21 into outputs 21b, using the inserted selected processing units 31* and 33*.

[0059] Fig.4 is a schematic flow diagram of an embodiment of the method 200 for training a combination of a neural network architecture 2 and a selection network 4.

[0060] In step 210, a set of training examples 7 is provided which are to be evaluated with regard to a given task.

[0061] In step 220, the neural network architecture 2 is operated with the previously described method 100. This processes the training examples 7 and 2a into outputs 2b.

[0062] In step 230, a given cost function 8 is used to evaluate • how well the output 2b is in relation to the given task, and • how many processing units 31-34 process the input 21a-23a of at least one layer 21-23 into contributions 31b-34b to the output 21b-23b of this layer 21-23.

[0063] The result is a rating of 8a.

[0064] In step 240, parameters 2c, which characterize the behavior of neural network architecture 2, and parameters 4c, which characterize the behavior of selection network 4, are optimized with the goal of improving the evaluation 8a by cost function 8. This optimization can be continued until any termination criterion, for example, convergence or the number of iterations, is met. The fully optimized state of parameters 2c and 4c is denoted by the reference symbols 2c* and 4c*, respectively. These parameters 2c* and 4c* also define the fully trained state 2* and 4* of neural network architecture 2 and selection network 4, respectively.

[0065] According to block 241, optimizing the parameters 2c, 4c may include backpropagating the evaluation 8a by the cost function 8 through the neural network architecture 2, or through the selection network 4, to changes in the respective parameters 2c, 4c. Then, according to block 242, the processing units 31-34 not selected according to the output 4b of the selection network 4 may also participate in the backpropagation through the neural network architecture 2.

Claims

[1] Method (100) for operating a neural network architecture (2) on a hardware platform (1), wherein the neural network architecture (2) is organized in layers (21-23) and at least one layer (21-23) has a plurality of processing units (31-34) that receive the input (21a-23a) of the layer (21-23) and provide a contribution (31b-34b) to the output (21b-23b) of the layer (21-23), the method (100) comprising the following steps: • the input (21a-23a) of the at least one layer (21-23), and / or the input (2a) of the neural network architecture (2) as a whole, is fed (110) to a trained selection network (4); • using the output (4b) of the selection network (4), a selection (31*-34*) of processing units (31-34) for processing the input (21a-23a) of the layer (21-23) is determined (120); • the selected processing units (31*-34*) of the layer (21-23) are executed (130) on the hardware platform (1) so that the input (21a-23a) of the layer (21-23) is respectively processed into contributions (31b-34b) to the output (21b-23b) of the layer (21-23); • from these contributions (31b-34b) the output (21b-23b) of the layer (21-23) is formed (140). [2] Method (100) according to claim 1, wherein a convolutional layer having processing units (31-34) which generate a contribution (31b-34b) to the output (21b-23b) of the layer (21-23) from the input (21a-23a) of the layer (21-23) by applying one or more filters is selected as the layer (21-23) (105). [3] Method (100) according to one of claims 1 to 2, wherein the selection network (4) selects (111) a representative portion (4a*) from its input (4a) and processes (112) it to form its output (4b). [4] Method (100) according to claim 3, wherein the selection network (4) samples (111a) its input (4a) in a grid that ignores parts of this input (4a). [5] Method (100) according to one of claims 1 to 4, wherein • the selection network (4) assigns (113) to each processing unit (31-34) a score value (31a-34a) that represents the importance of this processing unit (31-34) for the formation of the output (21b-23b) of the layer (21-23); and • a predetermined number N of processing units (31-34) with the best score values (31a-34a) is selected (121) for processing the input (21a-23a) of the layer (21-23). [6] Method (100) according to claim 5, wherein the predetermined number N of processing units (31-34) is varied (114a) depending on the utilization of the hardware platform (1) and / or depending on the position of the layer (21-23) in the neural network architecture (2). [7] Method (100) according to one of claims 1 to 6, wherein • measurement data recorded with at least one sensor (5) are fed (106) as input (2a) to the neural network architecture (2) as a whole; • a control signal (6) is determined (150) from the output (2b) of the neural network architecture (2) as a whole; and • a vehicle (50), a driver assistance system (51), a robot (60), a quality control system (70), a system (80) for monitoring areas, and / or a medical imaging system (90), with which the control signal (6) is controlled (160). [8] Method (100) according to one of claims 1 to 7, wherein a hardware platform (1) is selected (107) whose resources are insufficient for the execution of all processing units (31-34) of the layer (21-23). [9] Method (100) according to one of claims 1 to 8, wherein at most 20%, preferably at most 15% and most preferably at most 10% of the processing units (31-34) are selected (122) for processing the input (21a-23a) of the layer (21-23). [10] Method (200) for training a combination of a neural network architecture (2) and a selection network (4) comprising the steps: • a set of training examples (7) is provided (210) which are to be evaluated with regard to a given task; • the neural network architecture (2) is operated (220) with the method (100) according to one of claims 1 to 9, wherein the training examples (7, 2a) are processed into outputs (2b); • with a given cost function (8) is evaluated (230), ◯ how well the outputs (2b) are in relation to the given task, and ◯ how many processing units (31-34) process the input (21a-23a) of at least one layer (21-23) into contributions (31b-34b) to the output (21b-23b) of this layer (21-23); and • Parameters (2c) that characterize the behavior of the neural network architecture (2) as well as parameters (4c) that characterize the behavior of the selection network (4) are optimized (240) to the goal of improving the evaluation (8a) by the cost function (8). [11] Method (200) according to claim 10, wherein • optimising the parameters (2c, 4c) involves backpropagating (241) the evaluation (8a) by the cost function (8) through the neural network architecture (2) or through the selection network (4) to changes in the respective parameters (2c, 4c) and • the processing units (31-34) not selected according to the output (4b) of the selection network (4) are also involved in the backpropagation through the neural network architecture (2) (242). [12] Neural network architecture (2) comprising a plurality of neurons organized in several layers (21-23), where • at least one layer (21-23) has a plurality of processing units (31-34) which receive the input (21a-23a) of the layer (21-23) and provide a contribution (31b-34b) to the output (21b-23b) of the layer (21-23), and • at least one selection network (4) is provided which receives the input (21a-23a) of the at least one layer (21-23) and / or the input (2a) of the neural network architecture (2) as a whole as input (4a) and is designed to activate or deactivate processing units (31-34) of the at least one layer (21-23) for execution on a hardware platform (1). [13] A computer program comprising machine-readable instructions which, when executed on one or more computers and / or compute instances, cause the computer(s) and / or compute instances to perform a method (100, 200) according to any one of claims 1 to 11. [14] A machine-readable data carrier containing the computer program according to claim 13. [15] One or more computers and / or compute instances equipped with the computer program according to claim 13 and / or with the machine-readable data carrier according to claim 14.

Images