Generating a simplified computer-implemented neural network

Abstract

To provide a method, a program, and a storage medium, for simplifying a computer-implemented neural network.SOLUTION: A method includes: a step for receiving a given neural network including a plurality of neural network substructures in a mode of two or more layers; a step for calculating an approximate neural network for the given neural network; a step for calculating, concerning the approximate neural network, a matrix depicting variance of loss functions obtained by pruning one out of the plurality of neural network substructures or the plurality of neural network substructures; and a step for pruning at least one neural network substructure from the plurality of neural network substructures based on the variance of the loss functions depicted in the matrix, and generating a simplified neural network.SELECTED DRAWING: None

Description

[Technical Field]

[0001] Technical Field The present disclosure relates to techniques for generating a simplified computer-implemented neural network from a given neural network, and for applying the simplified computer-implemented neural network. [Background technology]

[0002] background Computer-implemented neural networks are increasingly used in a variety of technical devices. In this case, neural networks for many technical devices may have a complex structure (e.g., with a large number of neurons, layers, and corresponding links). This may impose (excessively) high demands on the hardware required to apply the neural network. Therefore, it may be necessary to find a compromise between the performance of the computer-implemented neural network and its complexity.

[0003] To address the above-mentioned problems, so-called pruning methods have been developed. These pruning methods aim, on the one hand, to reduce the size of neural networks and simplify their overall structure, while, on the other hand, to maintain good overall performance of the simplified neural networks (or to minimize the overall performance loss as much as possible). Therefore, neural networks simplified using these methods can be used, for example, for small technical devices with limited hardware resources (e.g., power tools, gardening equipment, or household appliances). In other examples, it may be necessary to reduce the evaluation time of a computer-implemented neural network to ensure a sufficiently fast response of a technical device (e.g., an autonomous robot). For this reason, simplifying a computer-implemented neural network is advantageous. Summary of the Invention [Means for solving the problem]

[0004] Summary of the Invention The present invention relates to a method for generating a simplified computer-implemented neural network for a device. The method includes receiving a given neural network including a plurality of neural network substructures in the form of two or more layers. The method further includes calculating an approximation neural network for the given neural network and calculating a matrix for the approximation neural network that describes a change in a loss function due to pruning one or more neural network substructures from the plurality of neural network substructures. The matrix takes into account a change in the loss function due to simultaneously pruning at least two neural network substructures from the plurality of neural network substructures located within different layers of the given neural network. The method finally includes pruning at least one neural network substructure from the plurality of neural network substructures based on the change in the loss function described in the matrix to generate a simplified neural network.

[0005] The invention further includes the use of the simplified computer-implemented neural network in a technical device, and a computer-implemented neural network generated using the corresponding method.

[0006] The present technology aims to generate a simplified neural network of smaller size (e.g., with fewer neurons and / or links and / or layers) compared to a given original neural network. In this case, the generated simplified neural network should not suffer too much loss in overall performance or accuracy (ideally, no loss in overall performance or accuracy should occur). Such a simplified (pruned) computer-implemented neural network may be suitable for technical devices with relatively reasonable hardware resources (e.g., portable electrical equipment or equipment without a permanent network connection) or in technical environments where relatively fast calculation and evaluation speeds are useful (e.g., in at least partially autonomous vehicles). However, reduced complexity and / or increased calculation and evaluation speeds may also be useful in other technical environments (e.g., to reduce current consumption). [Brief explanation of the drawings]

[0007] [Figure 1a] 1 is a schematic diagram of the original trained neural network before and after pruning (simplification), with neurons and their links depicted as nodes (circles) or edges (arrows). [Figure 1b] 1 is a schematic diagram of a pruning vector δp that can be used to transform the Hessian matrix H into a Q matrix (a matrix that describes the change in the loss function due to pruning one or more neural network substructures from multiple neural network substructures). The pruning vector δp is here associated with node p of the neural network and describes the change in the current weights due to pruning one or more links of node p. [Figure 2a]FIG. 1 shows a schematic diagram of all calculated elements of the Q matrix (black elements correspond to larger values) and pruning using the "Lin Sum OBD" pruning method for a neural network created on the Boston House Price Dataset (i.e., the standard data set). Neural structures corresponding to elements of the Q matrix marked by lines are iteratively removed. [Figure 2b] Similar to Fig. 2a, this figure shows a schematic of similar elements of the Q matrix and their pruning using the "Iterative Full OBD" pruning method. Neural structures corresponding to elements of the Q matrix marked by white circles are iteratively removed. [Figure 2c] Similar to Figures 2a and 2b, this figure shows a schematic of similar elements of the Q matrix and their pruning using the "structured OBD" ("structured optimal brain damage") pruning method. Neural structures corresponding to elements of the Q matrix marked by white circles are iteratively removed. [Figure 3] Figure 2 compares the test loss for the pruning methods used in Figures 2a-2c based on different pruning ratio values. A pruning ratio of zero corresponds to the case where the given neural network is not pruned. In this example, the trained neural network is selected as the given neural network. DETAILED DESCRIPTION OF THE INVENTION

[0008] Detailed Description The present disclosure relates to a method for generating a simplified computer-implemented neural network from a given neural network by pruning its structure.

[0009] FIG. 1a illustrates (schematically) an exemplary neural network 1. A neural network can be formed from multiple neurons (an exemplary neuron 2 is highlighted in FIG. 1a), which form nodes of the neural network 1 and are connected to each other via edges 3. The neurons of the computer-implemented neural network of the present disclosure are arranged in layers (e.g., the third layer 4 of FIG. 1a includes three neurons). In this disclosure, the incoming edges or links to a neuron (or node) are considered to be part of the respective layer (i.e., the incoming link and the node are located in the same layer). A computer-implemented neural network can include two or more, three or more, or five or more layers. Neurons and their links can have different structures and can be represented as nodes or edges using a graph. In this disclosure, any subset of elements of a neural network is referred to as a "neural network substructure." In some examples, a neural network substructure may include one or more edges, one or more nodes, or a combination of one or more nodes and edges (e.g., a node and edges leading to and leaving the node). Networks different from the network illustrated in FIG. 1a may include additional elements (e.g., feedback or memory). Such elements may also be part of or part of a neural network substructure. In other examples, elements of a neural network may be parameters for describing the neural network (this aspect is described in more detail below).

[0010] The output of a given neuron j may depend on the applied inputs of one or more neurons i. In particular, we can first form a weighted sum of the applied inputs for neuron j, and then, for all neurons, calculate the θ from neuron j to neuron i. ij We can define weights of the form (θij (Note that θ = 0 may mean that neuron j has no link with neuron i.) Finally, the output of neuron j can be calculated after applying the activation functions defined for each neuron to the pre-computed sum. Thus, in some examples, the topology of the neural network and / or the weights θ for all neurons may be ij A neural network can be defined by specifying the weights of the neural network. These weights may therefore also be elements of the neural network substructure in the sense of the present disclosure. That is, a neural network structure may include one or more weights of the neural network (these weights may correspond to one or more edges in the graphical description of the neural network). According to the terminology explained above, all weights of links or edges entering a node or neuron of a given layer belong to this layer. Weights of edges or links originating from this given layer will belong to other layers.

[0011] A computer-implemented neural network can be created and trained (e.g., fully or partially trainable) for use in a particular device, process data (e.g., sensor data) generated by the device, and calculate output data related to, for example, monitoring and / or controlling the device. Thus, the device's characteristics or its response to a given event can be determined, ultimately, by the topology and weights θ of the neural network. ij In another example, the neural network can be "hidden" in the weights θ ij In the present disclosure, the weight θ ij is discussed as an example parameter, although the techniques described herein can be used with other parameters as well.

[0012] Some methods used in the prior art for pruning the network structure from a trained neural network are implemented as follows: After neural networks with different topologies are generated depending on the task, the weights θ ij can be selected accordingly. This selection of weights is called training or learning of the neural network. This step is performed on a computer system. During "supervised learning", multiple input data sets x k (e.g., sensor data) and a corresponding number of desired output data sets y k (e.g., the state of the technical device, or the state of the environment of the technical device, or the control variable) can be used (i.e., the input data set and the output data set each form one pair). k ,y k ), k=1,...,N is called the training dataset. Training a neural network can be formulated as an optimization problem in which, given an input x k The output generated by the neural network for

number

number

number

[0013] Minimize all weights θ ij As a result of this minimization, the trained weights

number

number

[0014] A trained neural network may have a complex topology and a large number of neurons and links, which may lead to undesirably high computational hardware requirements. Furthermore, as already mentioned above, this complex neural network must first be simplified by pruning so that it can be used on a corresponding device. The right side of FIG. 1a shows a simplified (pruned) neural network 1a. For example, several edges (links) have been pruned between the top layer and the layer below it, layer 5. Furthermore, in the penultimate layer, layer 6, one node has been pruned along with its associated edges.

[0015] This simplification is achieved by pruning one or more neural network substructures to reduce the loss function L D, respectively. Pruning the structure may include deleting one or more links (i.e., edges) between neurons and / or completely deleting one or more neurons along with their incoming and outgoing links. In other cases (or other diagrams), pruning may include deleting or zeroing one or more weights (which may be an alternative description of deleting one or more links or edges). In other cases, the neural network may include elements beyond the structure shown in FIG. 1a (e.g., feedback or memory). Such network substructures may also be pruned by the methods of the present disclosure.

[0016] In some examples, the change in the loss function due to pruning the structure is calculated by the weights θ ij Loss function L for D can be approximated to a given order by a Taylor expansion of, for example, an expansion of the form:

number

[0017] where δθ is the weight of a given neural network (e.g.,

number

number

number

number

number

number

number

[0018] Loss function L D The formula for the Taylor expansion of is shown only as an example, and other formulas can be adopted depending on the normalization selected for each vector (e.g., δθ and δθ T In the second term of the Taylor expansion shown above, we may assume a factor of 1 / 2.)

[0019] When multiple neural network substructures are pruned, the change in the loss function can be expressed as a sum over multiple elements of a matrix that describes the change in the loss function due to pruning one or more neural network substructures from the multiple neural network substructures. For example, the change in the loss function can be expressed in the following form: δL D (θ)≒1 / 2Σ beschn.pq Q pq where the matrix Q pq teeth,

number

[0020] The formula for the change in the loss function seems simple, but δL D There can be some difficulties in calculating (θ). First, the Hessian matrix H(θ) (and thus the matrix Q) pq ) is usually very large, P×P, where P denotes the total number of links in the neural network. For this reason, the Hessian matrix is ​​approximated by a diagonal (or nearly diagonal) matrix in some previously known methods in the prior art. However, this approximation ignores possible correlations between network structures in the sense defined above. Moreover, this can lead to (partially significant) inaccuracies when assessing which network substructures have an impact on the performance of the neural network. Second, δL D The calculation of (θ) involves a number of computational steps on a computer-implemented system, which depend on the number of training data sets N and the pruning vector δ p With the dimension P of 2 ) Furthermore, the number of computational steps is completely independent of the fact that the respective dimension S of the network substructure being considered for pruning may be much smaller than the total number of links in the neural network P. As a result, the above evaluation for complex structures with large P may be computationally difficult to perform. Third, an additional problem may arise in that training neural networks (especially relatively large and complex neural networks) may be computationally very expensive and therefore a time-consuming task. These problems can be addressed by the techniques of the present disclosure in some implementations.

[0021] The first step of the method for generating a simplified computer-implemented neural network of the present disclosure may include, for example, receiving a given neural network that includes multiple neural network substructures in the form of two or more layers. Each layer may be provided, for example, by multiple neurons and corresponding incoming links. In this case, the weighted output of a neuron located in one layer may be the input of another neuron that may be located in another layer. For example, neural network 1 in FIG. 1a includes four layers. For the given neural network, the current weights

number

[0022] training dataset (x k ,y k) can contain different kinds of data, and in each pair, the input data (x k ) and output data (y k ) are paired (k=1,...,N). For example, the input data and the output data may each be a scalar (e.g., a scalar measurement), a vector of any length (i.e., a length greater than or equal to 1), or a matrix. The input data may represent environmental influences or internal operating states of a technical device. In one example, the input data may include sensor data. Alternatively or additionally, the input data may include image data and / or audio data. The output data may be a device or environmental state, or a recognized event (e.g., a "battery almost empty" state or a "raining" state for an electrical device). In a further example, the output variable may be a control variable (e.g., for an actuator) or may otherwise identify a reaction of the device.

[0023] According to the present invention, first, an approximate neural network is calculated for a given neural network, which is calculated based on the current weights of the given neural network.

number

number

number

number

number

[0024] In a further step of the method according to the invention, the loss function δL is calculated for the approximate neural network already generated. D Matrix Q describing the change in (θ) pq As explained above, these changes can be caused by pruning one or more neural network substructures from the plurality of neural network substructures. pq can describe the change in the loss function due to the simultaneous pruning of at least two neural network substructures from multiple neural network substructures located within different layers of a given neural network (i.e., the matrix Q pq (The matrix Q can take into account the respective correlations between at least two network substructures.) Furthermore, in some instances, pq can consider the change in the loss function due to the simultaneous pruning of multiple network substructures located within one layer of a given neural network. In some examples, the matrix can describe the change in the loss function due to the simultaneous pruning of multiple pairs (e.g., all possible pairs) of at least two network substructures from multiple neural network substructures.

[0025] (rather than just based on the main diagonal as in some prior art methods) pqBased on the change in the loss function described in, for example, multiple (e.g., all) calculated elements of this matrix, it is possible to identify whether any neural network substructures make a small or even negligible contribution to the overall performance of the original given neural network (e.g., the overall performance does not degrade beyond a predetermined measure). For example, in this context, after pruning, the loss function δL D Only those substructures that do not cause an increase or excessive increase in (θ) can be classified as substructures to be pruned. Therefore, such classified substructures can be pruned from a given neural network to generate a simplified neural network for a device. The resulting simplified neural network can provide data for the device more quickly and / or require fewer hardware resources.

[0026] The above linearization is based on the resulting pruning δL D The change in (θ) is calculated by using a second-order Taylor series for the given neural network.

number

[0027] Further, the method may include, for example, calculating the gradient φ(X) of the given neural network for the current weights. In one example, the calculation of the gradient φ(X) may use a current matrix X, which may include N pairs of input data from the input data set. The gradient φ(X) may then be multiplied by a network substructure pruning vector (also referred to as a perturbation vector) δq to obtain a second vector φ of dimension N. q =φ(X)δ q where each pruning vector (δ p ) corresponds to the network substructure to be pruned. In this way, multiple second vectors φ are used to account for possible structural changes in a given neural network. q (i.e., S second vectors). The computational cost scales as O(N P) for these steps. In another example, the gradient φ(X) and the second vector φ q The calculation of can be performed using Jacobi-vector products, which can also be computationally more efficient than many prior art methods.

[0028] As already mentioned, in one example, a given neural network may be a trained neural network. In this case, the training data set (X) may consist of a number of pairs (X={(x i ,y i )|i≦n}), where each pair is formed by a set of inputs and a set of outputs of the technical device, and a given neural network can generate a set of outputs for each pair of inputs. Furthermore, the vector of first derivatives of the loss function is the trained weights

number

number

[0029] In another example, the given neural network is an undertrained neural network (the term "undertrained" can be understood in the sense explained above), and the vector of first derivatives of the loss function is the current weights

number

[0030] Using an imperfectly trained neural network to find the matrix Q pq Calculating φ may include three steps: First, as in the first example of the trained neural network, a second vector φ obtained from a plurality of second vectors for a plurality of pairs (e.g., each pair) of neural network substructures of the incompletely trained neural network is calculated. p and φ qIn this way, the (first) contributions to both the main and subdiagonal elements can be calculated. Furthermore, the matrix Q pq The second contribution to the main diagonal of δ is considered, which means that the current weights are subsequently assigned to multiple pruning vectors δ p and a plurality of first vectors δ p Finally, the two contributions are added together. This can be done, for example, by using the following form:

number

[0031] In this method, regardless of whether a trained neural network or an incompletely trained neural network is used as the given neural network, the matrix Q pq In some cases, the computational cost of 2 ) can be scaled by the matrix Q pq The total computational cost of the method according to the present invention is O(N S 2 )+O(N P). Therefore, the matrix Q pq Calculating it directly requires a significantly larger number of computational steps, O(N P 2 ), the method according to the invention can obtain computational cost advantages, especially for larger structures with S<<P.

[0032] In the next step of the method, in accordance with the above description, the matrix Q pq The information in is used to estimate which substructures can be pruned.

[0033] In the first method, we first calculate the matrix Q pqAll components in each column of can be summed to obtain multiple sums. Then, taking the calculated sums into account, multiple neural network substructures are iteratively pruned. This can be done, for example, starting with the neural network substructure for which the smallest sum of the multiple sums was calculated. These neural network substructures can be pruned first. The results of this so-called "Lin Sum OBD" ("optimal brain damage") pruning method are exemplarily shown in FIG. 2a for a neural network created for the Boston housing price dataset (i.e., a standard dataset). Four exemplary columns 7a-7d are shown in FIG. 2a, where the sums are represented by point density. The rightmost column 7d has the smallest sum. Therefore, the corresponding neural network substructure can be pruned first.

[0034] In an alternative method, we first calculate the matrix Q pq The neural network substructure corresponding to the minimum amount of main diagonal elements of is removed. Then, multiple neural network substructures are iteratively removed, with each subsequent neural network substructure to be removed corresponding to the minimum amount of sums from the main diagonal elements and off-diagonal elements of the matrix associated with the previously removed neural network substructure. Figure 2b shows the application of this "Iterative Full OBD" pruning method to the same neural network as Figure 2a. Exemplary diagonal and off-diagonal elements are represented by circles, with the size of the circle being inversely proportional to the size of the main diagonal element / the sum of the main diagonal and off-diagonal elements.

[0035] In a further alternative method, we first create a matrix (Q pq ), optionally with the matrix (Q pq ) can be removed. In one example, we first remove the neural network substructures corresponding to the smallest main diagonal elements of the matrix Q pqThe calculated main diagonal elements of can be arranged in ascending order. The neural network substructure corresponding to the smallest main diagonal element is then removed. The method according to the present invention then involves iteratively removing multiple neural network substructures, each subsequent neural network substructure to be removed corresponding to the next largest value of the arranged main diagonal element. Figure 2c shows an example of the result of this "structured OBD" ("structured optimal brain damage") pruning method for the same neural network as Figures 2a and 2b. Exemplary diagonal elements are represented by circles, the size of which is inversely proportional to the size of the main diagonal element.

[0036] In Figure 3, the test losses for the pruning methods used in Figures 2a-2c are compared based on different pruning ratio values. A pruning ratio of zero corresponds to the case where the given neural network is not pruned. In this example, the trained neural network is selected as the given neural network. As this figure shows, the overall performance of the pruned network in the case of the "Iterative Full OBD" method or the "structured OBD" method remains substantially unaffected for pruning ratio values ​​less than 0.7 or 0.4.

[0037] Generally, all three pruning methods are performed until the size of the simplified neural network falls below a desired size. In one example, the desired size may be given by a minimum number of neurons in the simplified neural network or in one layer of the simplified neural network. In another example, the desired size may be defined by a minimum number of links between neurons in the simplified neural network or by a minimum number of links between neurons in one layer of the simplified neural network. The desired size may also be given, for example, as a minimum number of unpruned layers or structures of the simplified neural network. In another example, the pruning method is performed until the overall performance of the simplified neural network falls below a predefined threshold. For example, the overall performance can be estimated using a test loss (see, e.g., FIG. 3), and the test loss itself can be calculated based on a loss function. In one example, the predefined threshold can be defined as the ratio between the overall performance of the simplified neural network and the overall performance of the given neural network. In another example, the predefined threshold may correspond to a selected number.

[0038] As already mentioned, the simplified computer-implemented neural network of the present disclosure can be used in a variety of technical applications. In general, the present disclosure also relates to a method for pruning a computer-implemented neural network and a subsequent method for using the computer-implemented neural network in a technical device. This use may include, for example, open-loop control (or closed-loop control) of the technical device by means of the simplified neural network, recognizing an operating state (e.g., a malfunction) of the technical device or a state of the environment of the technical device by means of the simplified neural network, or evaluating an operating state of the technical device or a state of the environment of the technical device by means of the simplified neural network. In this case, the input data may include state data (e.g., at least partially sensor data) related to the internal state of the device. Additionally or alternatively, the input data may include state data (e.g., at least partially sensor data) related to the environment of the device. The output data of the simplified neural network may characterize the operating state or other internal state of the device (e.g., whether an error or critical operating state exists). The output data may be used to control the device in response to the characterized operating state or other internal state. Alternatively or additionally, the output data may include control data for the device.

[0039] In some cases, the device may be an electrical appliance (e.g., a tool, household appliance, or garden appliance). In other examples, the device may be a device in the form of the Internet of Things. Alternatively or additionally, the device may be a battery-operated device. As noted above, simplified computer-implemented neural networks are advantageous in these environments because they can be constructed in a relatively resource-efficient manner.

[0040] The simplified computer-implemented neural network can be used for classification of time series, and in particular for classification of image data (i.e., the device is an image classifier). The image data can be, for example, camera image data, lidar image data, radar image data, ultrasound image data, or thermal image data (e.g., generated by a corresponding sensor). The image data can include individual images or video data. In some examples, the computer-implemented neural network can be configured for or used in a monitoring device (e.g., for manufacturing processes and / or quality assurance) or a medical imaging system (e.g., for retrieving diagnostic data). The image classifier can be configured to receive and classify image data into multiple classes. In some examples, the method includes a one-dimensional (R n ) input vector, and a two-dimensional (R m The image classification method may include mapping the input image data to output data in the form of an output vector (e.g., a vector of a plurality of input images). For example, the components of the input vector may represent a plurality of received image data. Each component of the output vector may represent the result of an image classification computed based on a simplified computer-implemented neural network. In some examples, the image classification may include semantic segmentation of the image (e.g., region-wise and pixel-wise classification of the image). The image classification may be, for example, object classification. For example, the presence of one or more objects in the image data may be detected (e.g., in a driver assistance system for automatically recognizing traffic signs or lanes).

[0041] In another example (or additionally), the computer-implemented neural network can be configured or used to monitor the operating state and / or environment of an at least partially autonomous robot. The at least partially autonomous robot can be an at least partially autonomous vehicle (or other at least partially autonomous mobile or transport means). In another example, the at least partially autonomous robot can be an industrial robot. In another example, the device can be a machine or a group of machines (e.g., a factory facility). For example, the operating state of a machine tool can be monitored. In these examples, the output data y can include information about the operating state and / or environment of the respective technical device.

[0042] In further examples, the system to be monitored may be a communications network. In some examples, the network may be a telecommunications network (e.g., a 5G network). In these examples, the input data x may include utilization data at nodes of the network, and the output data y may include information regarding resource allocation (e.g., channels, bandwidth of channels, or other resources of the network). In other examples, network malfunctions may be recognized.

[0043] In other examples (or additionally), the computer-implemented neural network may be configurable or usable for open-loop control (or closed-loop control) of a technical device. The device itself may be one of the devices discussed above (or below) (e.g., an at least partially autonomous robot or machine). In these examples, the output data y may include control variables of the respective technical system.

[0044] In yet other examples (or additionally), the computer-implemented neural network may be configurable or usable to filter a signal. In some cases, the signal may be an audio signal or a video signal. In these examples, the output data y may include the filtered signal.

[0045] The disclosed method for generating and applying a simplified computer-implemented neural network can be implemented on a computer-implemented system. The computer-implemented system may have at least one processor, at least one memory (which may contain a program that, when executed, performs the disclosed method), and at least one interface for input and output. The computer-implemented system may be a "standalone" system or may be a distributed system that communicates over a network (e.g., the Internet).

[0046] The present disclosure also relates to a (simplified) computer-implemented neural network generated using the disclosed method. The present disclosure also relates to a computer program configured to perform all the steps of the disclosed method. The present disclosure further relates to a machine-readable storage medium (e.g., an optical storage medium or a non-volatile memory, e.g., a flash memory) on which is stored a computer program configured to perform all the steps of the disclosed method.

Claims

1. 1. A computer-implemented method for generating a simplified computer-implemented neural network for a device, comprising: receiving a given neural network that includes a plurality of neural network substructures in the form of two or more layers; computing an approximate neural network for the given neural network; a matrix (Q) describing a change in a loss function due to pruning one or more neural network substructures from the plurality of neural network substructures for the approximate neural network; ｐｑ ), wherein the matrix (Q ｐｑ ) describes the change in the loss function due to simultaneous pruning of at least two neural network substructures from a plurality of neural network substructures located within different layers of the given neural network; The matrix (Q ｐｑ pruning at least one neural network substructure from the plurality of neural network substructures based on a change in the loss function described in A method comprising:

2. computing the approximate neural network includes linearizing the current weights of the given neural network; The method of claim 1.

3. The matrix (Q ｐｑ ) is calculated by adding multiple pruning vectors (δ ｐ ) to generate a plurality of second vectors (φ ｐ ) the gradient (φ(X)) is given by the current weights of the given neural network, The plurality of pruning vectors (δ ｐ ) from each pruning vector (δ ｐ ) corresponds to the network substructure to be pruned, The plurality of pruning vectors (δ ｐ ) from each pruning vector (δ ｐ ) describes the change in the current weights due to pruning each neural network substructure, Each second vector (φ ｐ ) are multiplied by the respective pruning vectors (δ ｐ ) are associated with 3. The method according to claim 1 or 2.

4. the given neural network is a trained neural network; a training data set (X) describing the behavior of the device; The training data set (X) consists of a number of pairs (X = {(x ｉ , y ｉ ) | i ≦ n}), each pair being formed from a set of inputs and a respective set of outputs of said device; the given neural network generates a respective set of outputs for each pair of the set of inputs; The method of claim 3.

5. The matrix (Q ｐｑ ) is calculated as a second vector (φ) from said plurality of second vectors for a plurality of pairs of neural network substructures from said plurality of neural network substructures of said trained neural network, optionally for each pair; ｐ , φ ｑ ), The method of claim 4.

6. the given neural network is an incompletely trained neural network; the vector of first derivatives of the loss function takes values ​​depending on the weights calculated after using the current weights, which do not correspond to a local minimum of the loss function for the current weights; The method of claim 3.

7. The matrix (Q ｐｑ ) is calculated as a second vector (φ) from said plurality of second vectors for a plurality of pairs of neural network substructures from said plurality of neural network substructures, optionally for each pair; ｐ , φ ｑ ) computing the first contribution by computing the scalar product of The current weights are subsequently assigned to a number of pruning vectors (δ ｐ ) and multiple pruning vectors (δ ｐ ) by calculating the scalar product of the vectors of the first derivatives of the loss function according to ｐｑ ) and summing the first contribution and the second contribution; The method of claim 6, comprising:

8. The pruning step comprises: The matrix (Q ｐｑ summing all elements in each column of (x,y,y) to obtain multiple sums; iteratively removing a plurality of neural network substructures starting from the neural network substructure corresponding to the smallest sum of the plurality of sums; The method of claim 5 or 7, comprising:

9. The pruning step comprises: The matrix (Q ｐｑ ) by removing the neural network substructure corresponding to the minimum amount of main diagonal elements of Iteratively removing a plurality of neural network substructures; Including, Each subsequent neural network substructure to be deleted is added to the matrix (Q ｐｑ ) corresponding to the minimum sum of multiple sums from the main diagonal and off-diagonal elements of The method according to claim 5 or 7.

10. The pruning step comprises: The matrix (Q) has predefined values. ｐｑ Optionally, said matrix (Q ｐｑ ) by removing the neural network substructures corresponding to the smallest main diagonal elements of Iteratively removing a plurality of neural network substructures; Including, Each subsequent neural network substructure to be deleted is ｐｑ ) corresponding to the next largest value on the main diagonal of The method according to claim 5 or 7.

11. the iterative elimination is performed until the size of the simplified neural network is reduced below a desired size.

11. The method according to any one of claims 8 to 10.

12. The iterative pruning is performed until the overall performance of the simplified neural network falls below a predefined threshold.

11. The method according to any one of claims 8 to 10.

13. the change in the loss function is approximated for the current weights using a second-order Taylor series for the given neural network; Optionally, the Taylor series also includes a first order term.

13. The method according to any one of claims 1 to 12.

14. the neural network is an image classifier; the image classifier is configured to receive input data in the form of image data and to classify the image data into one or more classes, optionally based on a semantic segmentation of the image data.

14. The method according to any one of claims 1 to 13.

15. controlling the device with the simplified neural network; or Recognizing the operating state of the device or the state of the environment of the device by the simplified neural network, or assessing the operational state of the device or the state of the environment of the device by the simplified neural network; 15. The method of claim 1, further comprising:

16. A computer program configured, when executed by a computer, to cause the computer to perform all the steps of the method according to any one of claims 1 to 14.

17. A machine-readable storage medium having stored thereon the computer program of claim 16.

Images