Generating simplified computer-implemented neural networks

Abstract

A simplified computer-implemented neural network is generated. The invention relates to techniques for generating a simplified computer-implemented neural network for a device. The method comprises receiving a pre-specified neural network comprising a plurality of neural network sub-structures in two or more layers. The method further comprises computing an approximated neural network for the pre-specified neural network and computing a matrix relating to the approximated neural network, the matrix describing a change in a loss function resulting from pruning one or more neural network sub-structures from the plurality of neural network sub-structures. The matrix describes the change in the loss function by simultaneously pruning at least two network sub-structures from the plurality of neural network sub-structures located within different layers of the two or more layers of the pre-specified neural network. Finally, the method comprises pruning at least one neural network sub-structure from the plurality of neural network sub-structures based on the change in the loss function described in the matrix to generate a simplified neural network.

Description

Technical Field

[0001] This disclosure relates to techniques for generating simplified computer-implemented neural networks from a pre-given neural network, and techniques for applying simplified computer-implemented neural networks. Background Technology

[0002] Computer-implemented neural networks are increasingly used in various technological devices. Here, many of these devices' neural networks can have complex structures (e.g., a large number of neurons, layers, and corresponding connections). This can place (excessive) demands on the hardware required to apply neural networks. Therefore, it may be necessary to find a trade-off between the performance and complexity of computer-implemented neural networks.

[0003] To address the aforementioned issues, so-called pruning methods have been developed. Pruning methods aim, on the one hand, to reduce the size of the neural network and simplify its overall structure, but on the other hand, to maintain good overall performance of the simplified neural network (or minimize the impact on overall performance). Therefore, for example, for smaller technological devices with limited hardware resources (such as power tools, gardening equipment, or household appliances), neural networks simplified according to these methods can be used. In other examples, it may be necessary to reduce the evaluation time of a computer-implemented neural network to ensure that technological devices (such as autonomous robots) react quickly enough. For this purpose, simplifying the computer-implemented neural network can also be advantageous. Summary of the Invention

[0004] This invention relates to a method for generating a simplified computer-implemented neural network for a device. The method includes: receiving a pre-given neural network comprising multiple neural network substructures in two or more layers. The method further includes: computing an approximate neural network for the pre-given neural network and computing a matrix associated with the approximate neural network, the matrix describing a change in a loss function caused by pruning one or more neural network substructures from the multiple neural network substructures. This matrix takes into account the change in the loss function by simultaneously pruning at least two neural network substructures from multiple neural network substructures located in different layers of two or more layers of the pre-given neural network. Finally, the method includes: pruning at least one neural network substructure from the multiple neural network substructures based on the change in the loss function described in the matrix to generate a simplified neural network.

[0005] Furthermore, the method includes: the application of a simplified computer-implemented neural network in a technical device, and the computer-implemented neural network generated using a corresponding method.

[0006] The present invention aims to generate simplified neural networks that are smaller in size (e.g., with fewer neurons and / or connections and / or layers) compared to the original, pre-given neural network. In this case, the overall performance or accuracy loss of the generated simplified neural network should not be too large (and ideally, no loss of overall performance or accuracy occurs). Such simplified (pruned) computer-implemented neural networks can be adapted to technical devices with relatively manageable hardware resources (e.g., portable electronic devices or devices without permanent network connections), or to technical environments requiring higher computational and evaluation speeds (e.g., in the case of at least semi-autonomous vehicles). However, reduced complexity and / or increased computational and evaluation speeds can also be used in other technical environments (e.g., reduced power consumption). Attached Figure Description

[0007] Figure 1a The diagram schematically illustrates the neural network initially trained before and after pruning (simplification). Neurons and their connections are shown as nodes (circles) or edges (arrows).

[0008] Figure 1b The pruning vector is illustrated schematically. This pruning vector can be used to transform the Hesser matrix H into a Q matrix (a matrix that describes the change in the loss function by pruning one or more neural network substructures from multiple neural network substructures). Pruning vector Here, with the nodes of the neural network p Related, and through pruning nodes p One or more connections describe the changes in the current weights.

[0009] Figure 2a The diagram schematically illustrates all the computed components of the Q matrix (black elements correspond to larger values) and the pruning performed using the "Lin Sum OBD" pruning method for neural networks created for the Boston Housing dataset (i.e., the standard dataset). The neural structures corresponding to the line-labeled components of the Q matrix are iteratively removed.

[0010] like Figure 2a In that way, Figure 2b The diagram schematically illustrates the identical components of the Q matrix and the pruning performed using the Iterative Full OBD (OBD) pruning method. The neural structures corresponding to the components of the Q matrix marked with white circles are iteratively removed.

[0011] like Figure 2a and Figure 2b In that way, Figure 2cThe diagram schematically illustrates the identical components of the Q-matrix and the pruning process using the "structured optimal brain damage" (OBD) pruning method. The neural structures corresponding to the components of the Q-matrix marked with white circles are iteratively removed.

[0012] Figure 3 The correlation between different pruning ratio values ​​and the following was compared. Figures 2a-2c The test loss of the pruning method used is shown. A pruning ratio of zero corresponds to the case where the pre-given neural network is not pruned. In this example, the trained neural network is selected as the pre-given neural network. Detailed Implementation

[0013] This disclosure relates to a method for generating a simplified computer-implemented neural network from a pre-given neural network by pruning its structure.

[0014] exist Figure 1a An exemplary neural network 1 is schematically sketched in the diagram. The neural network can consist of multiple neurons (exemplary neuron 2 is shown in the diagram). Figure 1a (Highlighted in the image), the neurons form nodes of neural network 1 and are interconnected via edges 3. The neurons of the computer-implemented neural network of this disclosure are arranged in multiple layers (e.g., ...). Figure 1a The third layer 4 contains three neurons. In this disclosure, edges or connections leading to neurons (or nodes) are considered part of the corresponding layer (i.e., input connections and nodes are in the same layer). Computer-implemented neural networks can contain two or more layers, three or more layers, or five or more layers. Neurons and their connections can have different structures and can be represented as nodes or edges using a graph. In this disclosure, any subset of the elements of a neural network is called a "neural network substructure." In some examples, a neural network substructure may include one or more edges, one or more nodes, or a combination of one or more nodes and edges (e.g., a node and edges leading to and from that node). Besides... Figure 1a The network shown may contain additional elements (such as feedback or memory). These elements may also be substructures of the neural network or part of it. In other examples, the elements of the neural network may be parameters used to describe the neural network (this is explained in more detail below).

[0015] The output of a given neuron j can depend on the inputs applied to one or more neurons i. In particular, a weighted sum of the applied inputs to neuron j can be formed first, where a formula from neuron j to neuron i can be defined for all neurons. The weights. This can be represented as neuron j being unconnected to neuron i. Finally, the output of neuron j can be determined after applying the activation functions defined for each neuron to the sum of the previously calculated values. Therefore, in some examples, this can be achieved by specifying the topology of the neural network and / or the weights of all neurons. To define a neural network, we can use the following: Therefore, weights can also be elements of a neural network substructure in the sense of this disclosure. That is, a neural network structure can contain one or more weights of the neural network (one or more edges can correspond to a graphical description of the neural network). According to the language rules described above, all weights of connections or edges leading to nodes or neurons in a given layer belong to that layer. The weights of the starting edges or connections of that given layer belong to another layer.

[0016] Computer-implemented neural networks can be created and trained to process data accumulated in a device (e.g., fully or partially trained) and, for example, to compute output data relevant to monitoring and / or controlling the device. Thus, the properties of the device or its response to specific events can ultimately be "hidden" in the topology and weights of the neural network. In other examples, neural networks can also be used through weights. Other and / or additional parameters are used to describe it. In this disclosure, weights These are discussed as exemplary parameters. However, the techniques described herein can also be used for other parameters.

[0017] Some existing techniques for pruning network structures from trained neural networks operate as follows: after generating the neural network, its topology depends on the task, and weights can be selected accordingly. The selection of these weights is called training or learning the neural network. This step is performed on a computer system. During "supervised learning," multiple input datasets are provided. (e.g., sensor data) and corresponding multiple desired output datasets (For example, the state of a technical device or its environment or control variables) (i.e., the input and output datasets form a pair). Each pair consists of N pairs. The resulting dataset is called the training dataset. Training a neural network can be formulated as an optimization problem, where the weights of the neural network... It must be adjusted so that the neural network responds to a given input. The generated output (in The transfer function of the neural network (which is the set of parameters θ, such as the weight set) should be as close as possible to the desired output. This optimization problem can be represented as a loss function. Minimization (in this disclosure, minimization refers to the minimization of the loss function, even though this may in some cases correspond to maximization of the expression). The loss function can take different forms. In one example, the loss function is the mean squared error (“MSE”) function, for example: Can target all weights The process involves minimization. As a result of this minimization, the trained weights are determined. The weights lead to the loss function The local (or global) minimum. Therefore, a trained neural network is built. The purpose of this training is not only to reproduce the desired output during the training phase ( Furthermore, it is able to provide other inputs. Provide reliable output (i.e., unknown output).

[0018] Trained neural networks can have complex topologies and a large number of neurons and connections, thus leading to high and undesirable hardware requirements in computation. As mentioned above, such complex neural networks are first simplified using pruning methods to make them usable on suitable devices. Figure 1a On the right, a simplified (pruned) neural network 1a is shown. For example, several edges (connections) are pruned between the top layer and the bottom layer 5. Additionally, nodes with associated edges are pruned in the penultimate layer 6.

[0019] This simplification can include: analyzing the loss function caused by pruning one or more neural network substructures. The corresponding changes. Pruning the structure can include: removing one or more connections (i.e., edges) between neurons and / or completely removing one or more neurons along with their access and outgoing connections. In other cases (or in another image), pruning can include: removing or setting one or more weights to zero (this could be an alternative description for removing one or more connections or edges). In other cases, the neural network can include more than Figure 1a The network structure shown includes elements such as feedback or memory. The network structure can also be pruned using the methods of this disclosure.

[0020] In some examples, the change in the loss function due to structural pruning can be expressed through the loss function. Regarding weight Taylor expansions are used to approximate the expression up to a definite order. For example, the expansion of the following expression can be given: .

[0021] here, It is a pre-defined neural network (e.g., with weights of 10 ... A given weight vector of a trained neural network. The surrounding perturbation vectors reflect the weights after pruning. The corresponding changes in the text. pass The transpose of is generated. = It is the Heather matrix, which is derived from the loss function. Regarding weight The second-order partial derivatives are composed of the second-order partial derivatives, which in this example are for the trained weights. Computed. This applies here to (at least approximately) the trained network. 0, because Having weights for training The minimum value (at least approximately within the predetermined numerical precision range) can be obtained. Terms of higher order than second order in the Taylor expansion can be ignored. The change in the loss function is then determined by... Given. For example... Figure 1b As explained, Weights corresponding to the respective substructures (e.g., node p) The pruning vector. In this case, the pruning vector describes the change in the current weight caused by pruning one or more connections of node p. The pruning vector can also be defined for other network substructures (e.g., for combinations of two or more nodes with corresponding access edges, where the nodes are in different layers or in one layer, or the nodes are adjacent nodes).

[0022] loss function The Taylor expansion shown is only an example, and different expressions can be used depending on the normalization of the chosen vectors (for example, the factor ½ in the second term of the Taylor expansion shown above can be recorded in...). and middle).

[0023] When pruning multiple neural network substructures, the change in the loss function can be expressed as the sum of multiple components of a matrix, which describes the change in the loss function caused by pruning one or more neural network substructures. For example, the change in the loss function can be determined by the following formula: (beschn.: pruning), where the matrix Defined as The running indices p and q in the sum range from 1 to S, where S represents the total number of network substructures. (Hesse matrix) From the second-order partial derivative This matrix is ​​composed of, and therefore typically has off-diagonal components (i.e., cross terms), which can be non-negligiblely small. This can be illustrated by the fact that the loss function... It could be about weights The nonlinear function. Therefore, through the Heather matrix. Defined matrix It can also have a non-diagonal shape. In short, the change in the loss function can be represented as the sum of two terms in some examples: The first item is a matrix. The sum of the diagonal components, where p = q, is the matrix that individually considers the change in the loss function due to pruning of each substructure p. Each diagonal component... Given only by the corresponding pruned substructure p. The second term now represents the matrix. The sum of all off-diagonal components, where p ≠ q. Therefore, each off-diagonal component This can describe the change in the loss function caused by simultaneously pruning substructures p and q. In this sense, it can refer to the correlation between the two substructures p and q, because this change in the loss function only occurs when both substructures are pruned simultaneously.

[0024] Although the expression for the change in the loss function seems simple, There are some difficulties in calculating this. First, the Heather matrix... (and therefore the matrix) ) dimensions Typically very large, where P represents the total number of connections in the neural network. For this reason, in some currently known methods in the prior art, the Hesser matrix is ​​approximated by a diagonal matrix (or almost a diagonal matrix). However, this approximation ignores the possible correlations between network structures in the sense defined above. However, this can lead to (particularly significant) inaccuracies in evaluations of which network substructures within the network substructures affect the performance of the neural network. Secondly, The calculation involves a large number of computational steps on a computer-implemented system, the number of which varies according to... pruning vector The scaling factor is determined by the dimension P and the size of the training dataset N. Furthermore, the number of computational steps is entirely independent of the fact that the corresponding dimension S of the pruned network substructure can be significantly smaller than the total number of connections P in the neural network. Therefore, the above estimation for complex structures with large P may be computationally difficult to perform. Thirdly, an additional problem arises: training neural networks (especially relatively large and complex ones) can be computationally very expensive and therefore a lengthy task. In some implementations, the techniques of this disclosure can address these problems.

[0025] A first step of the method disclosed herein for generating a simplified computer-implemented neural network may, for example, include: receiving a pre-given neural network comprising multiple neural network substructures in two or more layers. For example, each layer may be given by multiple neurons and corresponding access connections. Here, the weighted output of a neuron in one layer may be the input of another neuron that can be set in another layer. For example, according to Figure 1a Neural network 1 contains four layers. For a given neural network, the current weights can be specified. In some cases, it may also receive training datasets used to train neural networks or datasets intended for training. In some cases, the pre-given neural network can be a trained neural network (e.g., the loss function has been minimized through training). In other examples, the pre-given neural network can be an incompletely trained neural network (e.g., the loss function has not yet been minimized through training). In some cases, an incompletely trained neural network can only perform a small fraction of the training epochs required for full training (a so-called epoch, which can represent a full training iteration on the training dataset, such as when performing Stochastic Gradient Descent or other methods) (e.g., less than 100 training epochs or less than 20 training epochs). The pruning method disclosed herein is particularly advantageous for incompletely trained neural networks. “Early” pruning can reduce training overhead.

[0026] Training dataset It can contain different types of data, with each type of data being input in a pair. and output data Combine into a pair For example, the input and output dates can be scalars (e.g., scalar measurements), vectors of arbitrary length (i.e., length 1 or greater), or matrices, respectively. Input data can represent environmental influences or the internal operating state of a technical device. In one example, input data may include sensor data. Alternatively or additionally, input data may include image data and / or audio data. Output data can be the state of the device or environment or an identified event (e.g., the state of an electronic device such as "battery almost depleted" or "it's raining"). In other examples, output parameters can be control parameters (e.g., for an actuator) or other ways of identifying the device's response.

[0027] According to the present invention, an approximate neural network is first computed for a pre-given neural network. This may include the current weights around the pre-given neural network. An approximation. For example, by surrounding the current weights. Linearization can be used to generate approximate neural networks. For example, a neural network can be approximated by adding a term that scales linearly with the weights to the sum of the evaluations of the neural network's transfer function with respect to a set of weights. This can be described, for example, as follows: ,in It is the gradient of the transfer function of the neural network, the gradient of which is applied to the current weights. In this case, it is computed on the training dataset. This measure allows other steps of the method to be performed in a resource-efficient manner (e.g., using fewer computational steps on a computer system).

[0028] In another step of the method according to the invention, the matrix associated with the already generated approximate neural network can be calculated. This matrix describes the loss function. Changes. As mentioned above, these changes can be caused by pruning one or more neural network substructures from multiple neural network substructures. Matrix This can describe the change (i.e., matrix) in the loss function caused by simultaneously pruning at least two network substructures from multiple network substructures located in different layers of a pre-given neural network. The correlation of responses between at least two network substructures can be considered. Furthermore, in some examples, the matrix... One can consider the change in the loss function caused by simultaneously pruning network substructures located within a layer of a pre-given neural network. In some examples, a matrix can describe the change in the loss function caused by simultaneously pruning multiple pairs of network substructures (e.g., all possible pairs) from multiple neural network substructures.

[0029] Based on matrix The changes in the loss function described herein, such as those based on multiple (e.g., all) calculated components of the matrix (rather than solely on the main diagonal components as in prior art methods), can determine which neural network substructures contribute little or even negligible to the overall performance of the original, pre-given neural network (e.g., the overall performance degradation is no greater than a predetermined measurement). For example, in this context, only those substructures that do not cause a loss function degradation after pruning... The increase may only cause the loss function Non-excessively ascending substructures can be classified as substructures to be pruned. Therefore, such classified substructures can be pruned from a pre-given neural network to enable the generation of simplified neural networks for devices. The resulting simplified neural networks can provide data to devices more quickly and / or require fewer hardware resources.

[0030] Here, if we use a pre-given neural network with respect to its current weights... The second-order Taylor series approximation is obtained by pruning. The linearization described above can correspond to an approximation in the loss function if the given neural network is not a trained neural network (but a partially trained neural network). In some cases, if the given neural network is not a trained neural network (but a partially trained neural network), the loss function changes according to the first derivative of the weights. Not zero.

[0031] Furthermore, the method may include, for example, calculating the aforementioned gradient of the pre-given neural network around the current weights. In one example, gradient The calculation can use the current matrix X, which can contain N pairs of input data from the input dataset. Then, the gradient can be... Multiply by the pruning vector (also known as the perturbation vector) of the network substructure. To obtain the second vector of dimension N. Each pruning vector Assign this to the network substructure to be pruned. In this way, multiple second vectors can be derived. (i.e., S second vectors) to account for possible structural variations in a pre-given neural network. The computational cost for these steps is as follows: Scaling. In other examples, gradients. and the second vector The calculation can be performed using the Jakobi vector product. This is also computationally more efficient than some existing methods.

[0032] As already mentioned, in one example, the pre-given neural network can be a trained neural network. In this case, the training dataset (X) can consist of multiple pairs of... The system is composed of pairs, each consisting of an input set and a corresponding output set from the technical device, and a pre-defined neural network that can generate the corresponding output set for each pair of input sets. Furthermore, the loss function is based on the first derivative of the weights using trained weights. The vector calculated later can have values The value corresponds to a local (or global) minimum of the loss function (within a predetermined precision range). In this example, the matrix... The computation of all components can be performed by computing the obtained second vector from multiple second vectors for multiple pairs (e.g., each pair) of neural network substructures of the trained neural network. and To perform ( ) scalar product ).

[0033] In other examples, the pre-given neural network can be an incompletely trained neural network (where the term "incompletely trained" is understood in the sense described above), in such a way that the first derivative of the loss function is used with the current weights. The vector calculated later can have the following values ​​based on its weights (e.g., values ​​that are clearly not equal to zero, where...). The value does not correspond to a local minimum of the loss function with respect to the current weights. In this case, the loss function will not have a local minimum with respect to the current weights. In one example, common training of the neural network can terminate after several optimization steps (epochs), and the resulting neural network can be used as a pre-given neural network for the method of this disclosure. In another example, the neural network can be initialized with determined weight values. The initialized neural network can then be implemented immediately in the method according to the invention and treated as a pre-given neural network. Using an incompletely trained neural network may be particularly suitable if common training of the neural network is computationally expensive or simply impossible due to limited hardware resources.

[0034] Calculate the matrix using an incompletely trained neural network. This can include three steps. First, as in the first example of a trained neural network, for multiple pairs (e.g., each pair) of neural network substructures of an incompletely trained neural network, the obtained second vector from multiple second vectors can be computed. and The scalar product. In this way, the (first) contribution of the main diagonal and secondary diagonal components can be calculated. Furthermore, consider the matrix... The second value of the main diagonal component is calculated by applying the loss function to multiple pruning vectors with the current weights. The vector of the first derivative and multiple first vectors The scalar product is used to perform the operation. Finally, the two contributions are added together. For example, this might happen in the following formula. ,in It is the Kronecker function.

[0035] Regardless of whether a trained neural network or an incompletely trained neural network is used as a pre-given neural network in this method, the matrix The computational cost can be reduced in some cases to Scaling. In summary, the total computational cost of the method according to the present invention can be used... Scaling to compute the matrix All components. The method of the present invention can therefore have the benefit of computational cost, particularly in which... In the case of a larger structure, because the matrix Direct calculation may require significantly more computational steps. .

[0036] In the next step of this method, consistent with the discussion above, a matrix is ​​used. The information in the data is used to estimate which substructures can be pruned.

[0037] In the first method, the matrix can be first... In each column, all components are summed to obtain multiple sums. Then, considering the calculated sums, multiple neural network substructures are iteratively removed. For example, this can be implemented starting with a neural network substructure that calculates the minimum value of the sum among multiple sums. These neural network substructures can be removed first. For Figure 2a The neural network in the figure exemplifies the results of this so-called "Lin Sum OBD" ("optimal braindamage") pruning method, which was created for the Boston Housing Dataset (i.e., the standard dataset). Four exemplary columns 7a-d are shown in the figure, where the magnitude of the sum is illustrated by the density of the holes. The right column 7d has the smallest sum. Therefore, the relevant neural network substructures can be pruned first.

[0038] In the replacement method, the matrix corresponding to the first step is to remove the first step. The neural network substructure with the minimum value of its main diagonal components. Subsequently, multiple neural network substructures are iteratively removed, where each subsequent substructure to be removed corresponds to the minimum value of the sum of the main diagonal and off-diagonal components of the matrix associated with the previously removed substructures. Figure 2b The text shows the relationship between... Figure 2aAn example of applying this "Iterative Full OBD" pruning method to the same neural network. Exemplary diagonal and off-diagonal components are represented by circles, where the size of the circle is inversely proportional to the size of the main diagonal component / the sum of the main diagonal and off-diagonal components.

[0039] In another alternative method, the matrix corresponding to a predefined value can be removed first. The main diagonal components of ) and the optional matrix ( The neural network substructure of the least diagonal component of the matrix. In one example, the matrix can first be... The calculated main diagonal components are arranged in ascending order. Then, the neural network substructure corresponding to the smallest main diagonal component is removed. Next, the method according to the invention includes iteratively removing multiple neural network substructures, wherein each subsequent neural network substructure to be removed corresponds to the next larger value of the arranged main diagonal components. This is used in conjunction with... Figure 2a and Figure 2b The results of this "structured OBD" ("structured optimal brain damage") pruning method on the same neural network are as follows: Figure 2c The example is shown below. An exemplary diagonal component is represented by a circle, where the size of the circle is inversely proportional to the size of the main diagonal component.

[0040] exist Figure 3 The comparison is related to different values ​​of the pruning ratio in Figures 2a-2c The test loss of the pruning method used is shown. A pruning ratio of zero corresponds to the case where the pre-given neural network is not pruned. In this example, the trained neural network is selected as the pre-given neural network. The overall performance of the pruned network for the Iterative Full OBD or Structured OBD methods is practically unaffected by pruning ratios less than 0.7 or 0.4, as shown in the figure.

[0041] Typically, all three pruning processes described above are performed until the size of the simplified neural network is lower than the desired size. In one example, the desired size can be given by the minimum number of neurons in the simplified neural network or within a layer of the simplified neural network. In other examples, the desired size can be defined by the minimum number of connections between neurons in the simplified neural network or by the minimum number of connections between neurons within a layer of the simplified neural network. The desired size can also be given, for example, as the minimum number of unpruned layers or structures in the simplified neural network. In other examples, pruning methods are performed until the overall performance of the simplified neural network falls below a predefined threshold. For example, the overall performance can be estimated using a test loss (see example...). Figure 3 This can be calculated using a loss function. In one example, the predefined threshold can be defined as the ratio of the overall performance of a simplified neural network to that of a pre-given neural network. In another example, the predefined threshold can correspond to a selected number.

[0042] As previously described, the simplified computer-implemented neural network of this disclosure can be used in a variety of technical applications. Generally, this disclosure also relates to methods for pruning a computer-implemented neural network and then using the computer-implemented neural network in a technical device. This use may include, for example, controlling (or regulating) a technical device using the simplified neural network, identifying the operating state (e.g., a malfunction) of the technical device or the state of the environment of the technical device using the simplified neural network, or evaluating the operating state of the technical device or the state of the environment of the technical device using the simplified neural network. In this case, the input data may include state data (e.g., at least some sensor data) related to the internal state of the device. Additionally or alternatively, the input data may include state data related to the environment of the device (e.g., at least some sensor data). The output data of the simplified neural network may characterize the operating state or other internal state of the device (e.g., the presence of an error or critical operating state). The output data can be used to control the device in response to the characterized operating state or another internal state. Alternatively or additionally, the output data may include control data for the device.

[0043] In some cases, the device can be an electrical appliance (e.g., a tool, household appliance, or gardening equipment). In other examples, the device can be an Internet of Things (“IoT”) device. Alternatively or additionally, the device can be a battery-powered device. As mentioned above, simplified computer-implemented neural networks may be advantageous in these environments because they can be designed to be relatively resource-efficient.

[0044] Simplified computer-implemented neural networks can be used to classify time series data, particularly image data (i.e., the device is an image classifier). Image data can be, for example, camera image data, LiDAR image data, radar image data, ultrasound image data, or thermal image data (e.g., generated by the corresponding sensor). Image data can include individual image or video data. In some examples, computer-implemented neural networks can be designed for monitoring devices (e.g., manufacturing processes and / or for quality assurance) or for medical imaging systems (e.g., for finding diagnostic data) or can be used in such devices. The image classifier can be designed to receive image data and classify it into several categories. In some examples, this can include a one-dimensional (...) containing image data The input data in the form of an input vector is mapped to the second dimension ( The output vector is in the form of an output data vector, representing the classification result. For example, the components of the input vector can represent multiple received image data. Each component of the output vector can represent the image classification result calculated using a simplified computer-implemented neural network. In some examples, image classification can include semantic segmentation of the image (e.g., pixel-by-pixel classification of the image). For example, image classification can be object classification. For example, the presence of one or more objects in image data can be detected (e.g., automatically identifying traffic signs or lanes in a driver assistance system).

[0045] In other examples (or additionally), a computer-implemented neural network can be designed or used to monitor the operational status and / or environment of at least a semi-autonomous robot. The semi-autonomous robot can be an autonomous vehicle (or another at least semi-autonomous means of movement or transport). In other examples, the semi-autonomous robot can be an industrial robot. In other examples, the device can be a single machine or a group of machines (e.g., an industrial site). For example, the operational status of a machine tool can be monitored. In these examples, the output data y can contain information related to the operational status and / or environment of the corresponding technological device.

[0046] In other examples, the system to be monitored can be a communication network. In some examples, the network can be a telecommunications network (e.g., a 5G network). In these examples, the input data x can contain utilization data in network nodes, and the output data y can contain information related to resource allocation (e.g., channels, bandwidth in network channels, or other resources). In other examples, network faults can be identified.

[0047] In other examples (or additionally), computer-implemented neural networks can be designed or used to control (or regulate) technological devices. These devices can be one of the devices discussed above (or below) (e.g., at least a semi-autonomous robot or machine). In these examples, the output data y can contain control variables of the corresponding technological system.

[0048] In other examples (or additionally), computer-implemented neural networks can be designed or used to filter the signal. In some cases, the signal can be an audio or video signal. In these examples, the output data y can contain the filtered signal.

[0049] The methods of this disclosure for generating and applying simplified computer-implemented neural networks can be executed on a computer-implemented system. The computer-implemented system may include at least one processor, at least one memory (which may contain a program that, when executed, performs the methods of this disclosure), and at least an interface for input and output. The computer-implemented system may be a "standalone" system or a distributed system communicating via a network (e.g., the Internet).

[0050] This disclosure also relates to (simplified) computer-implemented neural networks generated using the methods of this disclosure. This disclosure also relates to computer programs configured to perform all the steps of the methods of this disclosure. Furthermore, this disclosure relates to a machine-readable storage medium (e.g., an optical storage medium or a solid-state memory such as flash memory) on which a computer program is stored, the computer program being configured to perform all the steps of the methods within this disclosure.

Claims

1. A method for generating a simplified computer-implemented neural network for a device, the method comprising the steps of: Receive a pre-given neural network, which contains multiple neural network substructures in two or more layers, wherein the neural network is trained based on a training dataset consisting of an input dataset of the device and a corresponding output dataset, wherein the input dataset includes data collected by sensors and related to the internal state of the device or the state of the environment. For a pre-given neural network, compute an approximate neural network, wherein the computation of the approximate neural network includes linearizing around the current weights of the pre-given neural network; Calculate the matrix related to the approximate neural network ( ), the matrix ( This describes the change in the loss function caused by pruning one or more neural network substructures from multiple neural network substructures. The matrix ( This describes the change in the loss function caused by simultaneously pruning at least two network substructures from multiple neural network substructures located in different layers of a pre-given neural network, where the computation matrix ( This includes: the gradient ( Multiply by multiple pruning vectors ( ) to determine multiple second vectors ( ), wherein the gradient ( ) is given by the current weights of a pre-given neural network, where multiple pruning vectors ( Each pruning vector in ) ) are assigned to the network substructures to be pruned, where multiple pruning vectors ( Each pruning vector in ) describes the change in the current weights caused by pruning the corresponding neural network substructure, and where multiplication results in a second vector ( Assign the corresponding pruning vectors ( ); Based on the matrix ( The loss function described in [reference] is modified to prune at least one neural network substructure from multiple neural network substructures in order to generate a simplified neural network.

2. The method according to claim 1, The pre-given neural network is a trained neural network, where the training dataset (X) describes the characteristics of the device, and the training dataset (X) consists of multiple pairs (X= Each pair consists of an input set and a corresponding output set of the device, and a pre-given neural network generates a corresponding output set for each pair of input sets.

3. The method according to claim 2, Specifically, for multiple pairs of trained neural networks, and optionally for multiple neural network substructures from multiple neural network substructures in each pair, the second vector among multiple second vectors is calculated. Scalar product to perform matrix The calculation.

4. The method according to claim 1, The pre-given neural network is an incompletely trained neural network, in such a way that the vector of the first derivative of the loss function with respect to the weights calculated after using the current weights is assumed to have the following value, which does not correspond to a local minimum of the loss function with respect to the current weights.

5. The method according to claim 4, Where the matrix The calculation includes the following steps: For multiple pairs, and optionally for each pair of neural network substructures from multiple neural network substructures, by computing the second vector from multiple second vectors. The first contribution is calculated using the scalar product; The loss function is calculated for multiple pruning vectors that then use the current weights. The vector of first derivatives and multiple pruning vectors Scalar product to calculate the matrix The second contribution of the main diagonal component; Add the first contribution to the second contribution.

6. The method according to any one of claims 3 or 5 above, The pruning includes: For matrix Sum all components in each column to obtain multiple sums; Starting with the neural network substructure corresponding to the minimum value of the sum among multiple sums, iteratively remove multiple neural network substructures.

7. The method according to any one of claims 3 or 5 above, The pruning includes: Remove the matrix corresponding to The neural network substructure with the minimum value of the main diagonal component; Iteratively remove multiple neural network substructures, where each subsequent neural network substructure to be removed corresponds to the matrix. The minimum sum of multiple components consisting of main diagonal and off-diagonal components related to the previously removed neural network substructure.

8. The method according to any one of claims 3 or 5 above, The pruning includes: Remove the matrix corresponding to the predefined values. The main diagonal components, optionally, are matrices. The neural network substructure with the minimum main diagonal component; Iteratively remove multiple neural network substructures, where each subsequent neural network substructure to be removed corresponds to a matrix. The next larger value of the main diagonal component.

9. The method according to claim 6, Iterative removal is performed until the size of the simplified neural network is lower than the desired size.

10. The method according to claim 6, Iterative removal is performed until the overall performance of the simplified neural network falls below a predefined threshold.

11. The method according to any one of claims 1 to 5, The loss function is approximated by using a second-order Taylor series around the current weights of a pre-given neural network, optionally including first-order terms.

12. The method according to any one of claims 1 to 5, The neural network described therein is an image classifier, which is designed to receive input data in the form of image data and optionally classify the image data into one or more categories based on semantic segmentation of the image data.

13. The method according to any one of claims 1 to 5 further comprises: The device is controlled via a simplified neural network; or The operating status of the device or the state of the environment of the device is identified through a simplified neural network; or The operating status of the device or the state of the environment is evaluated using a simplified neural network.

14. A computer program product having a computer program configured to perform all the steps of the method according to any one of claims 1 to 12.

15. A machine-readable storage medium on which a computer program according to claim 14 is stored.

Images