Device and method of reparametrizating a residual network for computational efficiency

Abstract

A computer-implemented method (20) for reparameterizing a residual network (M), wherein the residual network (M) is a pre-trained neural network comprising residual connections that skip residual blocks, comprising the following steps: evaluating (S22) a baseline performance (P) of the residual network (M) on a first data set. executing a loop (S23) while a reduction in application performance (R) with respect to the baseline performance (P) is less than a given tolerable reduction (δ), the loop comprising the following steps: selecting a residual block (b ∈ M) of the residual blocks for reparameterization, executing a second loop through the set i ∈ ε, 2ε, ...,1: exchanging all non-linear activation functions f j (x) ∈ b by a new function f j (x) = (1 - ε) ∗ f j(x) + ε ∗ x and retraining M on a second dataset. Reparameterize residual block b into a single layer.

Description

[0001] The invention relates to a method for reparameterizing a residual network and a method for operating an actuator by the reparameterized residual network, a computer program and a machine-readable storage medium, as well as a system. State of the art

[0002] A residual neural network (also called a residual network, ResNet, see He, Kaiming, et al., "Deep residual learning for image recognition." Proceedings of the IEEE conference on computer vision and pattern recognition, 2016) is a deep learning model in which the weighting layers learn residual functions with respect to the layer inputs. From a practical perspective, a residual network is a network with at least one skip connection, also called a residual connection, that performs an identity assignment and is summed with the outputs of the skipped layer(s).

[0003] Residual connections are widely used in modern deep neural networks because they improve the trainability of networks. However, residual connections also significantly reduce the hardware efficiency of network inference. This is because a forward activation generated by the residual connection must be held in memory while the residual block is computed. The residual block can consist of the layer(s) skipped by the residual connection. Typically, this activation must be stored in DRAM and reloaded to compute the addition. This results in additional load / store operations, making memory traffic a bottleneck. Therefore, while residual connections are advantageous during training, which is why they are widely used, they introduce inefficiencies during inference.

[0004] Methods are known in the literature that exploit the advantages of residual connections during training and provide approaches to reduce or overcome the disadvantage of residual connections during inference.

[0005] Jha et al.'s work, "Deepreduce: ReLU reduction for fast private inference" (ICML 2021), proposes heuristically removing nonlinear activation functions (e.g., ReLU) from the neural network (NN) using an importance score. The importance score is calculated by training different variants of the NN, removing ReLU operations at different stages, and observing the accuracy. In the first optimization step, all ReLU operations are removed from some stages. In the second step, every other ReLU is removed from selected other stages.

[0006] Vasu et al.'s work, "MobileOne: An Improved One Millisecond Mobile Backbone" (CVPR 2023), proposes designing a specialized NN architecture that uses residual connections during training, but reparameterizes them into a residue-free network before inference, computing the same mathematical function. This is only possible by avoiding nonlinear operations in the residual block, unlike established and widely used networks such as ResNet, MobileNet, or EfficientNet architectures.

[0007] The work of Yu et al., "NetBooster: Empowering Tiny Deep Learning By Standing on the Shoulders of Deep Giants," DAC 2023, does not aim to reduce the complexity of neural networks, but rather strives to improve network training by first inflating the network (replacing a single layer with multiple layers with a nonlinear function between them), then training the inflated network, and finally reducing the architecture back to its original topology. The final step consists of progressively linearizing the newly introduced nonlinear activation functions: progressively interpolating between the activation function and the identity function until the activation function can finally be removed and the layers combined.

[0008] A disadvantage of the known methods is that they either require large computing and data resources to train many variants of the NN so that the importance value can be calculated, or that the developers have to use a special, non-standardized architecture of the neural network during training. Advantages of the invention

[0009] The present invention (data and computation) efficiently removes residual connections in trained neural network architectures for efficient inference while requiring only minimal computational and data resources. Disclosure of the invention

[0010] In a first aspect, the invention relates to a computer-implemented method for reparameterizing a residual network. The residual network may be a pre-trained neural network comprising residual connections that skip residual blocks. Reparameterization may be understood as reconfiguring the residual connection and the residual block into a non-residual layer or a non-residual layer sequence. In other words, reparameterization may comprise deleting at least one residual connection and converting the corresponding residual block of the deleted residual connection into a pure feedforward layer sequence or into a single layer, wherein the layer sequence or layer may be periodically modified to perform substantially the same computations as the residual connection in combination with the residual block.Thus, parameters, particularly weights, of the residual block are reparameterized to operate the reparameterized residual block without the residual connection in such a way that the performance of the residual network is substantially unchanged or outputs substantially similar or identical activations as with the residual connection. The reparameterized residual block may comprise multiple original layers that have been reparameterized in whole or in part, or the original layers of the residual block have been converted into a single new layer that performs essentially the same computations as the residual connection in combination with the residual block.

[0011] The procedure begins by evaluating the baseline performance of the residual network using an initial dataset. The initial dataset is preferably small. The term "small" can be understood to mean that the dataset should be large enough to obtain reasonable performance estimates. This might require at least hundreds of images, more likely several thousand. However, compared to typical training datasets containing millions of images, this is small.

[0012] This is followed by a step of executing a first loop while the performance degradation of the application compared to the baseline performance is less than a given tolerable degradation, e.g., a few percent, such as 1% - 5%.

[0013] The loop comprises the following steps: Selecting a residual block b of the residual blocks for reparameterization. Executing a second loop over a set i ∈ ε, 2ε, ...,1, where ε is a step size, preferably expected to be 0.005 < ε < 0.1: Replacing all non-linear activation functions f j (x) ∈ b by a new function f j (x) = (1 - ε) ∗ f j (x) + ε ∗ x and performing a retraining (so-called finetuning) or retraining of the linearized residual network on a second data set.

[0014] This is followed by a step of reparameterizing the selected residual block b into a sequence of feedforward layers or a single layer, as known from, for example, MobileOne and NetBooster. Note that progressive linearization can be used to remove nonlinear activations. Linear operations can be reparameterized, including the most common operations such as convolution, fully connected layers, or batch normalization. This is followed by a step of evaluating application performance on the first dataset and updating the performance degradation depending on application performance.

[0015] If the performance degradation is greater than the tolerable degradation, the current reparameterization is undone and the first loop is terminated.

[0016] Finally, the reparameterized residual network can be output or deployed on a target device.

[0017] It is proposed that the step of selecting a residual block be performed based on an estimated impact of the residual blocks on application performance and / or the residual blocks on hardware efficiency, e.g., on the device intended for inference. If the impact is smaller than a predefined threshold, the residual block is selected for reparameterization. The predefined threshold can be defined depending on the use case of the residual network. The impact on application performance and the impact on hardware efficiency can be combined into a single metric used to select the residual block. This can be done using simple scalar functions such as addition or multiplication applied to both impacts.

[0018] In a further aspect of the invention, a computer-implemented method for using the reparameterized residual network as a classifier for classifying sensor signals is proposed. The classifier is adopted with the method according to any of the preceding aspects of the invention, which comprises the following steps: receiving a sensor signal comprising data from the image sensor, determining an input signal that depends on the sensor signal, and feeding the input signal into the classifier to obtain an output signal that characterizes a classification of the input signal.

[0019] In a further aspect of the invention, a computer-implemented method for using a classifier trained to provide an actuator control signal for controlling an actuator is provided. Determining an actuator control signal depending on an output signal of the classifier, which can be determined as described in the previous section. It is proposed that the actuator controls an at least partially autonomous robot and / or an at least partially autonomous manufacturing machine and / or an at least partially autonomous access control system.

[0020] In a further aspect of the invention, a control system for operating the actuator is proposed. The control system comprises the classifier adopted according to one of the preceding aspects of the invention and is configured to operate the actuator in accordance with an output of the classifier.

[0021] Embodiments of the invention are explained in more detail with reference to the following figures. They show: Fig. 1 a schematic fusion of different layers into a final layer; Fig. 2 a schematic residual block as a component of a residual neural network; Fig. 3 is a flowchart of an embodiment of the invention; Fig. 4 a control system that controls an at least partially autonomous robot; Fig. 5 a control system that controls a manufacturing machine; Fig. 6 a control system that controls an access control system; Fig. 7 a control system that controls a monitoring system; Fig. 8 a control system that controls an automated personal assistant; Fig. 9 a control system that controls an imaging system; Fig. 10 a training system. Description of the embodiments

[0022] Layer fusion and reparameterization are well-known techniques, e.g., consecutive convolutional layers and batch normalization (BN) layers can be fused into a single convolutional layer, since the BN only performs a linear operation at inference time. Fig. Figure 1 schematically shows a fusion (1a) of multiple convolutional Conv2D layers and batch normalization (BN) layers fused into a single Conv2D layer. Similarly, consecutive convolutional layers can be merged into a single Conv2D layer by combining their respective kernels into a new (larger) kernel.

[0023] Residual branches without a non-linear activation function can be reparameterized into a single layer (1b) by modifying the kernel of the convolution operation accordingly.

[0024] Fig. Figure 2 shows an example of a part of a residual network. An input activation X of a previous part of the residual network, in particular of a previous layer of the residual network, is propagated through the residual block (2a) to obtain an output activation Y. The residual block (2a) comprises a residual block, a residual connection, and an addition layer. The residual connection forwards the input activation X directly to the addition layer, thereby skipping the layer of the residual block. The residual block comprises one or more layers. The layers can be any layers used in existing deep neural networks. Fig. Figure 2 shows examples of convolutional, ReLU, and batch normalization layers. The output of the residual block and the forwarded input activation are combined in the addition layer, which outputs the output activation Y. The merging operation of the addition layer can be a simple addition of the two inputs of the addition layer.

[0025] An object of the present invention is to reparameterize some of the residual connections in a residual network such that a Pareto-optimal trade-off is preferably achieved between application performance (e.g., maximizing accuracy, minimizing regression error) and hardware efficiency of inference (e.g., minimizing memory traffic), given a trained neural network that uses residual connections and a limited dataset.

[0026] Typically, this goal is formulated as a constrained optimization, i.e., maximizing hardware efficiency while maintaining a certain level of application performance. The main challenge is determining which residual blocks should be reparameterized.

[0027] Fig. Figure 3 shows an example of a method (20) for reparameterizing a residual network in algorithmic form.

[0028] In the first step (S21), relevant inputs for the procedure are obtained, where the following elements may be given: a. M: a pre-trained model that uses residual connections, b. D t : a potentially small labeled training dataset for short retraining, c. L: loss function for training, d. D v: a potentially small labeled validation dataset to estimate application performance (to obtain meaningful application performance figures, there should be at least hundreds to thousands of images, but can be more), e. δ: the tolerable reduction in application performance, and f. ε: the step size for progressive linearization (small number, as given above as an example).

[0029] In the second step (S22), an evaluation is performed. The basic application performance of M is evaluated using the small training dataset and stored as P.

[0030] In the next step (S23), a first loop is executed while the application performance degradation R is less than the tolerable degradation δ. The first loop comprises the following steps: 1. Identify a residual block b ∈ M to be linearized. 2. Execute a second loop for the sentence i ∈ ε, 2ε, ...,1: a. Replacing all non-linear activation functions f j (x) ∈ b by the new function f j (x) = (1 - ε) ∗ f j (x) + ∈ ∗ x for linearization. b. Perform a short retraining of M using D t and L 3. Reparameterize block b into a single convolutional layer. This is possible because block b no longer contains any nonlinear operations. MobileNet or NetBooster can be used. 4. Evaluate the application performance of M using D v and storing it as P'. 5. Update the power reduction: R←P-P'.

[0031] After step S23 is completed, an optional step of providing (S24) the reparameterized network to a target device for inference is performed.

[0032] On the target device, the provided network can be used to control (S25) an application, e.g., as in Fig. 4 to 9 are shown as examples.

[0033] A key challenge is to identify the blocks b ∈ M to be linearized, which is the first step of the loop of S23. This requires estimating the impact of block b on application performance and / or the impact of block b on hardware efficiency, taking into account hardware constraints.

[0034] There are several alternatives for calculating these metrics. Combinations are possible, for example, the hardware cost of a block can be calculated as "normalized feature map size + normalized latency." More generally, these individual scalar estimates can be combined using scalar operations such as (weighted) averaging, etc.

[0035] In a first variant for calculating the application impact metric, the metric is estimated using simple profiling. This involves the following steps: 1) Evaluate the baseline application performance of M and store it as P. 2) For each block b ∈ M that has not yet been reparameterized: a. Replacing non-linear activation functions f j (x) ∈ b by the identity function g j (x) = x. b. Evaluate the application performance of M with D v and save it as P b . c. Undo the change in the activation functions. 3) Estimate the impact of block b on application performance as P - P b .

[0036] A second approach to calculating the application impact metric involves estimating the metric using sensitivity analysis. This involves the following steps: 1) For each block b ∈ M that has not yet been reparameterized: a. Introducing a new variable s b to display the sensitivity. b. Replacing non-linear activation functions f j (x) ∈ b by the identity function g j (x, s b ) = (1 - s b ) ∗ f j (x) + s b * x. c. Initializing s b = 0, so that g j (x, s b ) = t j (x). d. Calculate the gradient ∂ / ∂s b L(D v ,M). e. Revert the change to the activation functions. 2) Estimate the impact of block b on application performance by s b .

[0037] In a first variant for calculating the hardware efficiency impact metric, the metric is estimated using a size of the residual feature map. This involves the following steps: 1) For each block b ∈ M that has not yet been reparameterized: a. Calculate the size of the residual feature map x b by |x b |, which is the cardinality of x b represents. 2) Estimate the impact of block b on hardware efficiency as |x b |.

[0038] In a second variant for calculating the hardware efficiency impact metric, the metric is estimated by profiling the block on real hardware or in a simulation. This involves the following steps: 1) For each block b ∈ M that has not yet been reparameterized: a. Form a profile of the latency l for the inference of this block. 2) Estimate the impact of block b on hardware efficiency as l.

[0039] A third variant for calculating the hardware efficiency impact metric is to estimate the metric by profiling the block on real hardware / in a simulation compared to the reparameterized block. This involves the following steps: 1) For each block b ∈ M that has not yet been reparameterized: a. Creating a latency profile l orig for the inference of this block. b. Creating a latency profile l reparametrized for the inference of this block when reparameterized (this requires only knowing the architecture after reparameterization, not the weights). 2) Estimate the impact of block b on hardware efficiency as l orig - l reparametrized .

[0040] Finally, the impact on application performance and the impact on hardware efficiency can be combined into a single metric that is used to select the block. This can be done using simple scalar functions such as addition or multiplication.

[0041] In Fig. Figure 4 illustrates an embodiment of an actuator with a control system 40. The actuator and its environment are collectively referred to as the actuator system. At preferably evenly spaced intervals, a sensor 30 detects a state of the actuator system. The sensor 30 may comprise multiple sensors. Preferably, the sensor 30 is an optical sensor that captures images of the environment. An output signal S from the sensor 30 (or, if the sensor 30 comprises multiple sensors, an output signal S for each of the sensors), which encodes the detected state, is transmitted to the control system 40.

[0042] The control system 40 receives a stream of sensor signals S. It then calculates a series of actuator control commands A based on the stream of sensor signals S. These commands are then transmitted to the actuator unit 10, which converts the control commands A into mechanical movements or changes in physical quantities. For example, the actuator unit 10 can convert the control command A into an electrical, hydraulic, pneumatic, thermal, magnetic, and / or mechanical movement or change. Specific, but non-limiting, examples include electric motors, electroactive polymers, hydraulic cylinders, piezoelectric actuators, pneumatic actuators, servomechanisms, electromagnets, stepper motors, etc.

[0043] The control system 40 receives the stream of sensor signals S from the sensor 30 in an optional receiving unit 50. The receiving unit 50 converts the sensor signals S into input signals. If no receiving unit 50 is present, any sensor signal S can also be used directly as an input signal. The input signal can, for example, be provided as an extract from the sensor signal S. Alternatively, the sensor signal S can be processed to yield an input signal. The input signal comprises image data corresponding to an image captured by the sensor 30. In other words, the input signal is provided according to the sensor signal S.

[0044] The input signal is then forwarded to a network provided by step S24, which may, for example, be an artificial neural network. The provided network may be a classifier 60.

[0045] The classifier 60 is parameterized by parameters φ, which are stored in and provided by the parameter memory St1.

[0046] The classifier 60 determines output signals y from the input signals x. The output signal y includes information that assigns one or more labels to the input signal. The output signals y are transmitted to an optional converter unit 80, which converts the output signals y into the control commands A. The actuator control commands A are then transmitted to the actuator unit 10 to control the actuator unit 10 accordingly. Alternatively, the output signals y can also be used directly as control commands A.

[0047] The actuator unit 10 receives actuator control commands A, is controlled accordingly, and executes an action corresponding to the actuator control commands A. The actuator unit 10 may include control logic that converts the actuator control command A into another control command, which is then used to control the actuator 10.

[0048] In further embodiments, the control system 40 may include a sensor 30. In still further embodiments, the control system 40 may alternatively or additionally include an actuator 10.

[0049] In one embodiment, the classifier 60 can be configured to detect lanes on a road ahead, e.g., by classifying a road surface and markings on the road and detecting lanes as areas of the road surface between the markings. Based on an output from a navigation system, a suitable lane for taking a chosen path can then be selected, and depending on a current lane and the target lane, a decision can then be made as to whether the vehicle 60 should change lanes or remain in the current lane. The control command A can then be calculated, e.g., by retrieving a predefined movement pattern from a database that corresponds to the identified action.

[0050] Likewise, when detecting road signs or traffic lights, depending on a detected type of road sign or a detected state of the traffic light, corresponding restrictions for possible movement patterns of the vehicle 60 can be retrieved, for example, from a database, a future path of the vehicle 60 that corresponds to the restrictions can be calculated, and the actuator control command A can be calculated to steer the vehicle so that the respective trajectory is executed.

[0051] Similarly, when detecting pedestrians and / or vehicles, an expected future behavior of the pedestrians and / or vehicles can be estimated, and based on the estimated future behavior, a trajectory can then be selected to avoid a collision with the pedestrian and / or the vehicle, and the actuator control command A can be calculated to steer the vehicle to execute the trajectory.

[0052] In still further embodiments, it may be conceivable that the control system 40 controls a display 10a instead of an actuator 10, wherein the display 10a may display the control command or the like.

[0053] In other embodiments, the display 10a may be an output interface to a playback device, such as a display, a light source, a speaker, a vibration motor, etc., which may be used to generate a sensory output signal based on the output of the classifier 60. The sensory output signal may be directly indicative of the classification of the classifier 60, but may also represent a derived sensory output signal, e.g., for use in guidance, navigation, or other types of control of a computer-controlled system.

[0054] Furthermore, the control system 40 may include a processor 45 (or multiple processors) and at least one machine-readable storage medium 46 having stored thereon instructions that, when executed, cause the control system 40 to perform a method according to an aspect of the invention.

[0055] In a preferred embodiment of Fig. 4, the control system 40 is used to control the actuator, which is an at least partially autonomous robot, e.g., an at least partially autonomous vehicle 100.

[0056] Sensor 30 may include one or more video sensors, one or more radar sensors, one or more ultrasonic sensors, one or more LiDAR sensors, and / or one or more position sensors (such as GPS). Some or all of these sensors are preferably, but not necessarily, integrated into vehicle 100. Alternatively or additionally, the sensor 30 may include an information system for determining a state of the actuator system. An example of such an information system is a weather information system that determines a current or future state of the weather in the environment 20.

[0057] For example, using the input signal, classifier 60 can detect objects in the vicinity of the at least partially autonomous robot. The output signal y can include information characterizing the location of objects in the vicinity of the at least partially autonomous robot. The control command A can then be determined in accordance with this information, e.g., to avoid collisions with the detected objects.

[0058] The actuator unit 10, which is preferably integrated into the vehicle 100, can be provided by a brake, a drive system, an engine, a drive train, or a steering system of the vehicle 100. Actuator control commands A can be determined such that the actuator unit (or actuators) 10 are controlled such that the vehicle 100 avoids collisions with the detected objects. Detected objects can also be classified according to what the classifier 60 most likely classifies them as, e.g., pedestrians or trees, and actuator control commands A can be determined depending on the classification.

[0059] In further embodiments, the at least partially autonomous robot can be another mobile robot (not shown) that can move, for example, by flying, swimming, diving, or walking. The mobile robot can be, among other things, an at least partially autonomous lawnmower or an at least partially autonomous cleaning robot. In all of the above-mentioned embodiments, the actuator control command A can be determined such that the drive unit and / or the steering and / or the brake of the mobile robot is / are controlled such that the mobile robot can avoid collisions with the detected objects.

[0060] In a further embodiment, the at least partially autonomous robot can be a gardening robot (not shown) that uses a sensor 30, preferably an optical sensor, to determine the condition of plants in the environment 20. The actuator unit 10 can be a nozzle for spraying chemicals. Depending on the detected type and / or condition of the plants, an actuator control command A can be determined to cause the actuator unit 10 to spray the plants with an appropriate amount of suitable chemicals.

[0061] In yet further embodiments, the at least partially autonomous robot can be a household appliance (not shown), such as a washing machine, a stove, an oven, a microwave, or a dishwasher. The sensor 30, e.g., an optical sensor, can detect the state of an object to be processed by the household appliance. If the household appliance is a washing machine, for example, the sensor 30 can detect a state of the laundry in the washing machine. The actuator control signal A can then be determined depending on a detected material of the laundry.

[0062] In Fig. Figure 5 shows an embodiment in which the control system 40 is used to control a manufacturing machine 11, e.g., a soldering machine, a punch, a cutting device, or a drill gun, of a manufacturing system 200, e.g., as part of a production line. The control system 40 controls an actuator unit 10, which in turn controls the manufacturing machine 11.

[0063] The sensor 30 can be an optical sensor that detects the properties, for example, of a manufactured product 12. The classifier 60 can determine a state of the manufactured product 12 from these detected properties. The actuator unit 10, which controls the production machine 11, can then be controlled for a subsequent production step of the manufactured product 12 depending on the determined state of the manufactured product 12. Or it is conceivable that the actuator unit 10 is controlled during the production of a subsequent manufactured product 12 depending on the determined state of the manufactured product 12.

[0064] In Fig. 6 shows an embodiment in which the control system controls an access control system 300. The access control system may be designed for physical access control. For example, it may comprise a door 401. The sensor 30 is designed to detect a scene relevant to the decision as to whether or not access should be granted. For example, it may be an optical sensor providing image or video data to detect a person's face. The classifier 60 may be designed to interpret this image or video data, for example, by comparing matches with known persons stored in a database and thus determining an identity of the person. The actuator control signal A may then be determined depending on the interpretation of the classifier 60, for example, according to the determined identity.The actuator unit 10 can be a lock that allows or denies access depending on the actuator control signal A. Non-physical, logical access control is also possible.

[0065] In Fig. 7 shows an embodiment in which the control system 40 controls a monitoring system 400. This embodiment is largely identical to that shown in Fig. 5. Therefore, only the differing aspects will be described in detail here. The sensor 30 is designed to detect a scene to be monitored. The control system does not necessarily control an actuator 10, but rather a display 10a. For example, the machine learning system 60 can determine a classification of a scene, e.g., whether the scene detected by the optical sensor 30 is suspicious. The actuator control signal A, which is transmitted to the display 10a, can then be designed, e.g., to cause the display 10a to adapt the displayed content depending on the determined classification, e.g., to highlight an object that the machine learning system 60 classifies as suspicious.

[0066] In Fig. Figure 8 shows an embodiment in which the control system 40 is used to control an automated personal assistant 250. The sensor 30 may be an optical sensor, for example, receiving video images of gestures from the user 249. Alternatively, the sensor 30 may also be an acoustic sensor, for example, receiving a voice command from the user 249.

[0067] The control system 40 then determines the actuator control commands A for controlling the automated personal assistant 250. The actuator control commands A are determined according to the sensor signal S of the sensor 30. The sensor signal S is sent to the control system 40. The classifier 60 may, for example, be configured to execute a gesture recognition algorithm to identify a gesture of the user 249. The control system 40 may then determine an actuator control command A for transmission to the automated personal assistant 250. It then transmits this actuator control command A to the automated personal assistant 250.

[0068] For example, the actuator control command A may be determined according to the user gesture recognized by the classifier 60. It may then include information that causes the automated personal assistant 250 to retrieve information from a database and output this retrieved information in a form suitable for receipt by the user 249.

[0069] In further embodiments, it is conceivable that the control system 40, instead of the automated personal assistant 250, controls a household appliance (not shown) that is controlled according to the identified user gesture. The household appliance may be a washing machine, a stove, an oven, a microwave, or a dishwasher.

[0070] In Fig. Figure 9 shows an embodiment of a control system 40 for controlling an imaging system 500, such as an MRI scanner, an X-ray scanner, or an ultrasound scanner. Sensor 30 may, for example, be an imaging sensor. Machine learning system 60 may then classify all or part of the acquired image. Actuator control signal A may then be selected according to this classification, thereby controlling display 10a. For example, machine learning system 60 may interpret a region of the acquired image as potentially abnormal. In this case, actuator control signal A may be determined to cause display 10a to display the imaging and highlight the potentially abnormal region.

[0071] In Fig. 10, an embodiment of a training system 500 is shown. The training device 500 consists of a provision system 51 that provides input images from a training data set. The input images are fed to the neural network 52 to be trained, which determines output variables therefrom. Output variables and input images are fed to an evaluator 53, which determines current hyperparameters therefrom, which are transferred to the parameter memory P, where they replace the current parameters. The evaluator 53 is designed to perform the steps of Fig. 3 to be executed.

[0072] The methods performed by the training device 500 may be implemented as a computer program stored on a machine-readable storage medium 54 and executed by a processor 55.

[0073] The term "computer" encompasses any device for processing predefined computational instructions. These computational instructions may be in the form of software, hardware, or a combination of software and hardware. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited non-patent literature

[0000] He, Kaiming, et al. “Deep residual learning for image recognition.” Proceedings of the IEEE conference on computer vision and pattern recognition, 2016

[0002] Jha et al. “Deepreduce: ReLU reduction for fast private inference” (ICML 2021

[0005] Vasu et al. „MobileOne: An Improved One Millisecond Mobile Backbone“ (CVPR 2023

[0006] Yu et al. „NetBooster: Empowering Tiny Deep Learning By Standing on the Shoulders of Deep Giants“, DAC 2023

[0007]

Claims

[1] A computer-implemented method (20) for reparameterizing a residual network (M), wherein the residual network (M) is a pre-trained neural network comprising residual connections that skip residual blocks, comprising the following steps: Receiving (S22) a base power (P) of the residual network (M) on a first data set, Executing a first loop (S23) while a reduction in an application performance (R) with respect to the base performance (P) is less than a given tolerable reduction (δ), the first loop comprising the following steps: a. Selecting a residual block (b ∈ M) of the residual blocks for reparameterization, b. Execute a second loop over a set i ∈ ε, 2ε, ...,1, where ε is a step size less than 1: i. Replacing non-linear activation functions f j (x) ∈ b by the new function f j (x) = (1 - ε) ∗ f j(x) + ε ∗ x, and ii. performing retraining of the linearized residual network (M) on a second dataset; c. Reparameterize residual block b with linear activation functions into one or more layers and d. Evaluate the application performance (P') of the residual network (M) using the first data set D v and updating the performance degradation (R) depending on the application performance (P'). [2] The method of claim 1, wherein the step of selecting a residual block (b ∈ M) is performed depending on an estimated impact of the residual blocks on the application performance (P') and / or the residual blocks on the hardware efficiency. [3] Method according to claim 2, wherein the estimated impact of the residual blocks on the application performance (P') is estimated by evaluating the base application performance (P), wherein for each residual block that has not yet been reparameterized, the non-linear activation functions (f j (x)) in the residual block by an identity function, where the application performance (P b ) of the modified residual network with identity function is evaluated, reversing the change in the activation functions, with the effect of the modified residual block on the application performance being the difference between the baseline performance and the application performance (P b ) of the modified residual network (P b ) is estimated. [4] Method according to claim 2 or 3, wherein the estimated impact of the residual blocks on the application performance (P') is estimated by the following steps: For each residual block that has not yet been reparameterized, initialize a variable (s b ) to indicate a sensitivity, Replacing the non-linear activation functions (f j (x)) in the residual blocks by the identity function g j (x, s b ) = (1 - s b )f j (x) + s b x; initially the variable (s b ) is set to zero, Calculating gradients according to: ∂ / ∂s b L(D,M), where L(D, M) is a loss function of the training of the residual network (M) for a given training data set (D), Undoing the change in the activation functions and determining the effect of the modified residual block on the application performance based on the value of the variable (s b ). [5] A method according to any one of claims 2 to 4, wherein the estimated impact of the residual blocks on the hardware efficiency is estimated by the following steps: for each residual block that has not yet been reparameterized, calculating the size of a residual feature map by counting the number of pixels, wherein the impact of the residual block on the hardware efficiency is estimated as the size of the residual map. [6] A method according to any one of claims 2 to 5, wherein the estimated impact of the residual blocks on the hardware efficiency is estimated by the following steps: for each residual block that has not yet been reparameterized, a latency for the inference of the residual blocks is determined, wherein the impact of the residual block on the hardware is estimated as the latency. [7] Method according to one of the preceding claims, wherein the estimated impact of the residual blocks on the hardware efficiency is estimated by the following steps: for each residual block that has not yet been reparameterized, determining a first latency (l orig ) for the inference of the residual block and determining a second latency (l reparametrized ) for the inference of the residual block after it has been reparameterized, where the effect of the residual block on hardware efficiency is defined as the difference between the first latency (l orig ) and the second latency (l reparametrized ) is estimated. [8] A computer-implemented method for using a reparameterized residual network for classifying sensor signals, wherein the reparameterized residual network is reparameterized using the method according to any one of claims 1 to 7, comprising the following steps: - receiving a sensor signal (S) comprising data from a sensor (30), - Determining an input signal (x) which depends on the sensor signal (S), and - feeding the input signal (x) into the classifier (60) to obtain an output signal (y) which characterizes a classification of the input signal (x). [9] A computer-implemented method for using the reparameterized residual network reparameterized by the method according to any one of claims 1 to 7 to provide an actuator control signal (A) for controlling an actuator (10), comprising all the steps of the method according to claim 8, and further comprising the step of: - Determining the actuator control signal (A) depending on the output signal (y). [10] Method according to claim 9, wherein the actuator (10) controls an at least partially autonomous robot (100) and / or a manufacturing machine (200) and / or an access control system (300). [11] A computer program designed to cause a computer to carry out the method according to any one of claims 1 to 10 with all its steps when the computer program is executed by a processor (45, 145). [12] A machine-readable storage medium (46, 146) on which a computer program according to claim 11 is stored. [13] System adapted to carry out the method according to any one of claims 1 to 10.

Images