Apparatus and method for re-parameterizing residual network for computational efficiency
By reparameterizing the residual network, the residual connection is converted into a non-residual layer sequence, the problem of memory traffic bottleneck in the inference process is solved, and hardware efficiency and network performance are improved.
Patent Information
- Application Number
- CN202510196293.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-02-23
- Filing Date
- 2025-02-21
- Publication Date
- 2025-08-26
AI Technical Summary
While existing residual connections improve training efficiency in deep neural networks, they lead to memory traffic bottlenecks during inference and reduce hardware efficiency.
By reparameterizing the residual network, the residual connections are converted into non-residual layer sequences, the residual connections are deleted and the residual blocks are converted into pure feedforward layers, keeping the network performance basically unchanged.
Without increasing computing and data resources, the inference efficiency is improved, memory traffic is reduced, and network performance is basically unchanged.
Smart Images
Figure CN120542484A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a method of reparameterizing a residual network and a method for operating an actuator by means of a reparameterized residual network, a computer program and a machine-readable storage medium, and a system. Background Art
[0002] A residual neural network (also known as a residual network, ResNet, see He, Kaiming et al., "Deep residual learning for image recognition", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016) is a deep learning model in which the weights of a layer learn a residual function with respect to the layer input. From a practical point of view, a residual network is a network with at least one skip connection (also known as a residual connection) that performs identity mapping by merging the output of the skipped layer(s) by addition.
[0003] Residual connections are widely used in modern deep neural networks because they improve the trainability of the network. However, residual connections also greatly reduce the hardware efficiency of network inference. The reason is that the forward activations through the residual connection need to be saved in memory when computing the residual block. A residual block can be (one or more) skipped layers through the residual connection. Typically, the activations need to be stored in DRAM and loaded back to compute the addition. This results in additional load / store operations, causing memory traffic to become a bottleneck. Therefore, residual connections are beneficial during training, which is why they are widely used, but introduce inefficiencies during inference.
[0004] There are known methods in the literature that exploit the advantages of residual connections during training and provide methods to reduce or overcome the disadvantages of residual connections during inference.
[0005] Jha et al.'s work, "Deepreduce: ReLU reduction for fast private inference" (ICML 2021), proposes using importance scores to heuristically remove nonlinear activation functions (such as ReLUs) from neural networks (NNs). The importance scores are calculated by training various variations of the NN with ReLU operations removed from different stages and observing the accuracy. The first optimization step removes all ReLU operations from some stages. The second step removes every other ReLU from selected other stages.
[0006] The work "MobileOne: An Improved One Millisecond MobileBackbone" by Vasu et al. (CVPR 2023) proposes designing a special NN architecture that uses residual connections during training but reparameterizes them before inference into a residual-free network while computing the same mathematical function. This is only possible by avoiding nonlinear operations in residual blocks, unlike established and widely used networks such as ResNet, MobileNet, or EfficientNet architectures.
[0007] The work of Yu et al., "NetBooster: Empowering Tiny Deep Learning By Standing on the Shoulders of Deep Giants", DAC 2023, does not aim to reduce the complexity of neural networks, but rather to improve their training by first inflating the network (replacing a single layer with several layers with nonlinear functions in between), then training the inflated network, and finally reducing the architecture back to the original topology. The final step is accomplished by progressive linearization of the newly introduced nonlinear activation function: progressively interpolating between the activation function and the identity function until the activation function can finally be removed and the layers can be combined.
[0008] A disadvantage of known methods is that they either require extensive computational and data resources to train many variants of the NN in order to compute importance scores, or require the developer to already use special non-standard neural network architectures during training.
[0009] Advantages of the present invention
[0010] The present invention (data and computation) efficiently removes residual connections in trained neural network architectures for efficient inference while requiring only minimal computational and data resources. Summary of the Invention
[0011] In a first aspect, the present invention relates to a computer-implemented method for reparameterizing a residual network. The residual network may be a pretrained neural network including residual connections that skip residual blocks. Reparameterization may be understood as reconfiguring the residual connections and residual blocks into non-residual layers or a sequence of non-residual layers. In other words, reparameterization may include deleting at least the residual connections and transforming the corresponding residual blocks of the deleted residual connections into a sequence of pure feedforward layers or into a single layer, wherein the sequence of layers or the layer may be regularly modified so that it effectively performs substantially the same computations as a residual block combined with a residual connection. Thus, the parameters, in particular the weights, of the residual block are reparameterized so that the performance of the residual network is not substantially degraded when the residual connection is not present, or substantially outputs activations similar to or identical to those with the residual connection. The reparameterized residual block may include several original layers, all or part of which have been reparameterized, or the original layers of the residual block have been converted into a new layer that performs substantially the same computations as a residual block combined with a residual connection.
[0012] The method begins by evaluating the baseline performance of the residual network on a first dataset. The first dataset is preferably a small dataset. The term "small" should be understood as meaning that the dataset should be large enough to obtain a reasonable estimate of performance. This may require at least hundreds of images, more likely thousands. Even so, this is still small compared to conventional training datasets containing millions of images.
[0013] The following step is to perform the first loop when the degradation of the application performance relative to the baseline performance is less than a given tolerable degradation (eg, a few percent, such as 1%-5%).
[0014] The loop includes the following steps: Select a residual block b from the residual blocks for reparameterization. Implement the second loop on the set i∈∈, 2∈, ..., 1, where ∈ is the step size, preferably expected to be 0.005<∈<0.1: Use the new function f j (x)=(1-∈)*f j (x)+∈ *x replaces all nonlinear activation functions f j (x)∈b, and retrain the linearized residual network on the second dataset.
[0015] The next step is to reparameterize the selected residual block b into a sequence of feedforward layers or a single layer, such as those known from MobileOne and NetBooster. Note that progressive linearization can be used to remove nonlinear activations. Linear operations can be reparameterized, including the most common operations such as convolution, fully connected layers, or batch normalization. The next step is to evaluate the application performance on the first dataset and update the performance degradation depending on the application performance.
[0016] If the performance degradation is greater than the tolerable degradation, the current reparameterization is reversed and the first cycle is terminated.
[0017] Finally, the reparameterized residual network can be output or deployed on the target device.
[0018] The following steps are proposed: selecting a residual block is performed based on an estimated impact of the residual block on application performance and / or an estimated impact of the residual block on hardware efficiency, such as the estimated impact on the target device used for inference. If the impact is less than a predefined threshold, the residual block is selected for reparameterization. The predefined threshold can be defined depending on the use case of the residual network. The impact on application performance and the impact on hardware efficiency can be combined into a single metric for residual block selection. This can be done using a simple scalar function (such as addition or multiplication) applied to the two impacts.
[0019] In another aspect of the present invention, a computer-implemented method is provided for using a reparameterized residual network as a classifier for classifying sensor signals. The classifier employs the method according to any of the preceding aspects of the present invention, comprising the steps of receiving a sensor signal comprising data from an imaging sensor, determining an input signal dependent on the sensor signal, and feeding the input signal into the classifier to obtain an output signal representative of the classification of the input signal.
[0020] In another aspect of the present invention, a computer-implemented method is provided for using a classifier trained to provide an actuator control signal for controlling an actuator. The actuator control signal is determined based on the class of the output signal, which can be determined as described in the previous section. Actuators are provided for controlling at least partially autonomous robots and / or manufacturing machines and / or access control systems.
[0021] In another aspect of the present invention, a control system for operating an actuator is provided, wherein the control system comprises a classifier according to any of the above aspects of the present invention and is configured to operate the actuator according to an output of the classifier. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Embodiments of the present invention will be discussed in more detail with reference to the following figures. The figures show:
[0023] Figure 1 Schematic diagram of different layers fused into one final layer;
[0024] Figure 2 Schematic residual block as part of a residual neural network;
[0025] Figure 3 Flowchart of an embodiment of the present invention;
[0026] Figure 4 a control system for controlling an at least partially autonomous robot;
[0027] Figure 5 control systems that control manufacturing machines;
[0028] Figure 6 Control systems that control access control systems;
[0029] Figure 7 Control systems that control surveillance systems;
[0030] Figure 8 control systems for controlling automated personal assistants;
[0031] Figure 9 a control system for controlling the imaging system;
[0032] Figure 10 Training system. DETAILED DESCRIPTION
[0033] Layer fusion and reparameterization are known techniques, for example, consecutive convolution and batch normalization (BN) layers can be fused into a single convolution layer since BN only performs linear operations at inference time. Figure 1 Schematic diagram showing the fusion of several convolutional layers Conv2D and batch normalization (BN) layers into a single Conv2D layer (1a). Similarly, consecutive convolutional layers can be fused into a single Conv2D by combining the corresponding kernels into a new (larger) kernel.
[0034] The residual branch without any nonlinear activation function can be reparameterized (1b) into a single layer by modifying the kernel of the convolution operation accordingly.
[0035] Figure 2 A portion of a residual network is shown as an example. The input activation X of the previous portion of the residual network, in particular the input activation X of the previous layer of the residual network, is propagated through a residual block (2a) to receive an output activation Y. The residual block (2a) includes a residual block, a residual connection, and an addition layer. The residual connection forwards the input activation X directly to the addition layer, thereby skipping the layers of the residual block. The residual block includes one or more layers. These layers can be all possible layers used in existing deep neural networks. Figure 2 The convolutional layer, ReLU layer, and batch normalization layer are shown as examples. The output of the residual block and the forwarded input activation are merged in the addition layer, which outputs the output activation Y. The merging operation of the addition layer can be a simple addition of the two inputs of the addition layer.
[0036] One goal of the present invention is to reparameterize some residual connections in residual networks so as to preferably achieve a Pareto optimal trade-off between application performance (e.g., maximizing accuracy, minimizing regression error) and hardware efficiency for inference (e.g., minimizing memory traffic), given a trained neural network using residual connections and a limited dataset is available.
[0037] Typically, this objective is formulated as a constrained optimization, i.e., maximizing hardware efficiency while maintaining a specific application performance level. The main challenge is how to identify which residual blocks should be reparameterized.
[0038] Figure 3 The method (20) for reparameterizing the residual network is exemplarily shown in an algorithmic manner.
[0039] In a first step (S21), relevant inputs to the method are obtained, wherein the following items may be given:
[0040] aM: pre-trained model using residual connections,
[0041] bD t : Potentially small labeled training datasets for short-term retraining,
[0042] cL: loss function used for training,
[0043] dD v : a potentially small labeled validation dataset to estimate application performance (to get meaningful application performance numbers this will likely be at least hundreds to thousands of images, but could be larger),
[0044] e.δ: The acceptable degradation in application performance, and
[0045] f.∈: step size for the gradual linearization (a small number, as exemplified above).
[0046] In the second step (S22), an evaluation is performed. The baseline application performance of M is evaluated on a small training dataset and stored as P.
[0047] In the next step (S23), when the application performance degradation R is less than the allowable degradation δ, a first loop is implemented. The first loop includes the following steps:
[0048] 1. Identify the residual block b∈M to be linearized.
[0049] 2. Implement the second loop on the set l∈∈, 2∈, ..., 1:
[0050] a. Use the new function f j (x)=(1-∈)*f j(x)+∈*x replaces all nonlinear activation functions f j (x)∈b is used for linearization.
[0051] b. Use D t and L performs short-term retraining on M.
[0052] 3. Reparameterize block b into a single convolutional layer. This is possible because block b no longer has nonlinear operations. MobileNet or NetBooster can be used.
[0053] 4. Use D v Evaluate the application performance of M and store it as P′.
[0054] 5. Update performance degradation: R←PP′.
[0055] After step S23 has terminated, an optional step of deploying (S24) the re-parameterized network to a target device for inference is performed.
[0056] On the target device, the deployed network can be used to control (S25) applications, e.g. Figures 4 to 9 As shown in FIG.
[0057] A key challenge is how to identify the block b∈M to be linearized, which is the first step of the S23 loop. This requires estimating the impact of block b on application performance and / or the impact of block b on hardware efficiency, thus taking into account hardware limitations.
[0058] There are several alternatives to compute these metrics. Combinations are possible, for example by estimating that the hardware cost of a block can be computed as "normalized size of feature map + normalized latency". More generally, these individual scalar estimates can be combined using scalar operations such as (weighted) averaging.
[0059] In a first variant for calculating the metric of impact on an application, the metric is estimated by simple profiling. It comprises the following steps:
[0060] 1) Evaluate the baseline application performance of M and store it as P.
[0061] 2) For each block b∈M that has not been reparameterized:
[0062] a. Use the identity function g j (x) = x instead of the nonlinear activation function f j (x)∈b.
[0063] b. Use D v Evaluate the application performance of M and store it as P b .
[0064] c. Restore the changes in activation function.
[0065] 3) Estimate the impact of data block b on application performance as PP b .
[0066] In a second variant for calculating the measure of impact on the application, the measure is estimated by a sensitivity analysis. It comprises the following steps:
[0067] 1) For each block b∈M that has not been reparameterized:
[0068] a. Introduce a new variable S b To indicate sensitivity.
[0069] b. Use the identity function g j (x,s b )=(1-s b )*f j (x)+s b *x replaces the nonlinear activation function f j (x)∈b.
[0070] c. Initialize s b = 0, so that g j (x,s b )=f j (x).
[0071] d. Calculate gradients
[0072] e. Restore changes to the activation function
[0073] 2) Through s b Estimate the impact of block b on application performance.
[0074] In a first variant for calculating the metric of the impact on hardware efficiency, the metric is estimated by the size of the residual feature map. It comprises the following steps:
[0075] 1) For each block b∈M that has not been reparameterized:
[0076] a. By |x b |Calculate the residual feature map x b The size of |x b | represents x b The cardinality of .
[0077] 2) Estimate the impact of block b on hardware efficiency as |x b |.
[0078] In a second variant for calculating the metric of the impact on hardware efficiency, the metric is estimated by profiling the block on real hardware / in simulation. It comprises the following steps:
[0079] 1) For each block b∈M that has not been reparameterized:
[0080] a. Profile the latency l of inference for this block.
[0081] 2) Estimate the impact of block b on hardware efficiency as l.
[0082] In a third variant for calculating the metric of the impact on hardware efficiency, the metric is estimated by profiling the block on real hardware / in simulation in comparison with the reparameterized block. It comprises the following steps:
[0083] 1) For each block b∈M that has not been reparameterized:
[0084] a. Analyze the latency of inference for this block l orig .
[0085] b. If reparameterized, profile the latency l for inference on this block reparametrized (This only requires knowing the reparameterized architecture, not the weights).
[0086] 2) Estimate the impact of block b on hardware efficiency as l orig -l reparametrized .
[0087] Finally, the impact on application performance and the impact on hardware efficiency can be combined into a single metric for block selection. This can be done using simple scalar functions such as addition or multiplication.
[0088] Figure 4 is an embodiment of an actuator with a control system 40. The actuator and its environment will be collectively referred to as the actuator system. Sensors 30 sense conditions of the actuator system at preferably evenly spaced distances. Sensors 30 may include multiple sensors. Preferably, sensors 30 are optical sensors that capture images of the environment. An output signal S of sensor 30 (or, if sensor 30 includes multiple sensors, an output signal S of each sensor) encoding the sensed condition is transmitted to control system 40.
[0089] Thus, the control system 40 receives a stream of sensor signals S. It then calculates a series of actuator control commands A based on the sensor signals S, and then transmits the control commands A to the actuator unit 10, which converts the control commands A into mechanical motion or changes in physical quantities. For example, the actuator unit 10 may convert the control commands A into electrical, hydraulic, pneumatic, thermal, magnetic, and / or mechanical motion or changes. Specific, but non-limiting, examples include electric motors, electroactive polymers, hydraulic cylinders, piezoelectric actuators, pneumatic actuators, servomechanisms, solenoids, stepper motors, and the like.
[0090] The control system 40 receives a stream of sensor signals S from the sensor 30 in an optional receiving unit 50. The receiving unit 50 converts the sensor signals S into input signals. Alternatively, in the absence of the receiving unit 50, each sensor signal S can be directly treated as an input signal. The input signal can be provided, for example, as an excerpt from the sensor signal S. Alternatively, the sensor signal S can be processed to generate the input signal. The input signal includes image data corresponding to the image recorded by the sensor 30. In other words, the input signal is provided based on the sensor signal S.
[0091] The input signal is then passed to the deployed network of step S24 , which may be, for example, an artificial neural network. The deployed network may be a classifier 60 .
[0092] The classifier 60 is parameterized by parameters φ stored in and provided by the parameter storage means St1 .
[0093] The classifier 60 determines the output signal y from the input signal x. The output signal y includes information for assigning one or more labels to the input signal. The output signal y is transmitted to an optional conversion unit 80, which converts the output signal y into a control command A. The actuator control command A is then transmitted to the actuator unit 10 for controlling the actuator unit 10 accordingly. Alternatively, the output signal y can be directly regarded as the control command A.
[0094] The actuator unit 10 receives the actuator control command A, is controlled accordingly, and performs an action corresponding to the actuator control command A. The actuator unit 10 may include control logic that transforms the actuator control command A into another control command, which is then used to control the actuator 10 .
[0095] In further embodiments, the control system 40 may include the sensor 30. In still further embodiments, the control system 40 may include the actuator 10, alternatively or additionally.
[0096] In one embodiment, the classifier 60 can be designed to identify lanes on the road ahead, for example by classifying the road surface and markings on the road, and identifying lanes as patches of the road surface between the markings. Based on the output of the navigation system, a suitable lane can then be selected for following the chosen path, and depending on the current lane and the target lane, a decision can then be made as to whether the vehicle 60 should switch lanes or remain in the current lane. The control command A can then be calculated by, for example, retrieving a predefined motion pattern from a database corresponding to the identified maneuvers.
[0097] Likewise, when a road sign or a traffic light is identified, depending on the identified type of road sign or the identified state of the traffic light, corresponding constraints on possible motion modes of the vehicle 60 can then be retrieved from, for example, a database, a future path of the vehicle 60 commensurate with the constraints can be calculated, and the actuator control command A can be calculated to steer the vehicle so as to execute the trajectory.
[0098] Similarly, when pedestrians and / or vehicles are identified, the expected future behavior of the pedestrians and / or vehicles can be estimated, and based on the estimated future behavior, a trajectory can then be selected to avoid collision with the pedestrians and / or vehicles, and the actuator control command A can be calculated to steer the vehicle to execute the trajectory.
[0099] In still further embodiments, it is contemplated that the control system 40 controls the display 10a instead of the actuator 10, wherein the display 10a may display control commands or the like.
[0100] In other embodiments, the display 10a may be an output interface to a presentation device, such as a display, a light source, a speaker, a vibration motor, etc., which may be used to generate a sensory-perceptible output signal, which may be generated based on the output of the classifier 60. The sensory-perceptible output signal may directly indicate the classification of the classifier 60, but may also represent a derived sensory-perceptible output signal, such as for guidance, navigation, or other types of control of a computer-controlled system.
[0101] Furthermore, the control system 40 may include a processor 45 (or multiple processors) and at least one machine-readable storage medium 46 having stored thereon instructions that, if executed, cause the control system 40 to implement a method according to an aspect of the present invention.
[0102] exist Figure 4 In a preferred embodiment, the control system 40 is used to control an actuator that is an at least partially autonomous robot, such as an at least partially autonomous vehicle 100 .
[0103] The sensors 30 may include one or more video sensors and / or one or more radar sensors and / or one or more ultrasonic sensors and / or one or more lidar sensors and / or one or more position sensors (such as GPS, for example). Some or all of these sensors are preferably, but not necessarily, integrated into the vehicle 100.
[0104] Alternatively or additionally, the sensor 30 may include an information system for determining the state of the actuator system. One example of such an information system is a weather information system that determines the current or future state of the weather in the environment 20.
[0105] For example, using the input signal, the classifier 60 can detect an object in the vicinity of the at least partially autonomous robot. The output signal y can include information indicating where the object is located in the vicinity of the at least partially autonomous robot. A control command A can then be determined based on this information, for example, to avoid a collision with the detected object.
[0106] The actuator unit 10, preferably integrated into the vehicle 100, can be provided by the brakes, propulsion system, engine, transmission, or steering of the vehicle 100. An actuator control command A can be determined such that the actuator unit (or multiple actuator units) 10 are controlled so that the vehicle 100 avoids a collision with the detected object. The detected objects can also be classified according to what the classifier 60 considers them to be most likely, such as pedestrians or trees, and the actuator control command A can be determined depending on the classification.
[0107] In another embodiment, the at least partially autonomous robot can be provided by another mobile robot (not shown), which can move, for example, by flying, swimming, diving, or walking. The mobile robot can in particular be an at least partially autonomous lawn mower or an at least partially autonomous cleaning robot. In all the above embodiments, the actuator command control A can be determined so that the propulsion unit and / or the steering device and / or the brakes of the mobile robot are controlled so that the mobile robot can avoid a collision with the identified object.
[0108] In another embodiment, the at least partially autonomous robot may be a gardening robot (not shown) that uses a sensor 30 (preferably an optical sensor) to determine the state of plants in the environment 20. The actuator unit 10 may be a nozzle for spraying chemicals. Depending on the identified type and / or identified state of the plant, the actuator control command A may be determined to cause the actuator unit 10 to spray the plant with an appropriate amount of an appropriate chemical.
[0109] In yet another embodiment, the at least partially autonomous robot can be provided by a household appliance (not shown), such as a washing machine, stove, oven, microwave, or dishwasher. A sensor 30 (e.g., an optical sensor) can detect the state of an object to be processed by the household appliance. For example, if the household appliance is a washing machine, the sensor 30 can detect the state of the laundry inside the washing machine. The actuator control signal A can then be determined based on the detected laundry material.
[0110] Figure 5 , an embodiment is shown in which a control system 40 is used to control a manufacturing machine 11 (e.g., a solder placement machine, a punch cutter, a cutting machine, or a gun drill) of a manufacturing system 200 (e.g., as part of a production line). The control system 40 controls the actuator unit 10, which in turn controls the manufacturing machine 11.
[0111] The sensor 30 may be an optical sensor that captures, for example, properties of the manufactured product 12. The classifier 60 may determine the state of the manufactured product 12 from these captured properties. The actuator unit 10 controlling the manufacturing machine 11 may then be controlled, depending on the determined state of the manufactured product 12, for subsequent manufacturing steps of the manufactured product 12. Alternatively, it is conceivable that the actuator unit 10 may be controlled during the subsequent manufacturing of the manufactured product 12, depending on the determined state of the manufactured product 12.
[0112] Figure 6 , an embodiment is shown in which the control system controls an access control system 300. The access control system can be designed to physically control access. For example, it can include a door 401. The sensor 30 is configured to detect a scene that is relevant to the decision whether access is to be granted. For example, it can be an optical sensor for providing image or video data for detecting a face. The classifier 60 can be configured to interpret the image or video data, for example by matching the identity with known persons stored in a database, and thereby determine the identity of the person. Depending on the interpretation of the classifier 60, for example, depending on the determined identity, an actuator control signal A can then be determined. The actuator unit 10 can be a lock that grants or does not grant access depending on the actuator control signal A. Non-physical, logical access control is also possible.
[0113] Figure 7 An embodiment is shown in FIG. 4 where a control system 40 controls a monitoring system 400. This embodiment is largely similar to Figure 5. Therefore, only the different aspects will be described in detail. The sensor 30 is configured to detect the monitored scene. The control system does not necessarily control the actuator 10, but rather controls the display 10a. For example, the machine learning system 60 can determine the classification of the scene, such as whether the scene detected by the optical sensor 30 is suspicious. The actuator control signal A transmitted to the display 10a can then, for example, be configured so that the display 10a adjusts the displayed content depending on the determined classification, such as highlighting objects that are considered suspicious by the machine learning system 60.
[0114] Figure 8 2 shows an embodiment in which the control system 40 is used to control the automated personal assistant 250. The sensor 30 may be an optical sensor, for example, for receiving a video image of the gestures of the user 249. Alternatively, the sensor 30 may be an audio sensor, for example, for receiving a voice command of the user 249.
[0115] Control system 40 then determines an actuator control command A for controlling automated personal assistant 250. Actuator control command A is determined based on sensor signal S from sensor 30. Sensor signal S is transmitted to control system 40. For example, classifier 60 can be configured to implement a gesture recognition algorithm to identify gestures made by user 249. Control system 40 can then determine actuator control command A for transmission to automated personal assistant 250. It then transmits actuator control command A to automated personal assistant 250.
[0116] For example, the actuator control command A may be determined from the identified user gesture recognized by the classifier 60. It may then include information that causes the automated personal assistant 250 to retrieve information from a database and output the retrieved information in a form suitable for the user 249 to receive.
[0117] In another embodiment, it is contemplated that the control system 40 controls a home appliance (not shown) that is controlled according to the identified user gesture instead of the automated personal assistant 250. The home appliance may be a washing machine, a stove, an oven, a microwave, or a dishwasher.
[0118] Figure 9, an embodiment of a control system 40 for controlling an imaging system 500, such as an MRI apparatus, an X-ray imaging apparatus, or an ultrasound imaging apparatus, is shown. Sensor 30 may be, for example, an imaging sensor. Machine learning system 60 may then determine a classification for all or part of the sensed image. Based on this classification, actuator control signal A may be selected to control display 10a. For example, machine learning system 60 may interpret a region of the sensed image as potentially abnormal. In this case, actuator control signal A may be determined to cause display 10a to display the image and highlight the potentially abnormal region.
[0119] Figure 10 5 is an embodiment of a training system 500. The training device 500 comprises a provider system 51 which provides input images from a training dataset. The input images are fed to a neural network 52 for training, which determines output variables from them. The output variables and the input images are supplied to an evaluator 53 which determines therefrom sensitive hyperparameters / parameters which are transferred to a parameter memory P where they replace the current parameters. The evaluator 53 is arranged to perform Figure 3 steps.
[0120] The program executed by the training device 500 may be implemented as a computer program stored on the machine-readable storage medium 54 and executed by the processor 55 .
[0121] The term "computer" encompasses any device designed to process predefined computing instructions, whether in the form of software, hardware, or a combination of software and hardware.
Claims
1. A computer-implemented method (20) for reparameterizing a residual network (M), wherein the residual network (M) is a pre-trained neural network including residual connections that skip residual blocks, the method comprising the following steps: receiving (S22) a baseline performance (P) of the residual network (M) on a first data set, When the application performance degradation (R) relative to the baseline performance (P) is less than a given allowable degradation (δ), a first loop (S23) is implemented, wherein: The first cycle includes the following steps: a. Select a residual block (b∈M) from the residual blocks for reparameterization, b. Perform a second loop on the set i∈∈, 2∈, ..., 1, where ∈ is a step size less than 1: i. Use the new function f j (x)=(1-∈)*f j (x)+∈*x instead of nonlinear activation function f j (x)∈b, and ii. retraining the linearized residual network (M) on the second dataset; c. Reparameterize the residual block b into one or several layers using a linear activation function, and d. In the first data set D v The application performance (P') of M is evaluated, and the performance reduction (R) is updated depending on the application performance (P').
2. The method according to claim 1, wherein the step of selecting a residual block (b∈M) is performed depending on an estimated impact of the residual block on application performance (P′) and / or on hardware efficiency.
3. The method according to claim 2, wherein: The estimated impact of the residual block on the application performance (P′) is estimated by evaluating the baseline application performance (P), where for each residual block that has not been reparameterized, the nonlinear activation function (f j (x)), where the application performance of the modified residual network with the identity function is evaluated (P b ), wherein the change of the activation function is restored, wherein the impact of the modified residual block on the application performance is estimated as the modified residual network (P b ) baseline performance and application performance (P b ) between .
4. The method according to claim 2 or 3, wherein the estimated impact of the residual block on the application performance (P′) is estimated by the following steps: For each residual block that has not been reparameterized, initialize the variable indicating the sensitivity (s b ). Using the identity function g j (x,s b )=(1-s b )f j (x)+s b x replaces the nonlinear activation function (f j (x)); Initially, the variable (s b ) is set to zero, according to Calculate the gradient: where For a given training dataset (D), L(D, M) is the loss function for training the residual network (M), Revert the activation function change, and By variable (s b ) determines the impact of the modified residual block on the performance of the application.
5. The method according to any one of claims 2 to 4, wherein the estimated impact of the residual block on the hardware efficiency is estimated by the following steps: for each residual block that has not been reparameterized, the size of the residual feature map is calculated by counting the number of pixels, wherein the impact of the residual block on the hardware efficiency is estimated as the size of the residual map.
6. The method according to any one of claims 2 to 5, wherein the estimated impact of the residual block on the hardware efficiency is estimated by the following step: for each residual block that has not been reparameterized, determining a delay for inference for the residual block, wherein the impact of the residual block on the hardware is estimated as the delay.
7. The method according to claim 1 , wherein the estimated impact of the residual blocks on the hardware efficiency is estimated by: for each residual block that has not been reparameterized, determining a first delay (l) for inference for the residual block. orig ), and determining a second delay (l for the inference of the residual block after it is reparameterized reparametrized ), wherein the impact of the residual block on the hardware efficiency is estimated as the first delay (l orig ) and the second delay (l reparametrized ) between .
8. A computer-implemented method for classifying sensor signals using a reparameterized residual network, wherein the reparameterized residual network is reparameterized using the method according to any one of claims 1 to 7, the computer-implemented method comprising the steps of: - receiving a sensor signal (S) comprising data from a sensor (30), - determining an input signal (x) that depends on said sensor signal (S), and - feeding the input signal (x) into the classifier (60) to obtain an output signal (y) characterizing the classification of the input signal (x).
9. A computer-implemented method for providing an actuator control signal (A) for controlling an actuator (10) using a reparameterized residual network, the reparameterized residual network being reparameterized using the method according to any one of claims 1 to 7, the computer-implemented method comprising all the steps of the method according to claim 8 and further comprising the steps of: - determining said actuator control signal (A) in dependence on said output signal (y).
10. The method according to claim 9, wherein: The actuator (10) controls an at least partially autonomous robot (100) and / or a manufacturing machine (200) and / or an access control system (300).
11. A computer program configured to cause a processor (45, 145) to carry out the method according to any one of claims 1 to 10 and all its steps if the computer program is carried out by the processor.
12. A machine-readable storage medium (46, 146) having stored thereon a computer program according to claim 11.
13. A system configured to implement the method according to any one of claims 1 to 10.