Apparatus and method for training a neural network, apparatus and method for using a trained neural network to perform multiple tasks

Abstract

An apparatus for training a neural network is provided. The apparatus comprises processing circuitry configured to obtain the neural network having multiple task-specific output interfaces and a plurality of layers shared by the task-specific output interfaces and train the neural network to perform multiple tasks through applying a weight standardization to weights of at least one convolutional layer of the plurality of layers.

Description

[0001] APPARATUS AND METHOD FOR TRAINING A NEURAL NETWORK, APPARATUS AND METHOD FOR USING A TRAINED NEURAL NETWORK TO PERFORM MULTIPLE TASKS

[0002] Field

[0003] The present disclosure relates to multi-tasking of neural networks. Examples relate to an apparatus and a method for training a neural network and an apparatus and a method for using a trained neural network to perform multiple tasks.

[0004] Background

[0005] In the realm of machine learning, when dealing with multiple tasks, it can be quite memoryintensive to have a separate machine-learning model for each task. Moreover, constantly switching between these task-specific models could be time-consuming, especially for on-chip deployment that requires re-burning of an on-chip model. On the other hand, relying on a single model to handle multiple tasks can pose challenges in preserving the variability of diverse input data and achieving high performance across all tasks. Specifically, normalization layers struggle to model the statistics of input (training) data from multiple distributions. Therefore, there is a demand for improved multi-tasking approaches in machine learning.

[0006] Summary

[0007] The demand is satisfied by the subject-matter of the independent claims.

[0008] Some aspects of the present disclosure relate to an apparatus for training a neural network, the apparatus comprising processing circuitry configured to obtain the neural network having multiple task-specific output interfaces and a plurality of layers shared by the task-specific output interfaces and train the neural network to perform multiple tasks through applying a weight standardization to weights of at least one convolutional layer of the plurality of layers.

[0009] Some aspects of the present disclosure relate to a method for training a neural network, the method comprising obtaining the neural network having multiple task-specific output interfaces and a plurality of layers shared by the task-specific output interfaces, and training the neural network to perform multiple tasks, wherein training the neural network comprises applying a weight standardization to weights of at least one convolutional layer of the plurality of layers.

[0010] Some aspects of the present disclosure relate to an apparatus for using a trained neural network to perform multiple tasks, the apparatus comprising processing circuitry configured to obtain the trained neural network, wherein the trained neural network has multiple task-specific output interfaces and a plurality of layers shared by the task-specific output interfaces, wherein at least two convolutional layers of the plurality of layers are coupled to each other without a batch normalization layer in between, and perform the multiple tasks using the trained neural network.

[0011] Some aspects of the present disclosure relate to a method for using a trained neural network to perform multiple tasks, the method comprising obtaining the trained neural network having multiple task-specific output interfaces and a plurality of layers shared by the task-specific output interfaces, wherein at least two convolutional layers of the plurality of layers are coupled to each other without a batch normalization layer in between, and performing the multiple tasks using the trained neural network.

[0012] Brief description of the Figures

[0013] Some examples of apparatuses and / or methods will be described in the following by way of example only, and with reference to the accompanying figures, in which

[0014] Figs, la and lb illustrate an example of an apparatus for training a neural network and an example of a neural network, respectively;

[0015] Fig. 2 illustrates another example of a neural network;

[0016] Fig. 3 illustrates an example of a method for training a neural network;

[0017] Fig. 4 illustrates an example of a method for training and deploying a neural network; Fig. 5 illustrates an example of an apparatus for using a trained neural network to perform multiple tasks; and

[0018] Fig. 6 illustrates an example of a method for using a trained neural network to perform multiple tasks.

[0019] Detailed Description

[0020] Some examples are now described in more detail with reference to the enclosed figures. However, other possible examples are not limited to the features of these embodiments described in detail. Other examples may include modifications of the features as well as equivalents and alternatives to the features. Furthermore, the terminology used herein to describe certain examples should not be restrictive of further possible examples.

[0021] Throughout the description of the figures same or similar reference numerals refer to same or similar elements and / or features, which may be identical or implemented in a modified form while providing the same or a similar function. The thickness of lines, layers and / or areas in the figures may also be exaggerated for clarification.

[0022] When two elements A and B are combined using an “or”, this is to be understood as disclosing all possible combinations, i.e., only A, only B as well as A and B, unless expressly defined otherwise in the individual case. As an alternative wording for the same combinations, "at least one of A and B" or "A and / or B" may be used. This applies equivalently to combinations of more than two elements.

[0023] If a singular form, such as “a”, “an” and “the” is used and the use of only a single element is not defined as mandatory either explicitly or implicitly, further examples may also use several elements to implement the same function. If a function is described below as implemented using multiple elements, further examples may implement the same function using a single element or a single processing entity. It is further understood that the terms "include", "including", "comprise" and / or "comprising", when used, describe the presence of the specified features, integers, steps, operations, processes, elements, components and / or a group thereof, but do not exclude the presence or addition of one or more other features, integers, steps, operations, processes, elements, components and / or a group thereof.

[0024] Fig. la illustrates an example of an apparatus 100. The apparatus 100 comprises processing circuitry 110 and optionally comprises interface circuitry 120. In case interface circuitry 120 is present, the interface circuitry 120 is communicatively coupled (e.g., via a wired or wireless connection) to the processing circuitry 110, e.g., for data exchange between the interface circuitry 120 and the processing circuitry 110.

[0025] The interface circuitry 120 may be any device or means for communicating or exchanging data. For instance, the interface circuitry 120 may be a set of electronic components, circuits, and / or subsystems for interaction between different interfacing entities such as devices, systems, or components. It may comprise voltage level shifters, buffers, amplifiers, filters, converters, multiplexers, demultiplexers, and / or various other electronic elements.

[0026] The processing circuitry 110 may be, e.g., a single dedicated processor, a single shared processor, or a plurality of individual processors, some of which or all of which may be shared, a digital signal processor (DSP) hardware, an application specific integrated circuit (ASIC), a microcontroller or a field programmable gate array (FPGA). The processing circuitry 110 may optionally be coupled to, e.g., read only memory (ROM) for storing software, random access memory (RAM) and / or non-volatile memory.

[0027] The apparatus 100 is to be considered in the context of training of a neural network 130. For this purpose, the processing circuitry 110 is configured to obtain the neural network 130. For instance, the interface circuitry 120 may receive the neural network 130 from memory (e.g., external to the apparatus 100) storing an architecture of the neural network 130. Alternatively, the apparatus 100 may include said memory which is therefore directly accessible by the processing circuitry 110 and / or the processing circuitry 110 may be configured to (at least partially) generate the neural network 130 (e.g., generate the architecture of the neural network 130). In the latter cases, the apparatus 100 may dispense with the interface circuitry 120.

[0028] The neural network 130 is a type of machine-learning model, i.e., a data structure and / or set of rules / operations representing a statistical model that is adjustable at least during a training phase of the neural network 130. The neural network 130 has a (data) architecture based on neurons or nodes which are coupled to each other. The neurons receive input or intermediate (interim) data, process it through a (mathematical) operation, and produce output or further intermediate data. The neurons are organized into layers, with at least one input layer receiving the input or initial data, one or more hidden layers performing intermediate operations, and an output layer producing the final output.

[0029] For implementing a training effect, the neural network 130 has learnable / adjustable parameters, e.g., weights and / or biases. The weights may represent a strength of connections between neurons in the neural network 130. For example, at a neuron of the neural network 130, an intermediate result from a previous layer or neuron may be modified based on its corresponding weights and a predefined mathematical operation of that neuron, and the result may then be used to produce an output value. The biases may control an activation function of the neural network 130. Further, the neural network 130 may be adjustable in terms of, e.g., regularization parameters which introduce penalty terms based on magnitudes of the weights, number and type of layers, type of activation functions, learning rate, batch size, dropout rate or alike.

[0030] The neural network 130 may have a specific data flow direction, that is, the neural network 130 may perform operations in a forward propagation manner (feed-forward), where data flows through the neural network 130 in a predefined direction. For instance, the input data may be provided to an input layer of the neural network 130, and data of interim results of the layers may propagate through subsequent downstream layers of the neural network 130 until an output layer is reached.

[0031] An architecture of the neural network 130 is further illustrated by Fig. lb. The neural network 130 has a plurality of layers 140, e.g., layers 140-1 and 140-2, and multiple (a plurality of) task-specific output interfaces 150, e.g., output interfaces 150-1 and 150-2. The plurality of layers 140 may have any number n > 2 layers. In the example of Fig. lb, two task-specific output interfaces 150 are shown. However, in other examples, there may be any number m > 2 of task-specific output interfaces.

[0032] The plurality of layers 140 are coupled to each other, e.g., an output of the layer 140-1 is connected for data flow with an input of the layer 140-2, thereby defining the above-mentioned data flow direction. In other words, the plurality of layers 140 are stacked one layer over another. The plurality of layers 140 are further coupled to the downstream task-specific output interfaces 150 such that the plurality of layers 140 are shared by the task-specific output interfaces 150. That is, the plurality of layers 140 include at least one layer which is not taskspecific, i.e., shared or used for multiple tasks and thus shared by the multiple task-specific output interfaces 150. Optionally, the plurality of layers 140 may also include task-specific layers, e.g., the data flow through the plurality of layers 140 may split up into multiple processing pipelines which converge or intersect further downstream.

[0033] The plurality of layers 140 may be or comprise a backbone of the neural network 130, whereas the output interfaces 150 may be or comprise headers of the neural network 130, for instance. The backbone refers to a base architecture or a feature extractor of the neural network 130 used in multi-task learning. The header or task head refers to a part of the neural network 130 that can be added on top of the backbone to perform a specific task. The header includes at least one layer that take the extracted features from the backbone and produces a final output for the target task. The header may in some examples be exclusively task-specific, thus, it is not shared across different tasks.

[0034] For multi-task learning, the backbone and header are combined to create the neural network 130. Multi-task learning is a learning paradigm where a single model (the neural network 130) is trained to perform multiple tasks. In this approach, the neural network 130 is enabled to jointly learn from multiple tasks in form of the shared plurality of layers 140, exploiting commonalities and relationships between the tasks, whereas task-specific characteristics of input data are considered separately in the output interfaces 150.

[0035] In the example of Fig. lb, the most downstream layer of the plurality of layers 140 is directly coupled to both output interfaces 150-1, 150-2, creating two processing pipelines, each for a different task. That is, the data flow through the neural network 130 may divide at the output interfaces 150. The output interfaces 150 may each comprise any number of layers including an output layer which is the most downstream layer of the neural network 130 for one task.

[0036] The plurality of layers 140 comprises at least one convolutional layer. For instance, layer 140- 1 or 140-2 or any other layer of the plurality of layers 140 may be a convolutional layer. Other layers of the plurality of layers 140 may be of any type, e.g., at least one of a dense layer, a pooling layer, a recurrent layer, a Long Short-Term Memory (LSTM) layer, a normalization layer, a dropout layer or alike. In some examples, the plurality of layers 140 comprise at least two convolutional layers shared by the task-specific output interfaces 150, or all layers of the plurality of layers 140 may be convolutional.

[0037] The convolutional layer (convolution layer or conv layer) is capable of applying convolutional operations to data input to the convolutional layer. This data may be provided as a 3D tensor, for instance. The convolutional layer may have a set of learnable filters or kernels (convolutional kernels) which may have a 2D matrix data structure. These filters may slide (convolve) over the data, performing element-wise multiplication with local regions of the tensor at different positions. The convolution operation involves applying these filters to the data by sliding them over the tensor, and at each position, the filter elements (weights) are multiplied with the corresponding elements in the local region of the tensor. The results may be summed or otherwise processed to produce a single value, e.g., in the form of an output tensor. The convolutional layer may have a certain stride which refers to the step size at which the filters slide over the data.

[0038] The plurality of layers 140 may in some examples comprise or be at least one encoder. For instance, the convolutional layer (or layers) may be an encoder. An encoder refers to a data architecture that is capable of transforming input data into a different representation or embedding. The encoder thereby compresses the input data into a lower-dimensional representation. By reducing the dimensionality of the data, the encoder may enable more efficient and effective processing by providing more informative data, as well as better generalization to new or unseen data.

[0039] Depending on the tasks, the task-specific output interfaces 150 may have a certain data architecture, e.g., they may include or be a decoder. The tasks may be computer vision tasks, for instance. For object detection, it may comprise any detector architecture, such as including a single shot multibox detector layer. They may generally include at least one of a fully connected layer, a single shot multibox detector layer, a fully convolutional layer, and a top-down layer. A fully connected layer or dense layer is a type of neural network layer where every neuron in the layer is connected to every neuron in the previous layer. The purpose of such a fully connected layer may be learning global patterns and relationships in the data. A fully connected layer may be used in a final stage of the neural network 130 for tasks like classification and regression.

[0040] The single shot multibox detector (SSD) layer may be capable of detecting multiple objects in an image. It combines multiple convolutional feature maps of different scales and aspect ratios to predict bounding boxes and associated class probabilities for objects in the input image.

[0041] A fully convolutional layer exclusively uses convolutional operations without any fully connected layers. In contrast to fully connected layers, which operate on fixed-size input data, fully convolutional layers may be able to process variable-sized input data while preserving spatial information. A fully convolutional layer may be used in semantic segmentation tasks or image-to-image translation.

[0042] A top-down layer may be particularly used for (hierarchical) image or data generation or refinement, image super-resolution, image segmentation, pose estimation, or alike. In such a top-down architecture, the layer generates data or predictions by first creating a higher-level representation and then progressively refining it in a step-by-step manner. It may have a series of deconvolutional or upsampling layers followed by convolutional or downsampling layers. The upsampling layers enlarge the representation spatially, while the downsampling layers compress it.

[0043] Using the architecture of the neural network 130 as explained with reference to Fig. lb, the training of the neural network 130 is performed in the following way: The processing circuitry 110 (of Fig. la) is configured to train the neural network 130 to perform multiple (at least two) tasks. The tasks may be or comprise, e.g., at least one of image classification, object detection, image segmentation, and pose estimation. The tasks may optionally include at least two tasks within one task category: For instance, the tasks may include different object classification tasks, e.g., classification of fruits and classification of animals. The processing circuitry 110 may train the neural network 130 by executing a training framework to adjust the weights or biases of the neural network 130. For instance, the training framework may comprise a data architecture or set of rules / operations to compare an output of the neural network 130 for a certain input to a label or desired output associated to that input. The processing circuitry 110 may therefore receive (e.g., via the interface circuitry 120) or determine input (training) data. Said training data may be multi-domain image data, for instance, i.e., the processing circuitry 110 may be configured to train the neural network 130 to perform the tasks based on the multi-domain image data. The multi-domain image data refers to a dataset that contains image samples from multiple distinct domains or sources (e.g., indoor and outdoor scene images). Each domain represents a different distribution of image data with its own unique characteristics, patterns, and statistical properties, which are specific for a certain task.

[0044] The training may be performed using any learning method. For instance, one or more of the following or other learning methods may be used. Supervised learning enables the neural network 130 to be trained on a labeled dataset where both the input training data and corresponding target labels are provided to the training framework. The neural network 130 is tasked with learning a mapping between the inputs and the labels. Tasks in supervised learning may include classification and regression. Unsupervised learning dispenses with labels, the training is based on an unlabeled dataset, i.e., only the input training data is provided without any corresponding target labels. The neural model 130 is tasked with finding patterns, structures, or representations in the data without explicit guidance. Clustering, dimensionality reduction, and generative modeling may be unsupervised learning tasks. Semi-supervised learning combines elements of both supervised and unsupervised learning. It involves training the neural model 130 on a dataset with a mix of labeled and unlabeled data. Reinforcement learning is a learning paradigm where an agent learns to make decisions and take actions in the training framework to increase a cumulative reward. The agent receives feedback in the form of rewards or penalties based on its actions, and it learns to improve its decision-making through trial and error.

[0045] Training the neural network 130 may include adjusting parameters, such as weights or biases, of the neural network 130. The parameters may be adjusted based on backpropagation, e.g., for iteratively updating the parameters, or alike. For example, the processing circuitry 110 may adjust the neural network 130 to decrease a difference between a predicted output and a desired output, based on a specified loss or error function.

[0046] For adjusting the parameters of the neural network 130, any training algorithm may be used, e.g., gradient descent, momentum, adagrad, or alike. For high computational efficiency, especially for large training datasets, stochastic gradient descent may be used. In such cases, the processing circuitry 110 is further configured to train the neural network 130 using said stochastic gradient descent (SGD). SGD is an optimization algorithm which - unlike traditional gradient descent where parameters are updated based on the average gradient of the loss function with respect to the entire training dataset - computes a gradient and updates the parameters for a subset of the training dataset, e.g., for each individual training sample. This makes the update process more frequent, causing the optimization to zig-zag around the optimal solution. The processing circuitry 110 may, for example, initialize the parameters of the neural network 130 with random or initial values, shuffle the training data, e.g., before each epoch (a complete pass through the entire training dataset), iterate through training samples such that for each sample in the training dataset, the gradient of the loss function with respect to the parameters is computed, and update the parameters using the computed gradient and, e.g., a learning rate (a hyperparameter that determines the step size for the updates). This may be repeated for a fixed number of epochs or until a convergence criteria is met.

[0047] To mitigate the problem of exploding gradients during training, the apparatus 100 may make use of adaptive gradient clipping. For instance, the processing circuitry 110 may be further configured to train the neural network 130 using said adaptive gradient clipping. Gradient clipping involves limiting a magnitude of the gradients during the training phase. For example, a certain threshold may be set to prevent the gradients from exceeding it, which may help stabilize training and prevents gradient values from becoming extremely large. Adaptive gradient clipping dynamically adjusts the clipping threshold based on the magnitude of the gradients. For instance, the processing circuitry 110 may perform adaptive gradient clipping by computing the gradients of the parameters with respect to a loss function, determining a norm (magnitude) of the gradients, e.g., using any measures such as an L2 norm (Euclidean norm) or a max norm and determining the clipping threshold based on the gradient norm. If the gradient norm exceeds a predefined threshold value, the processing circuitry 110 may rescale or clip the gradients to bring them within the specified range. The model parameters may then be updated using the clipped gradients instead of the original gradients.

[0048] Further, the processing circuitry 110 may jointly train the backbone and the header, or separately train the backbone and jointly train the pre-trained backbone and the header. In the latter case, the backbone remains fixed (frozen) during training, and only the header is fine-tuned on the target task using a smaller dataset. By leveraging the pre-trained backbone, the neural network 130 may benefit from general features learned from the source task. For example, in image recognition, transfer learning may be applied by using an already pre-trained backbone e.g., trained on ImageNet as a feature extractor and then add a task-specific header for a particular image classification task, such as recognizing different animals, objects, or scenes.

[0049] The processing circuitry 110 is configured to train the neural network 130 through applying a weight standardization to weights of at least one convolutional layer of the plurality of layers 140. The weight standardization may involve modifying the weights of the convolutional layer, e.g., such that they have a certain mean and / or variance such as zero mean and unit variance. For further improvements, the processing circuitry 110 may apply a weight standardization to respective weights of all convolutional layers of the plurality of layers 140. Unlike in conventional multi-task training, the apparatus 100 may use weight standardization to increase the performance of the neural network 130 and decrease the memory needed for multi-task learning.

[0050] Conventional multi-task training is based on specific normalization layers, e.g., batch normalization layers, downstream to the convolutional layers. However, the normalization layers may poorly deal with multi-domain data, specifically when the domains have significantly different statistical distributions. The primary reason for this is that normalization layers normalize the output of a convolutional layer, and thereby may produce domain shift (i.e., conflicting normalization when the activations or outputs for different domains get mixed and jointly normalized), domain bias (i.e., poor generalization when the normalization aims at making the optimization landscape more well-behaved leading to a bias towards certain domains) or unwanted cross-domain normalization (i.e., the normalization parameters of a mini-batch is not representative of the data distribution of any single domain). By contrast, the apparatus 100 may reduce the number of (batch) normalization layers or entirely dispense with (batch) normalization layers. Instead, weight standardization within the convolutional layers may stabilize and speed up the convergence of the training.

[0051] For adaptation to diverse multi-domain data, the processing circuitry 110 may be configured to apply a scaled weight standardization to the weights. Scaled weight standardization introduces an additional constant scaling parameter y to the standardized weights. This convolution layer can be variance-preserving for a correctly chosen y for activation functions, i.e., the scaling parameter may be set depending on the activation function of the layer. Mathematically, a scaled standardized weight W of a convolutional layer may be represented as in Equation 1 :

[0052] Equation 1, where p and c are the mean and variance (standard deviation) of the (original) weights W of one row i, respectively, N is the fan-in of the convolutional layer, and i,j are row and column index numbers of a weight matrix. The mean and the variance are determined by the following Equations 2 and 3 :

[0053] Equation 2 ft2= ( )E / ^ - ft)2

[0054] Equation 3

[0055] Contrary to conventional layers used for multi-task training, the apparatus 100 may use scaled weight standardization. Scaled weight standardization may eliminate mean shift since the mathematical expectation of the output of the convolutional layer is 0. It may further preserve variances of the input data as the variance of a scaled standardized weight strongly depends on a non-linearity function (e.g., a rectifier linear unit, ReLU) applied to the result of the convolution performed by the convolutional layer. The weight standardization may enable the neural network 130 to omit additional normalization layers, such as batch normalization layers. It may enable a modification of weights rather than a normalization of activations (output of activation functions). This may prevent the variance of multi-domain input data to be instable due to the normalization. The implementation of omitted normalization layers may be as follows: For example, at least two convolutional layers of the plurality of layers 140 may be coupled to each other without a batch normalization layer or a normalization layer in between. For example, at least two convolutional layers of the plurality of layers 140 in a network architecture, e.g., ResNet, may be coupled to each other without a batch normalization layer or a normalization layer in between. For example, at least two convolutional layers of the plurality of layers 140 in any network architecture except VGG (visual geometry group) may be coupled to each other without a batch normalization layer or a normalization layer in between. At least the upstream one of the two convolutional layers may be weight standardized. Depending on the architecture of the neural network 130, the omission of normalization layers may lead to the at least two convolutional layers (e.g., layer 140-1, 140-2) of the plurality of layers 140 being coupled directly coupled to each other. That is, no layer is between the two convolutional layers, and an output of one convolutional layer is connected directly to an input of the next convolutional layer. Alternatively, other layers than normalization layers (e.g., an activation layer or a recurrent layer) may be provided in between.

[0056] To further improve the performance of the apparatus 100, each of all convolutional layers of the plurality of layers 140 may be coupled to another one of the convolutional layers without a batch normalization layer in between, e.g., directly coupled to each other. In the case of a fully convolutional backbone, all of the plurality of layers 140 may be directly coupled to another one of the plurality of layers 140 or coupled without a (batch) normalization layer in between. Alternatively, only a subset of convolutional layers of the plurality of layers 140 may omit a subsequent normalization layer - at least one convolutional layer may then be coupled to a downstream normalization layer. Depending on the degree of implementation of normalization-free layers, the plurality of layers 140 may comprise fewer batch normalization layers than convolutional layers, i.e., by at least 1 (2 or 3 etc.) less batch normalization layers than convolutional layers. In a full implementation, the plurality of layers 140 do not comprise any (batch) normalization layer. The apparatus 100 may provide normalization-free multi-task learning for multi-domain data. It may enable the removal of batch normalization layers in the backbone and a replacement of convolutional layers with scaled weight standardization convolutional layers.

[0057] Fig- 2 illustrates another example of an architecture of a neural network 230. The neural network 230 may be obtained and trained by an apparatus as described herein, such as apparatus 100.

[0058] The neural network 230 has a plurality of layers 240 shared by task-specific output interfaces 250. The plurality of layers 240 include at least two convolutional layers 240-1, 240-2 which are weight standardized (WSConv). Batch normalization layers 260-1, 260-2 subsequent to the convolutional layers 240-1, 240-2 are removed from the architecture in the example of Fig. 2. Hence, the convolutional layers 240-1, 240-2 are directly coupled to each other.

[0059] The task-specific output interfaces 250 include four output interfaces 250-1 to 250-4. Specifically, in the example of Fig. 2, the task-specific output interfaces 250 include a first output interface 250-1 which has two top down layers for pose estimation, a second output interface 250-2 which is a fully convolutional network (FCN) with two layers for image segmentation, a third output interface 250-3 which has a single shot multibox detector (SSD) of one shared layer and two parallel layers for object detection and a fourth output interface 250-4 which has a fully connected (FC) layer for classification.

[0060] Fig- 3 illustrates a flowchart of an example of a method 300 for training a neural network. The method 300 may be performed by an apparatus as described herein, such as apparatus 100.

[0061] The method 300 comprises obtaining 310 the neural network having multiple task-specific output interfaces and a plurality of layers shared by the task-specific output interfaces. The method 300 further comprises training 320 the neural network to perform multiple tasks, wherein training 320 the neural network comprises applying a weight standardization to weights of at least one convolutional layer of the plurality of layers. More details and aspects of the method 300 are explained in connection with the proposed technique or one or more examples described above, e.g., with reference to Figs, la, lb and 2. The method 300 may comprise one or more additional optional features corresponding to one or more aspects of the proposed technique, or one or more examples described above.

[0062] Fig- 4 illustrates a flowchart of an example of a method 400 fortraining and deploying a neural network, such as the neural network 130 or 230. The method 400 may be computer-implemented and performed at least partially by an apparatus as described herein, such as apparatus 100. The deployment of the neural network may comprise a conversion and a compression of the trained neural network.

[0063] The method 400 comprises training 410 the neural network to perform multiple tasks. Training 410 the neural network comprises applying 420 a weight standardization to weights of at least one convolutional layer of a plurality of layers of the neural network. Training 410 the neural network further comprises using 430 PyTorch and using 440 Keras. PyTorch and Keras are deep learning frameworks used for building and training neural networks. PyTorch is an open- source deep learning framework which may allow for easier debugging and more natural integration with Python programming. Keras is an open-source deep learning library designed to provide a user-friendly and high-level API (application programming interface) for building neural networks.

[0064] The method 400 further comprises deploying the trained neural network by using 450 quantization and using 460 TensorFlow Lite. Quantization and TensorFlow Lite are two concepts related to deep learning and model deployment, particularly on resource-constrained devices like mobile phones, embedded systems, and loT (Internet of Things) devices.

[0065] Quantization may reduce the memory footprint and computational requirements of the neural network while maintaining acceptable performance. For instance, model parameters (weights and biases) of the neural network and activations may be initially stored as 32-bit floatingpoint numbers. However, 32-bit floating-point representations may be memory-intensive and computationally expensive, especially on devices with limited resources and processing capabilities. Quantization involves converting these 32-bit floating-point numbers to lower-precision fixed-point or integer representations, for example, 8-bit integer. Quantization may be performed during model training (training quantization) or after training during model conversion (post-training quantization).

[0066] TensorFlow Lite is a lightweight, cross-platform framework for running the neural network on mobile and embedded devices. It is a subset of the full TensorFlow framework, optimized for efficient inference on resource-constrained platforms. It offers a variety of deployment options, such as running the neural network locally on a device, using the neural network in Android or iOS applications, or leveraging the neural network for edge computing scenarios.

[0067] The neural network may be deployed on 8- or 16-Megabyte SRAM (static random-access memory), for instance. The neural network may achieve 1,2 or 20 TOPS (Tera operations per second) with an efficiency of 5 or 11 TOPS per Watt, respectively. It may further have a precision of 8 / 4 / 2 bits for weights or 8 / 4 / 2 bits of mixed precision.

[0068] Fig- 5 illustrates an example of an apparatus 500 for using a trained neural network 530 to perform multiple tasks. The example of Fig. 5 may correspond to an inference phase of the neural network explained above, such as neural network 130 or 230. The trained neural network 530 may be a result of training said neural network. The architecture of the trained neural network 530 may therefore correspond to the architecture of the neural network 130 or 230, for example. The apparatus 500 may be external to an apparatus for training a neural network, such as apparatus 100. Alternatively, the apparatus 500 may be integrated into a system together with the apparatus for training the neural network.

[0069] The apparatus 500 comprises processing circuitry 510 and optionally interface circuitry 520 for data exchange between the interface circuitry 520, the processing circuitry 510 and external devices, such as the apparatus 100. The processing circuitry 510 is configured to obtain (e.g., receive or generate) the trained neural network 530. Optionally, the processing circuitry 510 may be configured to deploy the trained neural network 530, e.g., on the apparatus 100 (e.g., an embedded system). The trained neural network 530 has multiple task-specific output interfaces and a plurality of layers shared by the task-specific output interfaces, as explained with reference to Fig. lb and Fig. 2. The task-specific output interfaces may comprise, e.g., at least one of a fully connected layer, a single shot multibox detector layer, a fully convolutional layer and a top-down layer. At least two convolutional layers of the plurality of layers are coupled to each other without a batch normalization layer and / or a normalization layer in between. Additionally or alternatively, the at least two convolutional layers of the plurality of layers are directly coupled to each other. For example, at least two convolutional layers of the plurality of layers in a network architecture, e.g., ResNet, may be coupled to each other without a batch normalization layer or a normalization layer in between, or may be directly coupled to each other. For example, at least two convolutional layers of the plurality of layers in any network architecture except VGG may be coupled to each other without a batch normalization layer or a normalization layer in between.

[0070] The processing circuitry 510 is further configured to perform the multiple tasks using the trained neural network 530. For example, the processing circuitry 510 may execute the trained neural network 530. Performing the multiple tasks may include inputting input data (e.g., multi-domain image data) to the trained neural network 530 which outputs, at one or more of its task-specific output layers, an output based on the input data. The output may be used to perform one or more of the multiple tasks or may correspond to the performance of the tasks. The tasks may comprise at least one of image classification, object detection, image segmentation and pose estimation.

[0071] The apparatus 500 may simplify a deployment of multi-task neural networks: only one single model may be necessary to support multiple tasks, e.g., four tasks. The model may be integrated on one chip for inference, reducing the deployment costs. During inference, the apparatus 500 may provide quick task switches without reloading extra models and increase the performance of the tasks. They may provide inherent support for next-generation chips. Since the inference can be realized as fully integrated application, privacy issues can be prevented (on-device Al model prediction).

[0072] As a further enhancement of the apparatus 500, each of all convolutional layers of the plurality of layers may be coupled to another one of the convolutional layers without a (batch) normalization layer in between. For example, the plurality of layers may comprise fewer batch normalization layers than convolutional layers, i.e., at least one batch normalization layer may be left out. For a full implementation, the plurality of layers do not comprise any batch normalization layers.

[0073] Depending on the specific implementation, the two convolutional layers may be weight standardized or all convolutional layers of the plurality of layers may be weight standardized. The weight standardization may be a scaled weight standardization, for instance. This may enable the additional use of a variance preserving ReLU.

[0074] More details and aspects of the apparatus 500 are explained in connection with the proposed technique or one or more examples described above, e.g., with reference to Figs, la, lb, 2, 3 and 4. The apparatus 500 may comprise one or more additional optional features corresponding to one or more aspects of the proposed technique, or one or more examples described above.

[0075] The apparatus 500 may enable a removal of normalization layers from multi-task learning for multi-domain data. The neural network 530 may be deployable to hardware with small memory and computation power. The apparatus 500 may further increase the performance of the tasks. For example, in a specific implementation of the apparatus 500, the accuracy of object classification may be increased from 60.16% to 92.27% the intersection over union of image segmentation may be increased from 55.28% to 72.69%, the mean average precision of object detection from 51.70% to 69.90% and the accuracy of pose estimation may be increased from 37.69% to 58.01%.

[0076] Moreover, the neural network model 530 trained based on techniques described herein, may predict results with small batch sizes (e.g., as small as one), whereas conventional techniques require larger batch sizes. This may be particularly beneficial for devices with resource constraints.

[0077] Fig- 6 illustrates a flowchart of an example of a method 600 for using a trained neural network to perform multiple tasks. The method 600 may be executed by an apparatus as described herein, such as apparatus 500. The method 600 comprises obtaining 610 the trained neural network having multiple taskspecific output interfaces and a plurality of layers shared by the task-specific output interfaces. The at least two convolutional layers of the plurality of layers are coupled to each other without a batch normalization layer in between. The method 600 further comprises performing 620 the multiple tasks using the trained neural network.

[0078] More details and aspects of the method 600 are explained in connection with the proposed technique or one or more examples described above, e.g., with reference to Fig. 5. The method 600 may comprise one or more additional optional features corresponding to one or more aspects of the proposed technique, or one or more examples described above.

[0079] In the following, some examples of the proposed concept are presented:

[0080] An example (e.g., example 1) relates to an apparatus for training a neural network, the apparatus comprising processing circuitry configured to obtain the neural network having multiple task-specific output interfaces and a plurality of layers shared by the task-specific output interfaces and train the neural network to perform multiple tasks through applying a weight standardization to weights of at least one convolutional layer of the plurality of layers.

[0081] Another example (e.g., example 2) relates to a previous example (e.g., example 1) or to any other example, further comprising that the processing circuitry is further configured to apply a weight standardization to respective weights of all convolutional layers of the plurality of layers.

[0082] Another example (e.g., example 3) relates to a previous example (e.g., one of the examples 1 or 2) or to any other example, further comprising that the processing circuitry is configured to apply a scaled weight standardization to the weights.

[0083] Another example (e.g., example 4) relates to a previous example (e.g., one of the examples 1 to 3) or to any other example, further comprising that at least two convolutional layers of the plurality of layers are coupled to each other without a batch normalization layer in between. Another example (e.g., example 5) relates to a previous example (e.g., one of the examples 1 to 4) or to any other example, further comprising that at least two convolutional layers of the plurality of layers are coupled to each other without a normalization layer in between.

[0084] Another example (e.g., example 6) relates to a previous example (e.g., one of the examples 1 to 5) or to any other example, further comprising that at least two convolutional layers of the plurality of layers are directly coupled to each other.

[0085] Another example (e.g., example 7) relates to a previous example (e.g., one of the examples 1 to 6) or to any other example, further comprising that each of all convolutional layers of the plurality of layers is coupled to another one of the convolutional layers without a batch normalization layer in between.

[0086] Another example (e.g., example 8) relates to a previous example (e.g., one of the examples 1 to 7) or to any other example, further comprising that the plurality of layers comprise fewer batch normalization layers than convolutional layers.

[0087] Another example (e.g., example 9) relates to a previous example (e.g., one of the examples 1 to 8) or to any other example, further comprising that the plurality of layers do not comprise any batch normalization layer.

[0088] Another example (e.g., example 10) relates to a previous example (e.g., one of the examples 1 to 9) or to any other example, further comprising that the at least one convolutional layer is based on a variance preserving rectifier linear unit, ReLU.

[0089] Another example (e.g., example 11) relates to a previous example (e.g., one of the examples 1 to 10) or to any other example, further comprising that the processing circuitry is further configured to train the neural network using stochastic gradient descent.

[0090] Another example (e.g., example 12) relates to a previous example (e.g., one of the examples 1 to 11) or to any other example, further comprising that the processing circuitry is further configured to train the neural network using adaptive gradient clipping. Another example (e.g., example 13) relates to a previous example (e.g., one of the examples 1 to 12) or to any other example, further comprising that the processing circuitry is configured to train the neural network to perform at least two tasks, wherein the tasks comprise at least one of image classification, object detection, image segmentation and pose estimation.

[0091] Another example (e.g., example 14) relates to a previous example (e.g., one of the examples 1 to 13) or to any other example, further comprising that the processing circuitry is configured to train the neural network based on multi-domain image data.

[0092] Another example (e.g., example 15) relates to a previous example (e.g., one of the examples 1 to 14) or to any other example, further comprising that the plurality of layers comprise at least one encoder.

[0093] Another example (e.g., example 16) relates to a previous example (e.g., one of the examples 1 to 15) or to any other example, further comprising that the plurality of layers comprise at least two convolutional layers shared by the task-specific output interfaces.

[0094] Another example (e.g., example 17) relates to a previous example (e.g., one of the examples 1 to 16) or to any other example, further comprising that the plurality of layers comprise a backbone of the neural network.

[0095] Another example (e.g., example 18) relates to a previous example (e.g., one of the examples 1 to 17) or to any other example, further comprising that the task-specific output interfaces comprise at least one of a fully connected layer, a single shot multibox detector layer, a fully convolutional layer, and a top-down layer.

[0096] An example (e.g., example 19) relates to a method for training a neural network, the method comprising obtaining the neural network having multiple task-specific output interfaces and a plurality of layers shared by the task-specific output interfaces, and training the neural network to perform multiple tasks, wherein training the neural network comprises applying a weight standardization to weights of at least one convolutional layer of the plurality of layers. An example (e.g., example 20) relates to an apparatus for using a trained neural network to perform multiple tasks, the apparatus comprising processing circuitry configured to obtain the trained neural network, wherein the trained neural network has multiple task-specific output interfaces and a plurality of layers shared by the task-specific output interfaces, wherein at least two convolutional layers of the plurality of layers are coupled to each other without a batch normalization layer in between, and perform the multiple tasks using the trained neural network.

[0097] An example (e.g., example 21) relates to an apparatus for using a trained neural network to perform multiple tasks, the apparatus comprising processing circuitry configured to obtain the trained neural network, wherein the trained neural network has multiple task-specific output interfaces and a plurality of layers shared by the task-specific output interfaces, wherein at least two convolutional layers of the plurality of layers are directly coupled to each other, and perform the multiple tasks using the trained neural network.

[0098] Another example (e.g., example 22) relates to a previous example (e.g., example 20 or 21) or to any other example, further comprising that the at least two convolutional layers are coupled to each other without a normalization layer in between.

[0099] Another example (e.g., example 23) relates to a previous example (e.g., one of the examples 20 to 22) or to any other example, further comprising that the two convolutional layers are directly coupled to each other.

[0100] Another example (e.g., example 24) relates to a previous example (e.g., one of the examples 20 to 23) or to any other example, further comprising that each of all convolutional layers of the plurality of layers are coupled to another one of the convolutional layers without a batch normalization layer in between.

[0101] Another example (e.g., example 25) relates to a previous example (e.g., one of the examples 20 to 24) or to any other example, further comprising that the plurality of layers comprise fewer batch normalization layers than convolutional layers. Another example (e.g., example 26) relates to a previous example (e.g., one of the examples 20 to 25) or to any other example, further comprising that the plurality of layers do not comprise any batch normalization layers.

[0102] Another example (e.g., example 27) relates to a previous example (e.g., one of the examples 20 to 24) or to any other example, further comprising that the two convolutional layers are weight standardized.

[0103] Another example (e.g., example 28) relates to a previous example (e.g., example 27) or to any other example, further comprising that all convolutional layers of the plurality of layers are weight standardized.

[0104] Another example (e.g., example 29) relates to a previous example (e.g., one of the examples 20 to 28) or to any other example, further comprising that the two convolutional layers are scaled weight standardized.

[0105] Another example (e.g., example 30) relates to a previous example (e.g., one of the examples 20 to 29) or to any other example, further comprising that the two convolutional layers are based on a variance preserving rectifier linear unit, ReLU.

[0106] Another example (e.g., example 31) relates to a previous example (e.g., one of the examples 20 to 30) or to any other example, further comprising that the processing circuitry is configured to perform at least two tasks, wherein the tasks comprise at least one of image classification, object detection, image segmentation and pose estimation using the trained neural network.

[0107] Another example (e.g., example 32) relates to a previous example (e.g., one of the examples 20 to 31) or to any other example, further comprising that the processing circuitry is configured to perform the tasks based on multi-domain image data.

[0108] Another example (e.g., example 33) relates to a previous example (e.g., one of the examples 20 to 32) or to any other example, further comprising that the plurality of layers comprise at least one encoder. Another example (e.g., example 34) relates to a previous example (e.g., one of the examples 20 to 33) or to any other example, further comprising that the task-specific output interfaces comprise at least one of a fully connected layer, a single shot multibox detector layer, a fully convolutional layer and a top-down layer.

[0109] An example (e.g., example 35) relates to a method for using a trained neural network to perform multiple tasks, the method comprising obtaining the trained neural network having multiple task-specific output interfaces and a plurality of layers shared by the task-specific output interfaces, wherein at least two convolutional layers of the plurality of layers are coupled to each other without a batch normalization layer in between, and performing the multiple tasks using the trained neural network.

[0110] Another example (e.g., example 36) relates to a non-transitory machine-readable medium having stored thereon a program having a program code for performing the method of any one of examples 19 or 35, when the program is executed on a processor or a programmable hardware.

[0111] Another example (e.g., example 37) relates to a program having a program code for performing the method of any one of examples 19 or 35, when the program is executed on a processor or a programmable hardware.

[0112] The aspects and features described in relation to a particular one of the previous examples may also be combined with one or more of the further examples to replace an identical or similar feature of that further example or to additionally introduce the features into the further example.

[0113] Examples may further be or relate to a (computer) program including a program code to execute one or more of the above methods when the program is executed on a computer, processor or other programmable hardware component. Thus, steps, operations or processes of different ones of the methods described above may also be executed by programmed computers, processors or other programmable hardware components. Examples may also cover program storage devices, such as digital data storage media, which are machine-, processor- or computer- readable and encode and / or contain machine-executable, processor-executable or computerexecutable programs and instructions. Program storage devices may include or be digital storage devices, magnetic storage media such as magnetic disks and magnetic tapes, hard disk drives, or optically readable digital data storage media, for example. Other examples may also include computers, processors, control units, (field) programmable logic arrays ((F)PLAs), (field) programmable gate arrays ((F)PGAs), graphics processor units (GPU), application-specific integrated circuits (ASICs), integrated circuits (ICs) or system-on-a-chip (SoCs) systems programmed to execute the steps of the methods described above.

[0114] It is further understood that the disclosure of several steps, processes, operations or functions disclosed in the description or claims shall not be construed to imply that these operations are necessarily dependent on the order described, unless explicitly stated in the individual case or necessary for technical reasons. Therefore, the previous description does not limit the execution of several steps or functions to a certain order. Furthermore, in further examples, a single step, function, process or operation may include and / or be broken up into several sub-steps, - functions, -processes or -operations.

[0115] If some aspects have been described in relation to a device or system, these aspects should also be understood as a description of the corresponding method. For example, a block, device or functional aspect of the device or system may correspond to a feature, such as a method step, of the corresponding method. Accordingly, aspects described in relation to a method shall also be understood as a description of a corresponding block, a corresponding element, a property or a functional feature of a corresponding device or a corresponding system.

[0116] The following claims are hereby incorporated in the detailed description, wherein each claim may stand on its own as a separate example. It should also be noted that although in the claims a dependent claim refers to a particular combination with one or more other claims, other examples may also include a combination of the dependent claim with the subject matter of any other dependent or independent claim. Such combinations are hereby explicitly proposed, unless it is stated in the individual case that a particular combination is not intended. Furthermore, features of a claim should also be included for any other independent claim, even if that claim is not directly defined as dependent on that other independent claim.

Claims

ClaimsWhat is claimed is:

1. An apparatus for training a neural network, the apparatus comprising processing circuitry configured to: obtain the neural network having multiple task-specific output interfaces and a plurality of layers shared by the task-specific output interfaces; and train the neural network to perform multiple tasks through applying a weight standardization to weights of at least one convolutional layer of the plurality of layers.

2. The apparatus of claim 1, wherein the processing circuitry is further configured to apply a weight standardization to respective weights of all convolutional layers of the plurality of layers.

3. The apparatus of claim 1, wherein the processing circuitry is configured to apply a scaled weight standardization to the weights.

4. The apparatus of claim 1, wherein at least two convolutional layers of the plurality of layers are coupled to each other without a batch normalization layer in between.

5. The apparatus of claim 1, wherein at least two convolutional layers of the plurality of layers are coupled to each other without a normalization layer in between.

6. The apparatus of claim 1, wherein at least two convolutional layers of the plurality of layers are directly coupled to each other.

7. The apparatus of claim 1, wherein each of all convolutional layers of the plurality of layers is coupled to another one of the convolutional layers without a batch normalization layer in between.

8. The apparatus of claim 1, wherein the plurality of layers comprise fewer batch normalization layers than convolutional layers.

9. The apparatus of claim 1, wherein the plurality of layers do not comprise any batch normalization layer.

10. The apparatus of claim 1, wherein the at least one convolutional layer is based on a variance preserving rectifier linear unit, ReLU.

11. The apparatus of claim 1 , wherein the processing circuitry is further configured to train the neural network using stochastic gradient descent.

12. The apparatus of claim 1, wherein the processing circuitry is further configured to train the neural network using adaptive gradient clipping.

13. The apparatus of claim 1, wherein the processing circuitry is configured to train the neural network to perform at least two tasks, wherein the tasks comprise at least one of image classification, object detection, image segmentation and pose estimation.

14. The apparatus of claim 1, wherein the processing circuitry is configured to train the neural network based on multi-domain image data.

15. The apparatus of claim 1 , wherein the plurality of layers comprise at least one encoder.

16. The apparatus of claim 1, wherein the plurality of layers comprise at least two convolutional layers shared by the task-specific output interfaces.

17. The apparatus of claim 1, wherein the plurality of layers comprise a backbone of the neural network.

18. The apparatus of claim 1, wherein the task-specific output interfaces comprise at least one of a fully connected layer, a single shot multibox detector layer, a fully convolutional layer, and a top-down layer.

19. A method for training a neural network, the method comprising: obtaining the neural network having multiple task-specific output interfaces and a plurality of layers shared by the task-specific output interfaces; andtraining the neural network to perform multiple tasks, wherein training the neural network comprises applying a weight standardization to weights of at least one convolutional layer of the plurality of layers.

20. An apparatus for using a trained neural network to perform multiple tasks, the appa- ratus comprising processing circuitry configured to: obtain the trained neural network, wherein the trained neural network has multiple task-specific output interfaces and a plurality of layers shared by the task-specific output interfaces, wherein at least two convolutional layers of the plurality of layers are coupled to each other without a batch normalization layer in between; and perform the multiple tasks using the trained neural network.

Images