Apparatus and method for training neural network

Abstract

An apparatus and method for training a neural network are provided. The apparatus includes processing circuitry configured to obtain a neural network having a plurality of task-specific output interfaces and a plurality of layers shared by the task-specific output interfaces, and train the neural network to perform a plurality of tasks by applying weight standardization to weights of at least one convolutional layer of the plurality of layers.

Description

Technical Field

[0001] The present disclosure relates to multitasking of neural networks. Examples relate to apparatus and methods for training neural networks and apparatus and methods for performing multiple tasks using trained neural networks. Background Art

[0002] In the field of machine learning, when dealing with multiple tasks, building a separate machine learning model for each task can take up a lot of memory. In addition, constantly switching between these task-specific models can be time-consuming, especially for on-chip deployments that require re-burning the on-chip model. On the other hand, relying on a single model to handle multiple tasks can pose challenges in maintaining the variability of diverse input data and achieving high performance across all tasks. Specifically, normalization layers have difficulty modeling the statistics of input (training) data from multiple distributions. Therefore, there is a need for improved multi-tasking methods in machine learning. Summary of the invention

[0003] This need is solved by the subject matter of the independent claims.

[0004] Some aspects of the present disclosure relate to an apparatus for training a neural network, the apparatus comprising a processing circuit configured to obtain a neural network having a plurality of task-specific output interfaces and a plurality of layers shared by the task-specific output interfaces, and to train the neural network to perform a plurality of tasks by applying weight normalization to weights of at least one convolutional layer of the plurality of layers.

[0005] Some aspects of the present disclosure relate to a method for training a neural network, the method comprising obtaining a neural network having a plurality of task-specific output interfaces and a plurality of layers shared by the task-specific output interfaces, and training the neural network to perform a plurality of tasks, wherein training the neural network comprises applying weight normalization to weights of at least one convolutional layer among the plurality of layers.

[0006] Some aspects of the present disclosure relate to an apparatus for performing multiple tasks using a trained neural network, the apparatus comprising a processing circuit configured to obtain a trained neural network, wherein the trained neural network has multiple task-specific output interfaces and multiple layers shared by the task-specific output interfaces, wherein at least two convolutional layers of the multiple layers are coupled to each other without a batch normalization layer therebetween, and perform the multiple tasks using the trained neural network.

[0007] Some aspects of the present disclosure relate to a method of performing multiple tasks using a trained neural network, the method comprising obtaining a trained neural network having multiple task-specific output interfaces and multiple layers shared by the task-specific output interfaces, wherein at least two convolutional layers of the multiple layers are coupled to each other without a batch normalization layer therebetween, and performing the multiple tasks using the trained neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Some examples of apparatus and / or methods will be described below, by way of example only, and with reference to the accompanying drawings, in which

[0009] Figure 1a and Figure 1b An example of an apparatus for training a neural network and an example of a neural network are respectively shown;

[0010] Figure 2 Another example of a neural network is shown;

[0011] Figure 3 An example of a method for training a neural network is shown;

[0012] Figure 4 Examples of methods for training a neural network and deploying a neural network are shown;

[0013] Figure 5 An example of an apparatus for performing multiple tasks using a trained neural network is shown; and

[0014] Figure 6 Examples of methods for performing multiple tasks using trained neural networks are shown. DETAILED DESCRIPTION

[0015] Some examples are now described in more detail with reference to the accompanying drawings. However, other possible examples are not limited to the features of these embodiments described in detail. Other examples may include modifications of features and equivalents and substitutes of features. In addition, the terms used herein to describe certain examples should not limit further possible examples.

[0016] Throughout the description of the accompanying drawings, the same or similar reference numerals represent the same or similar elements and / or features, which may be the same or implemented in a modified form while providing the same or similar functions. For clarity, the thickness of the lines, layers and / or regions in the drawings may also be exaggerated.

[0017] When two elements A and B are combined using "or", unless explicitly defined otherwise in individual cases, it should be understood that all possible combinations are disclosed, i.e. only A, only B, and A and B. As alternative expressions for the same combination, "at least one of A and B" or "A and / or B" can be used. The same applies to combinations of more than two elements.

[0018] If singular forms such as "a", "an" and "the" are used, and there is no explicit or implicit definition that the use of only a single element is mandatory, further examples may also use several elements to achieve the same function. If a function is described below as being implemented using multiple elements, further examples may use a single element or a single processing entity to achieve the same function. It should also be understood that the terms "comprises", "comprising", "includes" and / or "comprising" when used describe the presence of specific features, integers, steps, operations, processes, elements, components and / or combinations thereof, but do not exclude the presence or addition of one or more other features, integers, steps, operations, processes, elements, components and / or combinations thereof.

[0019] Figure 1a An example of an apparatus 100 is shown. The apparatus 100 includes a processing circuit 110 and optionally an interface circuit 120. Where the interface circuit 120 is present, the interface circuit 120 is communicatively coupled (e.g., via a wired or wireless connection) to the processing circuit 110, e.g., for data exchange between the interface circuit 120 and the processing circuit 110.

[0020] The interface circuit 120 may be any device or apparatus for communicating or exchanging data. For example, the interface circuit 120 may be a set of electronic components, circuits, and / or subsystems for interaction between different interface entities (e.g., devices, systems, or components). The interface circuit may include voltage level converters, buffers, amplifiers, filters, converters, multiplexers, demultiplexers, and / or various other electronic components.

[0021] For example, processing circuit 110 may be a single dedicated processor, a single shared processor, or multiple individual processors, some or all of which may be shared, digital signal processor (DSP) hardware, application specific integrated circuit (ASIC), microprocessor, or field programmable gate array (FPGA). For example, processing circuit 110 may optionally be coupled to read-only memory (ROM), random access memory (RAM), and / or non-volatile memory for storing software.

[0022] The apparatus 100 will be considered in the context of training a neural network 130. To this end, the processing circuit 110 is configured to obtain the neural network 130. For example, the interface circuit 120 may receive the neural network 130 from a memory (e.g., external to the apparatus 100) storing the architecture of the neural network 130. Alternatively, the apparatus 100 may include the memory, which may thus be directly accessible by the processing circuit 110, and / or the processing circuit 110 may be configured to (at least partially) generate the neural network 130 (e.g., generate the architecture of the neural network system 130). In the latter case, the apparatus 100 may omit the interface circuit 120.

[0023] The neural network 130 is a machine learning model, i.e. a data structure and / or a set of rules / operations representing a statistical model that is adjustable at least during the training phase of the neural network 130. The neural network 130 has a (data) architecture based on neurons or nodes coupled to each other. Neurons receive input or intermediate (temporary) data, process them by (mathematical) operations, and generate output or further intermediate data. Neurons are organized into layers, at least one input layer receiving input or initial data, one or more hidden layers performing intermediate operations, and an output layer generating a final output.

[0024] In order to achieve the training effect, the neural network 130 has learnable / adjustable parameters, such as weights and / or biases. The weights can represent the connection strength between neurons in the neural network 130. For example, at a neuron of the neural network 130, an intermediate result from a previous layer or neuron can be modified based on its corresponding weight and a predetermined mathematical operation of the neuron, and then the result can be used to generate an output value. These biases can control the activation function of the neural network 130. In addition, for example, the neural network 130 can be adjustable in terms of regularization parameters, which introduce penalty terms based on the size of the weights, the number and type of layers, the type of activation function, the learning rate, the batch size, the loss rate, etc.

[0025] The neural network 130 may have a specific data flow direction, that is, the neural network 130 may perform operations in a forward propagation manner (feedforward), where data flows through the neural network 130 in a predetermined direction. For example, input data may be provided to an input layer of the neural network 130, and data of intermediate results of each layer may be propagated through subsequent downstream layers of the neural network 130 until reaching an output layer.

[0026] The architecture of the neural network 130 is further described as follows Figure 1bAs shown. The neural network 130 has a plurality of layers 140, such as layers 140-1 and 140-2, and a plurality (a plurality) of task-specific output interfaces 150, such as output interfaces 150-1 and 150-2. The plurality of layers 140 may have any number (n≥2). Figure 1b In the example of , two task-specific output interfaces 150 are shown. However, in other examples, there may be any number (m≥2) of task-specific output interfaces.

[0027] The multiple layers 140 are coupled to each other, for example, the output of layer 140-1 is connected to the input of layer 140-2 for data flow, thereby defining the above-mentioned data flow direction. In other words, the multiple layers 140 are stacked one layer on top of another. The multiple layers 140 are further coupled to the downstream task-specific output interface 150, so that the multiple layers are shared by the task-specific output interface 150. That is, the multiple layers 140 include at least one non-task-specific (i.e., shared by multiple tasks or used for multiple tasks) layer, and are therefore shared by multiple task-specific output interfaces 150. Optionally, the multiple layers 140 may also include task-specific layers, for example, the data flow through the multiple layers 140 may be divided into multiple processing pipelines, which converge or intersect further downstream.

[0028] The plurality of layers 140 may be or include the backbone of the neural network 130, and the output interface 150 may be or include, for example, the head of the neural network 130. The backbone refers to the basic architecture or feature extractor of the neural network 130 used in multi-task learning. The head or task head refers to a portion of the neural network 130 that can be added to the backbone to perform a task-specific task. The head includes at least one layer that obtains the extracted features from the backbone and produces the final output of the target task. In some examples, the head may be task-specific, so it is not shared between different tasks.

[0029] For multi-task learning, the backbone and head are combined to create a neural network 130. Multi-task learning is a learning paradigm in which a single model (neural network 130) is trained to perform multiple tasks. In this approach, the neural network 130 is able to jointly learn from multiple tasks in the form of shared multiple layers 140, exploiting commonalities and relationships between tasks, while task-specific characteristics of the input data are considered separately in the output interface 150.

[0030] exist Figure 1bIn the example of FIG. 1 , the most downstream layer of the plurality of layers 140 is directly coupled to two output interfaces 150-1, 150-2, thereby creating two processing pipelines, each for a different task. That is, the data flow through the neural network 130 can be divided at the output interface 150. Each output interface 150 may include any number of layers, including an output layer that is the most downstream layer of the neural network 130 for one task.

[0031] Multiple layers 140 include at least one convolutional layer. For example, layer 140-1 or 140-2 or any other layer in multiple layers 140 can be a convolutional layer. Other layers in multiple layers 140 can be of any type, for example, at least one of a dense layer, a pooling layer, a recurrent layer, a long short-term memory (LSTM) layer, a normalization layer, a dropout layer, or the like. In some examples, multiple layers 140 include at least two convolutional layers shared by a task-specific output interface 150, or all layers in multiple layers 140 can be convolutional.

[0032] A convolutional layer (Convolution layer or Convlayer) is capable of applying a convolution operation to data input to the convolution layer. For example, the data may be provided as a 3D tensor. A convolutional layer may have a set of learnable filters or kernels (convolution kernels) which may have a 2D matrix data structure. These filters may slide (convolve) over the data to perform element-wise multiplication with local regions of the tensor at different locations. The convolution operation involves applying these filters to the data by sliding them over the tensor, and at each location, the filter elements (weights) are multiplied by the corresponding elements in the local region of the tensor. The results may be summed or otherwise processed to produce a single value, such as in the form of an output tensor. A convolutional layer may have a certain stride, which refers to the step size with which the filter slides over the data.

[0033] In some examples, the plurality of layers 140 may include or may be at least one encoder. For example, (one or more) convolutional layers may be encoders. An encoder refers to a data architecture capable of converting input data into different representations or embeddings. The encoder thereby compresses the input data into a lower dimensional representation. By reducing the dimensionality of the data, the encoder can achieve more efficient and effective processing by providing more informative data and better generalization for new or unseen data.

[0034] Depending on the task, the task-specific output interfaces 150 may have a certain data architecture, for example, they may include or may be a decoder. For example, these tasks may be computer vision tasks. For object detection, it may include any detector architecture, for example including a single-step multi-box detector layer. They may typically include at least one of a fully connected layer, a single-step multi-box detector layer, a fully convolutional layer, and a top-down layer.

[0035] A fully connected layer (or dense layer) is a neural network layer in which each neuron in the layer is connected to every neuron in the previous layer. The purpose of such a fully connected layer can be to learn global patterns and relationships in the data. Fully connected layers can be used in the final stages of the neural network 130 for tasks such as classification and regression.

[0036] The Single-Shot Multi-Box Detector (SSD) layer can detect multiple objects in an image. It combines multiple convolutional feature maps of different scales and aspect ratios to predict bounding boxes and associated class probabilities for objects in the input image.

[0037] The fully convolutional layer uses convolution operations exclusively without using any fully connected layers. Compared to fully connected layers that operate on fixed-sized input data, the fully convolutional layer can process variable-sized input data while preserving spatial information. The fully convolutional layer can be used for semantic segmentation tasks or image-to-image conversion.

[0038] Top-down layers can be used, in particular, for (hierarchical) image or data generation or refinement, image super-resolution, image segmentation, pose estimation, etc. In such a top-down architecture, the layer generates data or predictions by first creating a higher-level representation and then gradually refining it in a step-wise manner. It can have a series of deconvolutional or upsampling layers followed by convolutional or downsampling layers. Upsampling layers spatially enlarge the representation, while downsampling layers compress the representation.

[0039] Use reference Figure 1b The architecture of the neural network 130 explained above is used to perform the training of the neural network 130 in the following manner: the processing circuit 110 ( Figure 1a ) is configured to train the neural network 130 to perform a plurality of (at least two) tasks. The task may be or may include, for example, at least one of image classification, object detection, image segmentation, and pose estimation. Optionally, the task may include at least two tasks in one task category: for example, the task may include different object classification tasks, such as fruit classification and animal classification.

[0040] The processing circuit 110 may train the neural network 130 by executing a training framework to adjust the weights or biases of the neural network 130. For example, the training framework may include a data architecture or a set of rules / operations to compare the output of the neural network 130 for a certain input with a label or expected output associated with the input. The processing circuit 110 may therefore receive (e.g., via the interface circuit 120) or determine input (training) data. The training data may be multi-domain image data, for example, that is, the processing circuit 110 may be configured to train the neural network 130 to perform a task based on the multi-domain image data. Multi-domain image data refers to a data set containing image samples from multiple different domains or sources (e.g., indoor and outdoor scene images). Each domain represents a different distribution of image data, with its own unique characteristics, patterns, and statistical properties that are specific to a certain task.

[0041] Training can be performed using any learning method. For example, one or more of the following or other learning methods can be used. Supervised learning enables the neural network 130 to be trained on a labeled data set, where both the input training data and the corresponding target labels are provided to the training framework. The task of the neural network 130 is to learn the mapping between the input and the label. Tasks in supervised learning can include classification and regression. Unsupervised learning omits labels, and training is based on unlabeled data sets, that is, only input training data is provided without any corresponding target labels. The task of the neural model 130 is to discover patterns, structures, or representations in the data without explicit guidance. Clustering, dimensionality reduction, and generative modeling can be unsupervised learning tasks. Semi-supervised learning combines elements of supervised learning and unsupervised learning. It involves training the neural model 130 on a mixed data set of labeled data and unlabeled data. Reinforcement learning is a learning paradigm in which an agent learns to make decisions and take actions in a training framework to increase cumulative rewards. The agent receives feedback in the form of rewards or penalties based on its actions and learns to improve its decision making through trial and error.

[0042] Training the neural network 130 may include adjusting parameters, such as weights or biases, of the neural network 130. The parameters may be adjusted based on back propagation (e.g., to iteratively update the parameters), etc. For example, the processing circuit 110 may adjust the neural network 130 based on a specified loss or error function to reduce the difference between the predicted output and the desired output.

[0043] In order to adjust the parameters of the neural network 130, any training algorithm may be used, such as gradient descent, momentum, adaptive gradient (Adagrad), etc. In order to obtain high computational efficiency, especially for large training data sets, stochastic gradient descent may be used. In this case, the processing circuit 110 is further configured to use the stochastic gradient descent (SGD) to train the neural network 130. SGD is an optimization algorithm that is different from traditional gradient descent, which updates parameters based on the average gradient of the loss function relative to the entire training data set, while SGD calculates gradients and updates parameters for a subset of the training data set (e.g., for each individual training sample). This makes the update process more frequent, resulting in the optimization zigzagging around the optimal solution. For example, the processing circuit 110 may initialize the parameters of the neural network 130 with random or initial values, for example, shuffling the training data before each epoch (completely passing through the entire training data set), iterating the training samples so that for each sample in the training data set, the gradient of the loss function relative to the parameter is calculated, and the parameters are updated using the calculated gradient and, for example, the learning rate (a hyperparameter that determines the step size of the update). This can be repeated for a fixed number of rounds, or until a convergence criterion is met.

[0044] To mitigate the gradient explosion problem during training, the device 100 may use adaptive gradient clipping. For example, the processing circuit 110 may be further configured to train the neural network 130 using the adaptive gradient clipping. Gradient clipping involves limiting the magnitude of the gradient during the training phase. For example, a threshold may be set to prevent the gradient from exceeding the threshold, which may help stabilize the training and prevent the gradient value from becoming very large. Adaptive gradient clipping dynamically adjusts the clipping threshold based on the magnitude of the gradient. For example, the processing circuit 110 may perform adaptive gradient clipping by calculating the gradient of the parameter relative to the loss function, determining the norm (magnitude) of the gradient using any metric such as the L2 norm (Euclidean norm) or the maximum norm, and determining the clipping threshold based on the gradient norm. If the gradient norm exceeds a predefined threshold, the processing circuit 110 may rescale or clip the gradients so that they are within a specified range. The model parameters may then be updated using the clipped gradients instead of the original gradients.

[0045] In addition, the processing circuit 110 can jointly train the backbone and the head, or train the backbone separately and jointly train the pre-trained backbone and the head. In the latter case, the backbone remains fixed (frozen) during training, and the head is only fine-tuned on the target task using a smaller data set. By utilizing the pre-trained backbone, the neural network 130 can benefit from the general features learned from the source task. For example, in image recognition, transfer learning can be applied by using an already pre-trained backbone (e.g., a backbone trained on ImageNet) as a feature extractor and then adding a task-specific head for a specific image classification task (e.g., identifying different animals, objects, or scenes).

[0046] Processing circuit 110 is configured to train neural network 130 by applying weight normalization to weights of at least one convolutional layer in multiple layers 140. Weight normalization may involve modifying the weights of the convolutional layer, for example, so that they have a certain mean and / or variance, such as zero mean and unit variance. For further improvement, processing circuit 110 may apply weight normalization to the respective weights of all convolutional layers in multiple layers 140. Unlike traditional multi-task training, apparatus 100 may use weight normalization to improve the performance of neural network 130 and reduce the memory required for multi-task learning.

[0047] Traditional multi-task training is based on specific normalization layers downstream of the convolutional layers, such as batch normalization layers. However, normalization layers may have difficulty handling multi-domain data, especially when these domains have significantly different statistical distributions. The main reason is that the normalization layer normalizes the output of the convolutional layer, which may produce domain shift (i.e., conflicting normalization occurs when activations or outputs of different domains are mixed with each other and jointly normalized), domain bias (i.e., when normalization aims to make the optimization picture perform better, it leads to bias towards certain domains, resulting in poor generalization), or unwanted cross-domain normalization (i.e., the normalization parameters of a small batch cannot represent the data distribution of any single domain). In contrast, the device 100 can reduce the number of (batch) normalization layers or completely omit the (batch) normalization layers. In contrast, weight normalization inside the convolutional layer can stabilize and accelerate the convergence of training.

[0048] To accommodate different multi-domain data, processing circuit 110 may be configured to apply scaled weight normalization to the weights. Scaled weight normalization introduces an additional constant scaling parameter γ to the normalized weights. A properly chosen γ may keep the convolutional layer variance-preserving for the activation function, i.e., the scaling parameter may be set according to the activation function of the layer. Mathematically, the scaled normalized weights of the convolutional layer are It can be expressed as Equation 1:

[0049]

[0050] Equation 1, where μ and σ are the mean and variance (standard deviation) of the (original) weights W of a row i, N is the fan-in of the convolutional layer, and i, j are the row and column indices of the weight matrix. The mean and variance are determined by the following equations 2 and 3:

[0051] μ i =(1 / N)∑ j W ij

[0052] Equation 2

[0053]

[0054] Equation 3

[0055] In contrast to conventional layers for multi-task training, the apparatus 100 can use scaled weight normalization. Since the mathematical expectation of the output of the convolutional layer is 0, scaled weight normalization can eliminate mean shift. It can further preserve the variance of the input data because the variance of the scaled normalized weights depends largely on the nonlinear function (e.g., rectified linear unit ReLU) applied to the convolution result performed by the convolutional layer.

[0056] Weight normalization can enable the neural network 130 to omit additional normalization layers, such as batch normalization layers. It can implement the modification of weights instead of normalization of activations (outputs of activation functions). This can prevent the variance of multi-domain input data from being unstable due to normalization. The implementation of omitting the normalization layer can be as follows: for example, at least two convolutional layers in the plurality of layers 140 can be coupled to each other without a batch normalization layer or a normalization layer in between. For example, at least two convolutional layers in the plurality of layers 140 in a network architecture (e.g., ResNet) can be coupled to each other without a batch normalization layer or a normalization layer in between. For example, at least two convolutional layers in the plurality of layers 140 in any network architecture (except the visual geometry group (VGG)) can be coupled to each other without a batch normalization layer or a normalization layer in between. At least the upstream convolutional layer of the two convolutional layers can be weight-normalized. Depending on the architecture of the neural network 130, omitting the normalization layer may result in at least two convolutional layers (e.g., layers 140-1, 140-2) in the plurality of layers 140 being directly coupled to each other. That is, there is no layer between the two convolutional layers, and the output of one convolutional layer is directly connected to the input of the next convolutional layer. Alternatively, other layers (e.g., activation layers or recurrent layers) other than the normalization layer may be provided between the two convolutional layers.

[0057] To further improve the performance of the device 100, each of all convolutional layers in the plurality of layers 140 may be coupled to another convolutional layer without a batch normalization layer in between (e.g., directly coupled to each other). In the case of a fully convolutional backbone, all layers in the plurality of layers 140 may be directly coupled to another layer in the plurality of layers 140, or coupled to the other layer without a (batch) normalization layer in between. Alternatively, only a subset of the convolutional layers in the plurality of layers 140 may omit subsequent normalization layers - then at least one convolutional layer may be coupled to a downstream normalization layer. Depending on the degree of implementation without normalization layers, the plurality of layers 140 may include fewer batch normalization layers than convolutional layers, i.e., at least 1 (2 or 3, etc.) fewer batch normalization layers than convolutional layers. In a full implementation, the plurality of layers 140 does not include any (batch) normalization layers.

[0058] The apparatus 100 can provide normalization-free multi-task learning for multi-domain data. This can be achieved by removing the batch normalization layer in the backbone and replacing the convolutional layer with a scaled weight-normalized convolutional layer.

[0059] Figure 2 Another example of the architecture of the neural network 230 is shown. The neural network 230 may be obtained and trained by an apparatus as described herein (eg, apparatus 100).

[0060] The neural network 230 has a plurality of layers 240 shared by a task-specific output interface 250. The plurality of layers 240 include at least two convolutional layers 240-1, 240-2 that have been subjected to weight normalization (WSConv). Figure 2 In the example of , the batch normalization layers 260-1, 260-2 after the convolutional layers 240-1, 240-2 are removed from the architecture. Therefore, the convolutional layers 240-1, 240-2 are directly coupled to each other.

[0061] The task-specific output interface 250 includes four output interfaces 250-1 to 250-4. Figure 2 In the example, the task-specific output interface 250 includes a first output interface 250-1 having two top-down layers for pose estimation; a second output interface 250-2, which is a fully convolutional network (FCN) with two layers for image segmentation; a third output interface 250-3, which has a single-step multi-box detector (SSD) including a shared layer and two parallel layers for object detection; and a fourth output interface 250-4, which has a fully connected (FC) layer for classification.

[0062] Figure 3A flowchart of an example of a method 300 for training a neural network is shown. The method 300 may be performed by an apparatus as described herein (eg, apparatus 100).

[0063] The method 300 includes obtaining 310 a neural network having a plurality of task-specific output interfaces and a plurality of layers shared by the task-specific output interfaces. The method 300 also includes training 320 the neural network to perform a plurality of tasks, wherein training 320 the neural network includes applying weight normalization to weights of at least one convolutional layer of the plurality of layers.

[0064] Further details and aspects of method 300 will be provided in conjunction with the techniques presented above or one or more examples (e.g., reference Figure 1a , Figure 1b and Figure 2 The method 300 may include one or more additional optional features corresponding to one or more aspects of the proposed technology, or one or more of the above examples.

[0065] Figure 4 A flow chart of an example of a method 400 for training and deploying a neural network (e.g., neural network 130 or 230) is shown. Method 400 may be computer-implemented and performed at least in part by an apparatus as described herein (e.g., apparatus 100). Deployment of the neural network may include conversion and compression for the trained neural network.

[0066] Method 400 includes training 410 a neural network to perform a plurality of tasks. Training 410 the neural network includes applying 420 weight normalization to the weights of at least one convolutional layer in a plurality of layers of the neural network. Training 410 the neural network also includes using 430 PyTorch and using 440 Keras. PyTorch and Keras are deep learning frameworks for building and training neural networks. PyTorch is an open source deep learning framework that can allow for easier debugging and more natural integration with Python programming. Keras is an open source deep learning library designed to provide a user-friendly and advanced application programming interface (API) for building neural networks.

[0067] The method 400 also includes deploying the trained neural network by quantizing using 450 and using TensorFlow Lite using 460. Quantization and TensorFlow Lite are two concepts related to deep learning and model deployment, especially for resource-constrained devices such as mobile phones, embedded systems, and Internet of Things (IoT) devices.

[0068] Quantization can reduce the memory footprint and computational requirements of a neural network while maintaining acceptable performance. For example, model parameters (weights and biases) of a neural network and activations can initially be stored as 32-bit floating point numbers. However, 32-bit floating point representations can be memory intensive and computationally expensive, especially on devices with limited resources and processing power. Quantization involves converting these 32-bit floating point numbers to lower precision fixed point or integer representations, for example, 8-bit integers. Quantization can be performed during model training (training quantization) or during model conversion after training (post-training quantization).

[0069] TensorFlow Lite is a lightweight, cross-platform framework for running neural networks on mobile and embedded devices. It is a subset of the full TensorFlow framework optimized for efficient reasoning on resource-constrained platforms. It provides multiple deployment options, such as running neural networks locally on the device, using neural networks in Android or iOS applications, or using neural networks for edge computing scenarios.

[0070] For example, a neural network can be deployed on 8 or 16 megabytes of static random access memory (SRAM). A neural network can achieve 1, 2, or 20 TOPS (tera operations per second), with an efficiency of 5 or 11 TOPS per watt, respectively. It can also have 8 / 4 / 2 bits of weight precision or 8 / 4 / 2 bits of mixed precision.

[0071] Figure 5 An example of an apparatus 500 for performing multiple tasks using a trained neural network 530 is shown. Figure 5 The example of may correspond to the inference phase of the neural network explained above (e.g., neural network 130 or 230). Trained neural network 530 may be the result of training the neural network. For example, the architecture of trained neural network 530 may thus correspond to the architecture of neural network 130 or 230. Apparatus 500 may be external to an apparatus for training a neural network (e.g., apparatus 100). Alternatively, apparatus 500 may be integrated into a system together with an apparatus for training a neural network.

[0072] The apparatus 500 includes a processing circuit 510 and an optional interface circuit 520 for exchanging data between the interface circuit 520, the processing circuit 510 and an external device (e.g., the apparatus 100). The processing circuit 510 is configured to obtain (e.g., receive or generate) a trained neural network 530. Optionally, for example, the processing circuit 510 can be configured to deploy the trained neural network 530 on the apparatus 100 (e.g., an embedded system). The trained neural network 530 has a plurality of task-specific output interfaces and a plurality of layers shared by these task-specific output interfaces, as described in reference to FIG. Figure 1b and Figure 2 The task-specific output interface may include, for example, at least one of a fully connected layer, a single-step multi-box detector layer, a fully convolutional layer, and a top-down layer.

[0073] At least two convolutional layers in the plurality of layers are coupled to each other without a batch normalization layer and / or a normalization layer in between. Additionally or alternatively, at least two convolutional layers in the plurality of layers are directly coupled to each other. For example, at least two convolutional layers in the plurality of layers in a network architecture (e.g., ResNet) may be coupled to each other without a batch normalization layer or a normalization layer, or may be directly coupled to each other. For example, at least two convolutional layers in the plurality of layers in any network architecture other than VGG may be coupled to each other without a batch normalization layer or a normalization layer in between.

[0074] The processing circuit 510 is further configured to perform a plurality of tasks using the trained neural network 530. For example, the processing circuit 510 may execute the trained neural network 530. Performing the plurality of tasks may include inputting input data (e.g., multi-domain image data) to the trained neural network 530, which outputs an output based on the input data at one or more of its task-specific output layers. The output may be used to perform one or more of the plurality of tasks, or may correspond to the performance of a task. The task may include at least one of image classification, object detection, image segmentation, and pose estimation.

[0075] The apparatus 500 can simplify the deployment of multi-task neural networks: only a single model may be needed to support multiple tasks, for example four tasks. The model can be integrated on a chip for inference, thereby reducing deployment costs. During inference, the apparatus 500 can provide fast task switching without reloading additional models and improve the performance of tasks. They can provide inherent support for next-generation chips. Since inference can be implemented as a fully integrated application, privacy issues can be prevented (on-device AI model prediction).

[0076] As a further enhancement of the apparatus 500, each of all convolutional layers in the plurality of layers may be coupled to another convolutional layer with no (batch) normalization layer in between. For example, the plurality of layers may include fewer batch normalization layers than convolutional layers, i.e., at least one batch normalization layer may be omitted. For a full implementation, the plurality of layers do not include any batch normalization layers.

[0077] Depending on the implementation, both convolutional layers may be weight normalized, or all convolutional layers in the plurality of layers may be weight normalized. For example, the weight normalization may be scaled weight normalization. This may enable the additional use of variance-preserving ReLU.

[0078] Further details and aspects of the apparatus 500 will be described in conjunction with the techniques presented above or one or more examples (e.g., reference Figure 1a , Figure 1b , Figure 2 , Figure 3 and Figure 4 The apparatus 500 may include one or more additional optional features corresponding to one or more aspects of the proposed technology or one or more of the above-described examples.

[0079] The apparatus 500 can enable the removal of normalization layers from multi-task learning of multi-domain data. The neural network 530 can be deployed to hardware with small memory and computing power. The apparatus 500 can also further improve the performance of the task. For example, in a specific implementation of the apparatus 500, the accuracy of object classification can be increased from 60.16% to 92.27%, the intersection-over-union ratio of image segmentation can be increased from 55.28% to 72.69%, the average accuracy of object detection can be increased from 51.70% to 69.90%, and the accuracy of pose estimation can be increased from 37.69% to 58.01%.

[0080] In addition, the neural network model 530 trained based on the techniques described herein can predict results with a smaller batch size (e.g., as small as one), while the prior art requires a larger batch size. This can be particularly beneficial for resource-constrained devices.

[0081] Figure 6 A flowchart of an example of a method 600 for performing multiple tasks using a trained neural network is shown. The method 600 may be performed by an apparatus as described herein (eg, apparatus 500).

[0082] The method 600 includes obtaining 610 a trained neural network having a plurality of task-specific output interfaces and a plurality of layers shared by the task-specific output interfaces. At least two convolutional layers in the plurality of layers are coupled to each other without a batch normalization layer therebetween. The method 600 also includes performing 620 a plurality of tasks using the trained neural network.

[0083] Further details and aspects of method 600 will be described in conjunction with the techniques presented above or one or more examples (e.g., reference Figure 5 The method 600 may include one or more additional optional features corresponding to one or more aspects of the proposed technology or one or more of the above examples.

[0084] Here are some examples of the concepts presented:

[0085] An example (e.g., Example 1) relates to an apparatus for training a neural network, the apparatus comprising a processing circuit configured to obtain a neural network having a plurality of task-specific output interfaces and a plurality of layers shared by the task-specific output interfaces, and to train the neural network to perform a plurality of tasks by applying weight normalization to weights of at least one convolutional layer of the plurality of layers.

[0086] Another example (e.g., Example 2) relates to the previous example (e.g., Example 1) or any other example, and also includes: the processing circuit is further configured to apply weight normalization to the respective weights of all convolutional layers in the multiple layers.

[0087] Another example (eg, Example 3) relates to the previous example (eg, one of Example 1 or Example 2) or any other example, further comprising: the processing circuit is configured to apply the scaled weight normalization to the weight.

[0088] Another example (e.g., Example 4) relates to a previous example (e.g., one of Examples 1 to 3) or any other example, and further includes: at least two convolutional layers of the multiple layers are coupled to each other without a batch normalization layer in between.

[0089] Another example (e.g., Example 5) relates to the previous example (e.g., one of Examples 1 to 4) or any other example, and also includes: at least two convolutional layers of the multiple layers are coupled to each other without a normalization layer in between.

[0090] Another example (e.g., Example 6) relates to the previous example (e.g., one of Examples 1 to 5) or any other example, and also includes: at least two convolutional layers in the multiple layers are directly coupled to each other.

[0091] Another example (e.g., Example 7) relates to the previous example (e.g., one of Examples 1 to 6) or any other example, and also includes: each of all convolutional layers in the multiple layers is coupled to another convolutional layer without a batch normalization layer in between.

[0092] Another example (e.g., Example 8) relates to the previous example (e.g., one of Examples 1 to 7) or any other example, and also includes: the multiple layers include fewer batch normalization layers than convolutional layers.

[0093] Another example (e.g., Example 9) relates to the previous example (e.g., one of Examples 1 to 8) or any other example, and also includes: the multiple layers do not include any batch normalization layers.

[0094] Another example (e.g., Example 10) relates to a previous example (e.g., one of Examples 1 to 9) or any other example, further comprising: at least one convolutional layer is based on a variance-preserving rectified linear unit ReLU.

[0095] Another example (e.g., Example 11) relates to the previous example (e.g., one of Example 1 or Example 10) or any other example, and also includes: the processing circuit is further configured to train the neural network using stochastic gradient descent.

[0096] Another example (e.g., Example 12) relates to the previous example (e.g., one of Example 1 or Example 11) or any other example, and also includes: the processing circuit is further configured to train the neural network using adaptive gradient clipping.

[0097] Another example (e.g., Example 13) relates to a previous example (e.g., one of Examples 1 to 12) or any other example, and also includes: the processing circuit is configured to train a neural network to perform at least two tasks, wherein the task includes at least one of image classification, object detection, image segmentation, and pose estimation.

[0098] Another example (e.g., Example 14) relates to the previous example (e.g., one of Example 1 or Example 13) or any other example, and also includes: the processing circuit is further configured to train the neural network based on the multi-domain image data.

[0099] Another example (eg, Example 15) relates to a previous example (eg, one of Examples 1 to 14) or any other example, further comprising: the plurality of layers comprising at least one encoder.

[0100] Another example (e.g., Example 16) relates to a previous example (e.g., one of Examples 1 to 15) or any other example, further including: the multiple layers include at least two convolutional layers shared by a task-specific output interface.

[0101] Another example (e.g., Example 17) relates to a previous example (e.g., one of Examples 1 to 16) or any other example, further including: the plurality of layers comprising a backbone of the neural network.

[0102] Another example (e.g., Example 18) relates to a previous example (e.g., one of Examples 1 to 17) or any other example, further including: the task-specific output interface includes at least one of a fully connected layer, a single-step multi-box detector layer, a fully convolutional layer, and a top-down layer.

[0103] The examples (e.g., Example 19) relate to a method for training a neural network, the method comprising: obtaining a neural network having multiple task-specific output interfaces and multiple layers shared by these task-specific output interfaces, and training the neural network to perform multiple tasks, wherein training the neural network comprises applying weight normalization to weights of at least one convolutional layer among the multiple layers.

[0104] An example (e.g., Example 20) relates to an apparatus for performing multiple tasks using a trained neural network, the apparatus comprising a processing circuit configured to: obtain a trained neural network, wherein the trained neural network has multiple task-specific output interfaces and multiple layers shared by the task-specific output interfaces, wherein at least two convolutional layers of the multiple layers are coupled to each other without a batch normalization layer therebetween; and perform multiple tasks using the trained neural network.

[0105] An example (e.g., Example 21) relates to an apparatus for performing multiple tasks using a trained neural network, the apparatus comprising a processing circuit configured to: obtain a trained neural network, wherein the trained neural network has multiple task-specific output interfaces and multiple layers shared by the task-specific output interfaces, wherein at least two convolutional layers of the multiple layers are directly coupled to each other; and perform multiple tasks using the trained neural network.

[0106] Another example (e.g., Example 22) relates to the previous example (e.g., Example 20 or 21) or any other example, and also includes: at least two convolutional layers are coupled to each other without a normalization layer in between.

[0107] Another example (e.g., Example 23) relates to a previous example (e.g., one of Examples 20 to 22) or any other example, and also includes: two convolutional layers are directly coupled to each other.

[0108] Another example (e.g., Example 24) involves a previous example (e.g., one of Examples 20 to 23) or any other example, and also includes: each of all convolutional layers in the multiple layers is coupled to another convolutional layer without a batch normalization layer in between.

[0109] Another example (e.g., Example 25) relates to a previous example (e.g., one of Examples 20 to 24) or any other example, further including: the multiple layers include fewer batch normalization layers than convolutional layers.

[0110] Another example (e.g., Example 26) relates to a previous example (e.g., one of Examples 20 to 25) or any other example, and also includes: the multiple layers do not include any batch normalization layers.

[0111] Another example (e.g., Example 27) relates to a previous example (e.g., one of Examples 20 to 24) or any other example, and also includes: the two convolutional layers are weight normalized.

[0112] Another example (e.g., Example 28) relates to the previous example (e.g., Example 27) or any other example, and also includes: all convolutional layers in the multiple layers are weight normalized.

[0113] Another example (e.g., Example 29) relates to a previous example (e.g., one of Examples 20 to 28) or any other example, further comprising: two convolutional layers are scaled weight normalized.

[0114] Another example (e.g., Example 30) relates to a previous example (e.g., one of Examples 20 to 29) or any other example, further comprising: two convolutional layers are based on variance-preserving rectified linear units ReLU.

[0115] Another example (e.g., Example 31) relates to a previous example (e.g., one of Examples 20 to 30) or any other example, and also includes: the processing circuit is configured to perform at least two tasks using a trained neural network, wherein the task includes at least one of image classification, object detection, image segmentation, and pose estimation.

[0116] Another example (e.g., Example 32) relates to the previous example (e.g., one of Examples 20 to 31) or any other example, and also includes: the processing circuit is further configured to perform a task based on the multi-domain image data.

[0117] Another example (e.g., Example 33) relates to a previous example (e.g., one of Examples 20 to 32) or any other example, further including: multiple layers include at least one encoder.

[0118] Another example (e.g., Example 34) relates to a previous example (e.g., one of Examples 20 to 33) or any other example, further including: the task-specific output interface includes at least one of a fully connected layer, a single-step multi-box detector layer, a fully convolutional layer, and a top-down layer.

[0119] An example (e.g., Example 35) relates to a method of performing multiple tasks using a trained neural network, the method comprising: obtaining a trained neural network having multiple task-specific output interfaces and multiple layers shared by the task-specific output interfaces, wherein at least two convolutional layers of the multiple layers are coupled to each other without a batch normalization layer therebetween; and performing multiple tasks using the trained neural network.

[0120] Another example (eg, Example 36) relates to a non-transitory machine-readable medium having stored thereon a program having program code for performing the method of any one of Examples 19 or 35 when the program is executed on a processor or programmable hardware.

[0121] Another example (eg, Example 37) involves a program having program code, which, when executed on a processor or programmable hardware, is used to perform the method of any of Examples 19 or 35.

[0122] Aspects and features described with respect to a particular example in the previous examples may also be combined with one or more further examples to replace the same or similar features of the further examples, or to additionally introduce these features into the further examples.

[0123] Examples may also be or relate to a (computer) program including a program code, which, when executed on a computer, a processor or other programmable hardware component, is used to perform one or more of the above methods. Therefore, the steps, operations or processes of the above different methods may also be performed by a programmed computer, a processor or other programmable hardware component. Examples may also cover program storage devices, such as digital data storage media, which are machine-readable, processor-readable or computer-readable, and encode and / or contain machine-executable, processor-executable or computer-executable programs and instructions. For example, the program storage device may include, or be, a digital storage device, a magnetic storage medium such as a disk and a tape, a hard drive or an optically readable digital data storage medium. Other examples may also include a computer, a processor, a control unit, a (field) programmable logic array ((F)PLA), a (field) programmable gate array ((F)PGA), a graphics processor unit (GPU), an application-specific integrated circuit (ASIC), an integrated circuit (IC) or a system on a chip (SoC) system, which is programmed to perform the steps of the above methods.

[0124] It should also be understood that the disclosure of several steps, processes, operations or functions disclosed in the specification or claims should not be interpreted as implying that these operations must rely on the described order, unless explicitly stated in individual cases or necessary for technical reasons. Therefore, the preceding description does not limit the execution of several steps or functions to a certain order. In addition, in further examples, a single step, function, process or operation may include and / or be decomposed into several sub-steps, functions, processes or operations.

[0125] If some aspects related to an apparatus or system have been described, these aspects should also be understood as descriptions of the corresponding methods. For example, blocks, devices or functional aspects of an apparatus or system may correspond to features of the corresponding methods, such as method steps. Therefore, aspects described about methods should also be understood as descriptions of properties or functional features of the corresponding blocks, corresponding elements, corresponding apparatuses or corresponding systems.

[0126] The following claims are hereby incorporated into the detailed description, where each claim may stand on its own as a separate example. It should also be noted that, although in the claims, a dependent claim refers to a specific combination with one or more other claims, other examples may also include a combination of a dependent claim with the subject matter of any other dependent or independent claim. Unless it is stated in individual cases that a specific combination is not preferred, such a combination is explicitly proposed here. In addition, features of a claim shall also be included in any other independent claim, even if the claim is not directly defined as dependent on the other independent claim.

Claims

1. An apparatus for training a neural network, the apparatus comprising a processing circuit, the processing circuit being configured to: obtaining the neural network having a plurality of task-specific output interfaces and a plurality of layers shared by the task-specific output interfaces; and The neural network is trained to perform a plurality of tasks by applying weight normalization to weights of at least one convolutional layer among the plurality of layers.

2. The device according to claim 1, wherein: The processing circuit is configured to apply a scaled weight normalization to the weights.

3. The device according to claim 1, wherein: At least two convolutional layers among the plurality of layers are coupled to each other without a normalization layer or a batch normalization layer interposed therebetween.

4. The device according to claim 1, wherein: At least two convolutional layers among the plurality of layers are directly coupled to each other.

5. The device according to claim 1, wherein: The plurality of layers includes fewer batch normalization layers than convolutional layers.

6. The device according to claim 1, wherein: The processing circuit is configured to train the neural network to perform at least two tasks, wherein the tasks include at least one of image classification, object detection, image segmentation, and pose estimation.

7. The device according to claim 1, wherein: The task-specific output interface includes at least one of a fully connected layer, a single-step multi-box detector layer, a fully convolutional layer, and a top-down layer.

8. A method for training a neural network, the method comprising: Obtaining the neural network having a plurality of task-specific output interfaces and a plurality of layers shared by the task-specific output interfaces; as well as The neural network is trained to perform a plurality of tasks, wherein training the neural network comprises applying weight normalization to weights of at least one convolutional layer of the plurality of layers.

9. An apparatus for performing a plurality of tasks using a trained neural network, the apparatus comprising a processing circuit configured to: The trained neural network is obtained, wherein: The trained neural network has a plurality of task-specific output interfaces and a plurality of layers shared by the task-specific output interfaces, wherein at least two convolutional layers of the plurality of layers are coupled to each other without a batch normalization layer interposed therebetween; and The plurality of tasks are performed using the trained neural network.

10. A method for performing a plurality of tasks using a trained neural network, the method comprising: Obtaining the trained neural network having a plurality of task-specific output interfaces and a plurality of layers shared by the task-specific output interfaces, wherein at least two convolutional layers of the plurality of layers are coupled to each other without a batch normalization layer interposed therebetween; and The plurality of tasks are performed using the trained neural network.

Images