Method for training multi-task neural network

Abstract

To provide a method for training a multi-task neural network.SOLUTION: Some embodiments are directed to training a multi-task neural network. The multi-task neural network is configured to receive one input and generate a plurality of outputs. During training, task weight is calculated to weight individual losses. In order to normalize the task weight, a normalization function is applied.SELECTED DRAWING: Figure 3b

Description

[Technical Field]

[0001] The subject matter disclosed herein relates to methods for training multitasking neural networks, systems for training multitasking neural networks, and computer storage media. [Background technology]

[0002] background With the proliferation of neural networks in diverse domains, multi-task learning (MTL) is gaining importance. Multi-task learning (MTL) aims to solve multiple tasks simultaneously in a mutually beneficial manner. MTL allows related tasks to share their knowledge, thus making learning more efficient. Furthermore, MTL increases the amount of available training data [3, 29, 33].

[0003] One of the main challenges is finding the right implicit or explicit balance between each task to achieve good performance across tasks. An effective approach provides task-specific weights during optimization.

[0004] Not explicitly weighting tasks, also known as equal weighting (EW), has many potential pitfalls. Different tasks may be measured on different scales, such as meters and millimeters, which can lead to dramatically different loss scaling across tasks. Furthermore, some tasks may be more difficult and require more resources than others. Therefore, various proposals have been made for dedicated weighting methods that dynamically adjust weights along the course of training [21, 7, 15, 37, 20, 16, 2, 4, 22, 6].

[0005] Uncertainty Weighting (UW)

[16] is a gradient-based multi-task optimization (MTO) approach. In UW, task weights are learned using gradient descent. However, UW also exhibits several drawbacks. First, we observe that UW can be affected by poor initialization and inertia. Because uncertainty weights are typically initialized equally for all tasks, reaching task-specific and epoch-specific optima via gradient descent can be slow, especially since task weights often vary in magnitude. We refer to this phenomenon as inertial updating. We have empirically demonstrated this phenomenon. Furthermore, we observe that UW is prone to overfitting.

[0006] Another approach to obtaining task weights is to use a brute-force grid search for optimal fixed weights, which is called scalarization. Scalarization provides excellent performance for a range of MTL problems. The loss weights are tuned by performing an extensive grid search, which is computationally expensive. Scalarization can only be applied to problems with a small number of tasks because the combinatorial complexity becomes prohibitive. This method is computationally expensive and therefore inapplicable in many practical scenarios. Summary of the Invention [Problem to be solved by the invention]

[0007] overview It would be advantageous to have an improved method for training multi-task neural networks, and in particular for calculating task weights. [Means for solving the problem]

[0008] In one embodiment, the multi-tasking neural network is configured to receive one input and generate multiple outputs, and the input of the multi-tasking neural network includes dimensional data obtained from a sensor. In one embodiment, calculating the task weights includes applying a normalization function so that the task weights are normalized. For example, raw task weights can be calculated first, and then the raw task weights can be normalized by applying the normalization function. This can be done while iterating over the training set. In an advantageous approach for calculating the raw task weights, the raw task weights are selected to scale the corresponding individual losses to a constant. For example, task weight ω i , 1 / L i In proportion to i L i can be chosen to be a constant, for example 1. This means that the total loss L=Σ i ω i L i This also means that is a constant, e.g., equal to the number of tasks. This can be done for each training batch.

[0009] Multi-tasking neural networks are particularly useful in situations where multiple pieces of information need to be extracted from the same data. For example, the input data may represent a real-world environment for a machine agent, e.g., a robot or vehicle, operating within the environment. The input data may include multiple input modalities. Given such information, there are different pieces of information that may need to be extracted, e.g., object segmentation, object classification, distance estimation, etc. Multi-tasking neural networks may be trained to infer the state of the environment. A similar problem arises when deriving the state of a machine agent, where the input may include different pieces of data representing the agent's state.

[0010] In one embodiment, an analytically optimal uncertainty solution is calculated, which is particularly powerful in scenarios with highly unbalanced losses. In one embodiment, Soft Optimal Uncertainty Weighting (UW-SO) normalizes this optimal solution by a normalization function, for example, a softmax function with a temperature parameter. A large set of experiments was performed between different loss weighting methods, using different data sets and architectures.

[0011] A device configured to train a multi-task neural network according to one embodiment is an electronic device, for example a computer.

[0012] An embodiment of the method may be implemented on a computer as a computer-implemented method, or in dedicated hardware, or a combination of both. Executable code for an embodiment of the method may be stored on a computer program product. Examples of computer program products include memory devices, optical storage devices, integrated circuits, servers, online software, etc. Preferably, the computer program product includes non-transitory program code stored on a computer-readable medium for performing an embodiment of the method when said program product is run on a computer.

[0013] In one embodiment, the computer program comprises computer program code configured to perform all or part of the steps of an embodiment of the method when the computer program is run on a computer. Preferably, the computer program is embodied on a computer readable medium.

[0014] Further details, aspects and embodiments are described, by way of example only, with reference to the drawings, in which elements are illustrated for simplicity and clarity and are not necessarily drawn to scale, and in which elements corresponding to elements already described may have the same reference numerals. [Brief explanation of the drawings]

[0015] [Figure 1a] FIG. 1 is a schematic diagram illustrating an example of an embodiment of a system for training a multi-task neural network. [Figure 1b] FIG. 1 is a schematic diagram illustrating an example of an embodiment of a system for evaluating a multi-task neural network. [Figure 2] FIG. 2 is a schematic diagram illustrating an example of an embodiment of a training set. [Figure 3a] FIG. 1 is a schematic diagram illustrating an example of an embodiment of a multi-task neural network. [Figure 3b] FIG. 1 is a schematic diagram illustrating an example of an embodiment for training a multi-task neural network. [Figure 4a] FIG. 10 is a diagram illustrating an example of weight ratio evolution for one embodiment of training a multitasking neural network. [Figure 4b] FIG. 10 is a diagram illustrating an example of loss evolution for one embodiment of training a multitask neural network. [Figure 5] FIG. 10 illustrates a schematic example of performance for different choices of T in one embodiment. [Figure 6a] FIG. 10 is a diagram illustrating an example of a progression of task weights for a segmentation task in one embodiment. [Figure 6b] FIG. 10 is a diagram illustrating an example of a progression of task weights for depth tasks in one embodiment. [Figure 6c] FIG. 10 is a diagram illustrating an example of a transition of task weights for normal tasks according to an embodiment. [Figure 7]FIG. 1 is a diagram illustrating an example of an embodiment of a method for training a multi-task neural network. [Figure 8a] 1 is a diagram illustrating a computer-readable medium having a writable portion containing a computer program according to one embodiment; [Figure 8b] FIG. 1 is a schematic diagram illustrating a representation of a processor system according to one embodiment.

[0016] Explanation of symbols The following list of reference symbols and abbreviations corresponds to Figures 1-6c, 8a-8b and is provided to facilitate interpretation of the drawings and should not be construed as limiting the scope of the claims. 110 Training System for Training Multitasking Neural Networks 120 Evaluation System for Evaluating Multitask Neural Networks 111,121 processor systems 112,122 Storage 113,123 Communication Interface 200 training set 210,220 training pairs 211,221 training input 212-214 Multiple training outputs 222~224 Multiple training outputs 301 Input 321-323 Task-specific heads 331-333 Task-specific outputs 341-343 Task-specific losses 350 raw task weight units 351~353 Raw task weights 360 Normalization Functions 361~363 Task Weight 370 weighted loss units 380 learning units 411 Uncertainty Weighting 412 Soft Optimal Uncertainty Weighting 413 Equal Weighting 421 Uncertainty Weighting Training 422 Testing Uncertainty Weighting 423 Training Soft Optimal Uncertainty Weighting 424 Testing Soft-Optimal Uncertainty Weighting 611 Scalar 612 Equal Weighting 613 Uncertainty Weighting 614 Geometric Loss Strategy (GLS) 615 Impartial Multi-Task Learning (IMTL) 616 Optimal Uncertainty Weighting 617 Soft Optimal Uncertainty Weighting 1000,1001 Computer-readable medium 1010 Writable area 1020 Computer Program 1110 Integrated Circuits 1120 Processing Unit 1122 memory 1124 dedicated integrated circuits 1126 Communication Elements 1130 Interconnect 1140 Processor System DETAILED DESCRIPTION OF THE INVENTION

[0017] Description of the embodiment While the subject matter disclosed herein may be embodied in many different forms, one or more specific embodiments have been shown in the drawings and are described in detail herein, with the understanding that the present disclosure is to be considered as an exemplification of the principles of the subject matter disclosed herein and is not intended to be limited to the specific embodiments shown and described.

[0018] In the following, elements of an operational embodiment are described for the sake of understanding, however, it will be apparent that each element is configured to perform the functions described as being performed by the respective element.

[0019] Furthermore, the subject matter disclosed in this specification is not limited to the embodiments only, but also includes all other combinations of features described herein or described in multiple mutually different dependent claims.

[0020] Figure 1a illustrates a schematic diagram of an example embodiment of a system 110 for training a multi-task neural network. Figure 1b illustrates a schematic diagram of an example embodiment of a system 120 for evaluating a multi-task neural network.

[0021] The training system 110 is configured to train a multi-tasking neural network that receives one input (x) and produces multiple outputs (y1, y2, . . . , y k ) is configured to generate

[0022] The training system 110 repeatedly evaluates the multi-task neural network on the training inputs to obtain multiple outputs of the multi-task neural network, i.e., one output for each task of the neural network. Therefore, for each task, a loss can be calculated that indicates how well the multi-task neural network performed for that task. Using these losses directly to train the network would not work well. Each loss may be on a different scale, which could lead to dramatically different loss scaling between tasks. Therefore, the training system 110 first calculates raw task weights based on multiple individual losses and then applies a normalization function to these raw task weights. The normalized task weights are then used to weight task-specific losses, and these weighted task-specific losses are then used to train the network.

[0023] Evaluation system 120 is configured to evaluate the trained multi-tasking neural network. For example, training system 110 can transfer the trained multi-tasking neural network to evaluation system 120. For example, system 120 can be an autonomous vehicle that uses the trained multi-tasking neural network to evaluate sensor data, i.e., system 120 can use one or more or all of the outputs to derive control signals for controlling the autonomous vehicle, e.g., actuators of the autonomous vehicle.

[0024] Much of the computer code for evaluating the neural network can be shared between training system 110 and evaluation system 120. If evaluation system 120 is not configured for further training of the neural network, then evaluation system 120 does not need code for computing task weights. Furthermore, while training system 110 typically computes all outputs of the multi-tasking neural network for each training input, this is not strictly necessary. While evaluation system 120 may compute all outputs of the multi-tasking neural network for each training input, it may also be configured to compute only the desired output of multiple possible outputs, thereby conserving computational resources.

[0025] System 110 and / or system 120 may be implemented in a single device, such as a single computer. System 110 and / or system 120 may also be implemented as a distributed system, such as a geographically distributed system, with parts of the distributed system communicating via a computer network.

[0026] Typically, systems 110 and 120 are different systems. However, it is possible to use the same system for both. For example, one system can be configured for the training phase to act as system 110 and the inference phase to act as system 120.

[0027] The training system 110 may include a processor system 111, storage 112, and a communication interface 113. The evaluation system 120 may include a processor system 121, storage 122, and a communication interface 123.

[0028] In different embodiments of communication interface 113 and / or 123, the communication interface may be selected from different options. For example, the interface may be a network interface to a local area network or a wide area network, e.g., to the Internet, a storage interface to internal or external data storage, an application interface (API), etc.

[0029] Storage 112 and 122 may be, for example, electronic storage, magnetic storage, etc. Storage may include local storage, such as a local hard drive or electronic memory. Storage 112 and 122 may include non-local storage, such as cloud storage. In the latter case, storage 112 and 122 may include a storage interface to the non-local storage. Storage may include multiple separate sub-storages that together make up storage 112 and 122.

[0030] Storage 112 and / or 122 may be non-transitory storage. For example, storage 112 and / or 122 may store data in the presence of power, such as a volatile memory device, e.g., random access memory (RAM). For example, storage 112 and / or 122 may store data in the presence or absence of power, such as a non-volatile memory device, e.g., flash memory. Storage may include a volatile, writable portion, e.g., RAM, and a non-volatile, writable portion, e.g., flash. Storage may include a non-volatile, non-writable portion, e.g., ROM.

[0031] Systems 110 and 120 can communicate internally with each other, other systems, external storage, input devices, output devices, and / or one or more sensors via a computer network. The computer network may be the Internet, an intranet, a LAN, a WLAN, a WAN, etc. The computer network may be the Internet. Systems 110 and 120 may include connection interfaces configured to communicate within system 100 or outside system 100 as needed. For example, the connection interfaces may include connectors, such as wired connectors, e.g., Ethernet connectors, optical connectors, etc., or wireless connectors, such as antennas, e.g., Wi-Fi antennas, 4G antennas, or 5G antennas.

[0032] Communications interface 113 can be used to send or receive digital data, such as to receive training data and / or to transmit a trained neural network, such as parameters of a trained neural network. Communications interface 123 can be used to send or receive digital data, such as to receive a trained network, receive inputs, and / or transmit outputs or data derived from the outputs.

[0033] Training system 110 and assessment system 120 may have a user interface that may include well-known elements such as one or more buttons, a keyboard, a display, a touch screen, etc. The user interface may be configured to accommodate user interaction, for example, to initiate training or assessment.

[0034] Execution of systems 110 and 120 may be implemented in a processor system. Systems 110 and 120 may include functional units for implementing aspects of each embodiment. The functional units may be part of the processor system. For example, the functional units shown herein may be implemented in whole or in part in computer instructions stored in the system's storage and executable by the processor system.

[0035] A processor system may include one or more processor circuits, such as a microprocessor, a CPU, a GPU, etc. Systems 110 and 120 may include multiple processors. A processor circuit may be implemented in a distributed manner, such as multiple sub-processor circuits. For example, systems 110 and 120 may use cloud computing.

[0036] Typically, the training system 110 and the evaluation system 120 each include one or more microprocessors that execute appropriate software stored on the system, which may, for example, be downloaded and / or stored in corresponding memory, e.g., volatile memory such as RAM or non-volatile memory such as flash.

[0037] Instead of implementing functionality using software, systems 110 and 120 may be implemented in whole or in part in programmable logic, for example as a field programmable gate array (FPGA). The systems may also be implemented in whole or in part as so-called application specific integrated circuits (ASICs), e.g., as integrated circuits (ICs) customized for their particular application. For example, the circuits may be implemented in CMOS using a hardware description language such as Verilog, VHDL, etc. In particular, training system 110 and evaluation system 120 may include, for example, circuitry for cryptographic and / or computational processing.

[0038] In a hybrid embodiment, the functional units are implemented partly in hardware, e.g., as coprocessors, e.g., as neural network coprocessors, and partly in software stored and executed on the system.

[0039] Figure 3a shows a schematic example of one embodiment of a multitasking neural network 300. Figure 3a shows one possible architecture for a multitasking neural network. The multitasking neural network is configured to receive one input 301, also referred to as x, and generate multiple outputs. Outputs 331, 332, and 333 are shown. The outputs are represented as y1, y2,...,y k The number of outputs, i.e., the number of tasks, may be more than two or three, for example, at least four, at least ten, etc.

[0040] Thus, multiple different outputs are calculated for a single input. Note that input 301 typically includes a mix of input data types. For example, a single input may include one or two or more of an image, a radar signal, a lidar signal, an ultrasound signal, etc. to generate multiple outputs.

[0041] In the architecture shown in FIG. 3a, the multi-tasking neural network includes a shared backbone 310 and multiple task-specific heads. Task-specific heads 321, 322, and 323 are shown. The shared backbone 310 receives input 301 and generates intermediate results that are provided to a respective one of the multiple task-specific heads. Each task-specific head computes one of the outputs of the multi-tasking neural network 300. These task-specific heads need not be the same size and may, for example, have different numbers of layers. For example, a head that generates object segmentation or depth maps may have more layers than a head that classifies images.

[0042] The architecture of Figure 3a is an example of hard parameter sharing, in which one or more network layers are shared across all tasks, in this case, the lower-level layers. The network has multiple task-specific heads, each of which may contain one or more layers, whose parameters are not shared. The shared layers exploit underlying similarities to capture and learn common features across tasks, while the task-specific layers allow the model to fine-tune these shared features for the specific task of those task-specific layers.

[0043] Instead of one backbone 310 shared among all tasks, more complex architectures are possible. For example, some layers can be shared among all tasks, while one or more other layers can be shared across a subset of tasks. Such architectures allow for a high degree of flexibility and adaptability, and facilitate greater parameter efficiency.

[0044] Instead of hard-sharing parameters, all or some of the shared parameters can be soft-shared. In one embodiment using this approach, the model includes one module, e.g., one neural network, for each individual task, with each module having its own set of parameters. Some of the modules have soft-shared parameters, e.g., parameters of a particular layer(s), e.g., one or more layers below the module. The modules are then regularized to encourage their shared parameters to be similar. This regularization allows tasks to learn from each other without forcing them to share exactly the same feature representations. This may be achieved by adding a regularization term to a loss function that penalizes the distance between the parameter matrices of modules of different tasks.

[0045] For example, the architecture shown in Figure 3a may be tuned by replicating the backbone 310 for each task, i.e., by obtaining three copies of the backbone 310, one copy for each head. The three backbones are trained individually but regularized to approximate each other. According to one embodiment, the losses used to train each backbone-head combination can still be weighted by the task weights.

[0046] The embodiments presented herein focus on backbone / task-specific head architectures with hard-shared parameters, however, embodiments may be adapted for other options, e.g., soft-shared parameters.

[0047] The neural network may include different neural network layers, such as one or more attention layers including a self-attention layer, encoding layers, decoding layers, convolutional layers, normalization layers, activation layers such as ReLu, etc.

[0048] Input / Output In one embodiment, the input of the multitasking neural network, e.g., input 301, includes dimensional data, i.e., data having one or more dimensions, obtained from a sensor. The dimensional data in the input may be one-dimensional, such as audio data, temperature data, pressure data, time series data, etc. The dimensional data in the input may be two-dimensional, such as image data including visible light image data and infrared image data, radar signals, lidar signals, ultrasound signals, etc. The dimensional data in the input may be three-dimensional, such as volumetric data, such as medical imaging data from an MRI or CT scan, 3D models, e.g., computer-aided design (CAD) models, and virtual reality environments. The dimensional data in the input may be four-dimensional, such as spatiotemporal data including three spatial dimensions plus one time dimension, such as video data capturing motion over time or 4D medical imaging providing a 3D spatial reconstruction over time. In one embodiment, input 301 may include or may also include scalar data, e.g., zero-dimensional data, such as sensor measurements.

[0049] The input data may include multidimensional image data, e.g., two-dimensional (2D), three-dimensional (3D), or four-dimensional (4D) images, acquired by different acquisition modalities such as, but not limited to, standard X-ray imaging, computed tomography (CT), magnetic resonance imaging (MRI), ultrasound (US), positron emission tomography (PET), single photon emission computed tomography (SPECT), and nuclear medicine (NM).

[0050] In one embodiment, the image data includes one or more of audio data, image data, video data, lidar data, radar data, infrared data, ultrasound data, sensor data, temperature data, pressure data, electrophysiological recordings such as electrocardiography (ecg) data, electroencephalography (eeg) data, and / or electromyography (emg) data.

[0051] The output data depends on the particular task. The output data may include the same range of data types as the input data. For example, the output may include dimensional data as described above. The output may include scalar data, such as a classification of the input. The output may be a binary value.

[0052] For example, in one embodiment, the input data includes image data and possibly additional data. The multi-tasking neural network may have one output that classifies objects in the image data, one output with an estimated depth for the image data, e.g., a depth map, and one output with object segmentation for the image data. It is possible to have only two of these outputs, e.g., only the object segmentation and the estimated depth. It is also possible to have more outputs.

[0053] In one embodiment, the input data represents a real-world environment for a machine agent, e.g., a robot or vehicle, operating within the environment. The machine agent can operate autonomously or semi-autonomously. A multi-tasking neural network may be trained to infer the state of the environment and / or the machine agent.

[0054] After training, the multi-tasking neural network can be used to operate a machine agent. For example, an input representing the vehicle's environment and / or state, obtained from one or more sensors, such as a camera, radar, vehicle sensors, etc., is provided, and multiple outputs are calculated from the input by evaluating the neural network against the input. A control function is derived from one or more of the multiple outputs of the multi-tasking neural network. The control function is to be executed by the machine agent interacting with the real-world environment. The control function may include vehicle behavior, such as acceleration, deceleration, emergency braking, turning, etc.

[0055] In one embodiment, a semi-autonomous or autonomous vehicle, e.g., an automobile, includes a multi-tasking neural network configured to receive as inputs various inputs for the purposes of environmental perception, decision-making, and control signal generation. The network receives dimensional data obtained from one sensor or a combination of two or more dimensional data obtained from multiple sensors. Inputs to the multi-tasking neural network include, but are not limited to, camera images, lidar data, and / or radar signals that provide visual information about the environment, such as road conditions, traffic signs, and obstacles. The inputs may also include IMU (inertial measurement unit) data for precisely positioning the vehicle. Outputs of the multi-tasking neural network may include, but are not limited to, object detection and object classification, e.g., to identify and classify obstacles, vehicles, pedestrians, etc.; lane detection, e.g., to identify lane boundaries for lane-keeping and lane-changing maneuvers; and distance estimation, e.g., to assess the proximity of objects to maintain a safe following distance. The network can generate vehicle dynamics predictions, such as the vehicle's potential future position and velocity.

[0056] In one embodiment, the input comprises medical data and the plurality of outputs represent medical information derived from the input medical data. For example, tasks of the multi-task neural network may include segmentation of anatomical structures such as tumors, organs, and blood vessels, tumor detection and characterization, e.g., distinguishing between benign and malignant types based on imaging features, functional analysis, e.g., assessing the function of a particular organ or organ system, e.g., measuring blood flow in cardiac imaging, or assessing metabolic activity in PET scans.

[0057] For example, in one embodiment, the input includes cardiac magnetic resonance images (cardiac MRI). Tasks of the multi-task neural network may include cardiac structural segmentation, e.g., segmenting the heart into its major components such as the left ventricle, right ventricle, myocardium (heart muscle), and atria; myocardial tissue characterization, e.g., distinguishing between healthy and diseased myocardial tissue, e.g., identifying fibrotic or infarcted regions; functional analysis, i.e., analyzing functional aspects of the heart such as ejection fraction, cardiac output, and myocardial strain, e.g., quantifying blood flow within the heart chambers; and detecting cardiac abnormalities such as congenital heart disease, diseases, e.g., coronary artery disease.

[0058] FIG. 3 b illustrates a schematic diagram of an example of one embodiment for training a multi-task neural network, for example, neural network 300 .

[0059] To train the network, a training set is obtained. The training set includes a plurality of training pairs, each consisting of a single training input and a corresponding plurality of training outputs. Figure 2 shows a schematic of one embodiment of a training set 200. Set 200 includes a plurality of training pairs, each consisting of a single training input and a corresponding plurality of training outputs. Figure 2 shows two training pairs, pair 210 and pair 220. Typically, a training data set will include many more than this, e.g., more than 1,000, or more than 10,000, or more than 100,000 training pairs.

[0060] 2 shows the inputs, namely input 211 and input 221. Note that a single input may include multiple different data types, e.g., multiple camera images in addition to radar and / or lidar data. For each input, an expected output is provided for each task of the multi-task neural network. For example, given input data 211, outputs 212, 213, and 214 are expected, and given input data 221, outputs 222, 223, and 224 are expected.

[0061] For example, using the architecture of FIG. 3a, when backbone 310 receives input 301, task-specific heads 321, 322 and 323 are expected to generate 212, 213 and 214, respectively.

[0062] Typically, each training pair contains one input and one output for each task. This is not required. Some training pairs may contain one training input but training outputs for fewer than all tasks. Although embodiments herein assume that all training pairs are complete training pairs, embodiments may be tailored by calculating losses and task weights only for outputs present in the training pair.

[0063] During training, iterations are performed on the training set. To train on one training pair of the training set, the multi-task neural network is evaluated on the training inputs in that training pair. For example, given input 301, outputs 331, 332, and 333 can be obtained. Typically, during training, not all outputs are correct. Therefore, a loss can be calculated for each output. Each individual loss of the multiple individual losses indicates the difference between the output of the multi-task neural network and a corresponding training output of the multiple training outputs. A variety of conventional loss functions can be used for this purpose.

[0064] For example, for the task of object detection in camera images, the loss function may be a cross-entropy loss to classify the type of object, e.g., vehicle, pedestrian, etc. For example, the task of distance estimation to an object, using the same camera images as input, may use a mean squared error (MSE) loss function to quantify the difference between the predicted distance and the true distance.

[0065] Thus, input 301 results in three outputs 331, 332, and 333 (in this example) and three corresponding losses, i.e., loss 341, loss 342, and loss 343, respectively. To balance the learning of the three tasks, task weights are calculated to weight the losses. For example, weighted loss unit 370 can use task weights 361-363 to weight individual losses, e.g., losses 341-343. Learning unit 380 then uses the weighted losses to train the neural network. Training can use a conventional training algorithm, such as backpropagation or one of its variations. For example, training can train each of the heads using the corresponding weighted loss. Training preferably includes training backbone 310, e.g., as part of training each of the task heads.

[0066] Task Weight Using the individual losses, raw task weights unit 350 can compute raw task weights from multiple individual losses. The inventors have discovered that applying a normalization function to the raw task weights improves training.

[0067] Typically, raw task weights are calculated from individual losses only, but this is not required, and additional factors can be considered. For example, additional factors that can be considered to calculate raw task weights can include one or more of the following: task importance, e.g., the relative importance of each task. For example, in a semi-autonomous vehicle scenario, the task of pedestrian detection can be assigned a higher importance than lane detection. An additional factor is data imbalance. For example, if one task has more training data than other tasks, training can be adjusted to give greater emphasis to tasks with less data. Calculating the task weights can use any conventional method of calculating raw task weights. The inventors have discovered that one particularly advantageous way to calculate task weights is to use the inverse of the loss. This has the effect of scaling the loss to a constant. For example, the individual losses (L1, L2, . . . , L) for the multiple obtained outputs of a multi-task neural network can be calculated as k ), the raw task weights (ω1,ω2, ,ω k ) for all i and a fixed positive scalar γ, say γ=1, i =γ / L i It can be calculated as follows:

[0068] To obtain task weights from raw task weights, we use a normalization function, e.g., (ω1,ω2,···,ω k )→Normalize(ω1,ω2,···,ω k ) is applied. In one embodiment, the normalization function is a probability distribution function, and the output of the normalization function is a number between 0 and 1, whose sum is 1. For example, the normalization function may be softmax, temperature-scaled softmax

[38] , soft-margin softmax

[39] , Taylor softmax

[40] , or any combination thereof.

[0069] Note that task weights are computed from the individual losses but are not taken into account for computing gradients, except for scaling. This may be implemented by applying a so-called gradient stopping operator to multiple task weights.

[0070] Batch Processing In one embodiment, iterations on the training set are batched. Batches of training pairs consisting of training inputs and training outputs can be collected from the training data. The same partially trained multi-task neural network can be evaluated on the training data in the batches. Parameters of the multi-task neural network can be adjusted based on the aggregated and calculated individual losses weighted by the task weights. The adjusted multi-task neural network can then be used for the next training batch. Different batch sizes can be used, such as at least 2, at least 16, at least 32, etc. A batch size of 1 corresponds to updating after each training pair.

[0071] For example, the following pseudocode can be used: [Table 1]

[0072] In the following, some further optional refinements, details and embodiments are presented: Some embodiments are explained in more detail using mathematical notation.

[0073] In MTL, we aim to solve K tasks for some input data points x∈X. To this end, x is defined as a set of k tasks that can be solved by a particular mapping {f k :X→Y k} to label {y k ∈Y k} k∈[1,K]We assume a hard task shared parameter θ in a hydra-like neural network architecture. That is, all tasks receive the same intermediate feature z = f(x;θ) from a shared backbone, and each task head has a task-specific parameter θ k output f k (x)=f' k (z;θ k )

[29] . However, other architectures for multitask neural networks may alternatively be used.

[0074] The network calculates the loss L for all tasks. k Typically, simply summing these losses leads to imbalance in learning as tasks with large losses may dominate training. Therefore, the goal is to obtain improved task weights ω for all tasks. k , possibly finding dynamic task weights so that all tasks benefit in terms of their final performance metric, loss L = Σ k ω k L k The goal is to optimize the

[0075] Some embodiments are directed to training a multi-task neural network, where task weights for weighting individual losses are calculated during training, and a normalization function is applied to normalize the task weights.

[0076] The first loss weighting method, UW-O (Optimal Uncertainty Weighting), computes weights based on an analytical solution instead of learning them using gradient descent, as in UW

[16] . The UW-O method does not apply a regularization function. The second loss weighting method, UW-SO (Soft Optimal Uncertainty Weighting), normalizes the analytically optimal weights using a regularization function. It assumes a softmax function, but other regularization functions may be used.

[0077] UW-O: Analytical solution In the approach of

[16] , the weight loss is based on task-specific aleatory uncertainty. The exact weighting formula depends on the type of loss. For example, for tasks with L1 loss, we assume a Laplace posterior distribution, σ with uncertainty for each task k∈K. k By identifying the inputs and treating these as learnable parameters, we can derive the L1 loss.

number

[0078] Instead of learning σ, we analytically derive the optimal solution and show its derivation. For example, in the case of L1 loss, we define the objective for a task with L1 loss as

number

[16] , where σ kMinimize the UW loss function with respect to σ k Solving for results in an analytically optimal solution: σ k =L k (2)

[0079] σ k The total loss function (Eq. 1) is expressed as its analytical solution L k Substituting in gives the following loss:

number

number

[0080] For example, UW-O, in particular, k In applications where the magnitudes of sg[L k ] and the individual losses L k It performs better than

[16] because it cancels out any large scale differences in . Furthermore, UW-O alleviates the initialization / inertia problems of approaches that learn task weights, such as in

[16] or

[21] .

[0081] UW-O does not include a normalization function. An example of a computer-implemented method (400) for training a multitasking neural network using UW-O is the following method:

[0082] A computer-implemented method (400) for training a multitasking neural network, wherein the multitasking neural network receives one input (x) and generates multiple outputs (y1, y2, . . . , y k), wherein the input of the multitasking neural network includes dimensional data obtained from the sensor, and the method is configured to: Obtaining a training set (410) that includes training pairs, each of which includes a training input and a corresponding training output; Iterating over the training set (420) and performing the iteration (420) includes, for one training pair of the training set, Evaluating (421) a multi-task neural network against the training inputs in the training pair, thereby obtaining multiple outputs of the multi-task neural network; Multiple individual losses (L1, L2, . . . , L) are used for multiple outputs of a multitask neural network. k ), where each individual loss of the plurality of individual losses indicates a difference between an output of the multi-task neural network and a corresponding training output of the plurality of training outputs (422); ·Task weights (ω1,ω2,···,ω k ) (423), where the task weights (ω1, ω2, ,ω k ) (423) includes calculating task weights from the plurality of individual losses, where the plurality of task weights are calculated by multiplying the corresponding plurality of individual losses by a constant (e.g., ω i L i = 1) (423), and Tuning the parameters of the multitask neural network based on the calculated individual losses weighted by the task weights (424); and A method (400) comprising:

[0083] UW-SO: UW-O transformation using normalization While UW-O implementations, e.g., as defined above, perform well in some cases, e.g., in scenarios with asymmetric losses, UW-O nevertheless does not reliably outperform

[16] or

[21] across all datasets and architectures.

[0084] To challenge not only UW but also high-performance scalarization approaches, we additionally propose UW-SO. Experimental results confirm the effectiveness of tuning task weights so that they sum to 1, for example, by applying a normalization function. For example, applying a softmax function to UW-O (Equation 3) gives UW-SO:

number

[0085] UW-SO achieves powerful results while overcoming the inherent dependence of computational cost on the number of tasks in scalarization.

[0086] In equation (4), T represents the softmax temperature, and the higher the value of T, the lower the Σ k∈K ω k = 1 results in a more even distribution of task weights. In contrast to scalarization, which requires tuning K task weights and is therefore infeasible for a large number of tasks, our novel method, UW-SO, requires tuning only one hyperparameter, T. In this section, we provide more details on the analytically optimal derivation of uncertainty-based task weights. Note that the analytical solution to UW, referred to as UW-O, is different for each different loss criterion (e.g., L2 and cross-entropy losses produce different constants in the denominator compared to L1). However, we simplify this by adopting a unified formula (see Equation 3) independent of task type, which reduces implementation overhead. Empirically, unification has been shown to further improve results.

[0087] Nevertheless, in the following paragraphs we provide a detailed derivation of UW-O for L1, L2 and cross-entropy losses.

[0088] L1 loss For regression tasks evaluated by L1 loss, we define the likelihood as a Laplace distribution, and the objective is therefore given as in

[16] .

number

number

[0089] σ k is assumed to be positive, and therefore only positive losses are allowed. σ k The analytical solution L k Substituting this, The following losses are obtained:

number

number

[0090] To show that UW-O is indeed insensitive to loss scaling, we use task-specific weights ω in the loss function. kHowever, as our experiments also show, these weights cancel out, which makes UW-O very effective for scenarios with highly unbalanced losses.

number

[0091] L2 loss For regression tasks with L2 loss, the UW objective is as follows:

number

number

[0092] Cross-entropy loss The derivation for classification tasks evaluated by cross-entropy loss is as follows:

number

[0093] Surprisingly, in each of these cases, the corresponding individual losses can be scaled by a constant (e.g., ω i L i = 1). Using these task weights directly works particularly well in scenarios with highly imbalanced losses. However, in each of these scenarios, we can add a normalization factor to normalize the task weights.

[0094] Test Results We compare our method with the most common loss weighting approach.

[0095] We use three common computer vision MTL datasets: two datasets for scene understanding—NYUv2

[27] and Cityscapes [8]—and the binary attribute dataset CelebA

[23] . Both NYUv2 and Cityscapes include the tasks of semantic segmentation and depth estimation. The third task in NYUv2 is surface normal estimation. CelebA constitutes a binary classification problem with 40 classes.

[0096] Architecture: For NYUv2, we use SegNet[1] and a pre-trained ResNet-50 / ResNet-101 on ImageNet with DeepLabHead. For Cityscapes, we use SegNet and a DeepLabV3+[5] network with a pre-trained ResNet-50 / ResNet-101 backbone. The single-task learning (STL) baseline is trained using only SegNet. For CelebA, we also use ResNet-18 for STL. All architectures are trained on the training data set and the best Δ m Further, a line search is used to find the epoch with Δ m Different learning rates and weight decays were tested to find the best combination for Δ m and task-specific metrics are reported. Indicator: A widely established Δ m We use the single-task baseline M b :

number

[0097] Comparing the three weighted losses for different MTO algorithms, we find that interestingly, scalarization and UW-SO result in comparable weighted losses, indicating that UW-SO finds similar weighting schemes at a fraction of the computational cost.

[0098] We observed comparable results for Cityscapes: UW-SO performed best when using one of the ResNet architectures, and only scalarization on SegNet outperformed UW-SO. In contrast to NYUv2, scalarization performed poorly when the network was relatively large, due to poor performance for the difficult and highly sensitive relative depth error. While a more fine-grained search of the task weights might yield better results, our weight search with a step size down to 0.02 performed sufficiently well to keep the computational cost within a feasible range.

[0099] Considering the more difficult setup with 40 tasks in CelebA, a Δ of -4.0 m With a score of 8.95 and a mean error of 8.95, UW-SO clearly surpasses the performance of all other methods (see table below). In contrast to the other two datasets, scalarization was ruled out due to the infeasibility of performing a grid search across 40 task weights. We attempted to run 50 different random weight combinations, but were unable to beat the performance of EW, so we omit reporting these results. Furthermore, GLS is not reported because the loss diverges due to numerical instability for many tasks. In contrast to the previous results on the other datasets, EW, RLW, and DWA show strong performance on CelebA compared to UW and IMTL-L. This observation is likely due to the tendency of UW and IMTL-L to overfit a subset of tasks, which is reflected in the validation delta. m The score is shown by the time it takes to reach its minimum value (e.g., epoch 6 for UW). In this section, we analyze the overfitting behavior of UW in comparison with UW-SO. m As shown by the scores, positive transfer is achieved by training multiple tasks simultaneously and using appropriate MTOs. To verify that the improvement in average performance of UW-SO is not solely due to a limited set of tasks, we examine task-level performance and noteworthy find that UW-SO outperforms UW / UW-O / IMTL-L / RLW / EW / DWA in 34 / 34 / 31 / 27 / 24 / 24 of the 40 tasks. [Table 2] Test data results on CelebA using ResNet-18. Average test error (5 runs) across all 40 tasks. The softmax temperature chosen for UW-SO is T=100. GLS and scalarization are ruled out due to infeasibility; see the main text for details.

[0100] FIG. 4a illustrates an embodiment for training a multitasking neural network. 1 shows a schematic diagram of an example of the evolution of weight ratios. The horizontal axis shows the number of epochs. The vertical axis shows the Bald weight ratio. Uncertainty weighting 411 (UW), soft-optimal uncertainty weighting 412 (UW-SO), and equal weighting 413 (EW) are shown.

[0101] Figure 4b shows a schematic diagram of an example of loss evolution for one embodiment of training a multitask neural network. The horizontal axis indicates the number of epochs. The vertical axis of Figure 4b indicates Bald loss. Shown are training uncertainty weights 421 (train UW), testing uncertainty weights 422 (test UW), training soft-optimal uncertainty weights 423 (train UW-SO), and testing soft-optimal uncertainty weights 424 (test UW-SO).

[0102] Figures 4a and 4b show a comparison of the weight ratios and loss trends for UW and UW-SO on the Bald task of CelebA. UW shows superior training performance driven by large task weights, but is unable to generalize to unseen data, as evidenced by the increased test loss. UW-SO reduces task weights, reducing overfitting.

[0103] 5 shows a schematic example of performance for different choices of T in one embodiment. The horizontal axis shows temperature T. The vertical axis shows Δ mThe figure shows the performance of UW-SO for different choices of T. The figure shows a clear and reasonably flat minimum for Cityscapes, which makes optimizing T easy. The UW-SO approach has strong performance, but requires tuning a single hyperparameter, T. In Figure 5, we show how performance changes as the temperature, T, is tuned for Cityscapes with SegNet. Best performance is achieved for T = 20, but values ​​close to this also perform well. A line search for T was performed with steps of 5, and further with steps of 2 near the optimum. We conclude that it is possible to find a good value for T with acceptable tuning effort.

[0104] Figure 6a schematically illustrates an example of task weight evolution for a segmentation task in one embodiment. Figure 6b schematically illustrates an example of task weight evolution for a depth task in one embodiment. Figure 6c schematically illustrates an example of task weight evolution for a normal task in one embodiment. In all three figures, the horizontal axis indicates epochs and the vertical axis indicates task weights. The figures show scalar 611, uniform weighting 612, uncertainty weighting 613, geometric loss strategy 614, unbiased multi-task learning 615, optimal uncertainty weighting 616, and soft-optimal uncertainty weighting 617.

[0105] Comparing task weights throughout the training process using different weighting methods provides a better understanding of the weighting principles of each MTO approach. Figure 6 exemplarily shows the weights for three tasks on the NYUv2 dataset using SegNet.

[0106] In addition to the similarities between scalarization and UW-SO, we can also observe that UW-O and IMTL-L exhibit fairly similar task weight transitions: IMTL-L transitions the weights fairly smoothly due to smaller gradient updates, while UW-O exhibits more fluctuations and jitter around IMTL-L.

[0107] 7 illustrates a schematic diagram of an example of an embodiment of a method 400 for training a multitasking neural network. A multitasking neural network receives one input (x) and produces multiple outputs (y1, y2, . . . , y k ), and the input of the multi-tasking neural network includes dimensional data obtained from the sensors. Obtaining a training set (410) that includes training pairs, each of which includes a training input and a corresponding training output; Iterating over the training set (420) and performing the iteration (420) includes, for one training pair of the training set, Evaluating (421) a multi-task neural network against the training inputs in the training pair, thereby obtaining multiple outputs of the multi-task neural network; Multiple individual losses (L1, L2, . . . , L) are used for multiple outputs of a multitask neural network. k ), where each individual loss of the plurality of individual losses indicates a difference between an output of the multi-task neural network and a corresponding training output of the plurality of training outputs (422); ·Task weights (ω1,ω2,···,ω k ) (423), where the task weights (ω1, ω2, ,ω k ) (423) includes calculating raw task weights from the plurality of individual losses and applying a normalization function to the raw task weights (423); Tuning the parameters of the multitask neural network based on the calculated individual losses weighted by the task weights (424); and Includes:

[0108] In one embodiment, the method may include a training phase and an inference phase. In the training phase, a multi-tasking neural network is trained, for example, as described herein. In the inference phase, the trained multi-tasking neural network is used. For example, the inference phase may include receiving an input (x) and applying the trained multi-tasking neural network to the input to obtain corresponding outputs of the multi-tasking neural network.

[0109] For example, the inference phase may be performed in or for a machine agent. The input data may represent a real-world environment for a machine agent, e.g., a robot or a vehicle, operating within the environment. A multi-tasking neural network may be trained to infer the state of the environment and / or the machine agent, and the inference phase may include: Deriving a control function for a machine agent from one or more of a plurality of outputs of the multitasking neural network, the control function to be executed by the machine agent interacting with a real-world environment. Includes:

[0110] For example, the machine learning method may be a computer-implemented method. For example, accessing training data and / or receiving input data may occur using a communications interface, e.g., an electronic interface, a network interface, a memory interface, etc. For example, storing or retrieving parameters, e.g., network parameters, from electronic storage, e.g., a memory, a hard drive, etc. For example, applying a neural network to the training data data and / or adjusting stored parameters to train the network may occur using an electronic computing device, e.g., a computer.

[0111] A neural network may have multiple layers, which may include, for example, convolutional layers, during training and / or application. For example, a neural network may have at least 2, 5, 10, 15, 20, or 40 or more hidden layers. The number of neurons in a neural network may be, for example, at least 10, 100, 1000, 10,000, 100,000, 1,000,000, or more.

[0112] Those skilled in the art will appreciate that many different ways of performing the method are possible. For example, the steps may be performed in the order presented, but the order of the steps may be changed, or some steps may be performed in parallel. Furthermore, other method steps may be inserted between steps. The inserted steps may represent improvements to the method as described herein or may be unrelated to the method. For example, some steps may be performed at least partially in parallel. Furthermore, a given step need not be completely completed before the next step begins.

[0113] Embodiments of the method may be implemented using software, which includes instructions for causing a processor system to perform a method according to one embodiment, e.g., method 400. The software may include only those steps performed by a particular subentity of the system. The software may be stored on a suitable storage medium, such as a hard disk, floppy, memory, optical disk, etc. The software may be transmitted wired or wirelessly, or as a signal using a data network, e.g., the Internet. The software may be available for download and / or for remote use on a server. Embodiments of the method may be implemented using a bitstream arranged to configure programmable logic, e.g., a field programmable gate array (FPGA), to perform the method.

[0114] It will be understood that the subject matter disclosed herein also extends to computer programs, particularly computer programs on or in a carrier, configured to carry out the subject matter disclosed herein. The programs may be in the form of source code, object code, intermediate source code, and object code, such as partially compiled form, or any other form suitable for use in implementing embodiments of the present methods. An embodiment relating to a computer program product includes computer-executable instructions corresponding to each processing step of at least one of the methods described above. These instructions may be subdivided into subroutines and / or stored in one or more files, which may be statically or dynamically linked. Another embodiment relating to a computer program product includes computer-executable instructions corresponding to each device, unit, and / or part of at least one of the systems and / or products described above.

[0115] FIG. 8a illustrates a computer-readable medium 1000 having a writable portion 1010 and a computer-readable medium 1001 also having a writable portion. The computer-readable medium 1000 is shown in the form of an optically readable medium. The computer-readable medium 1001 is shown in the form of an electronic memory, in this case a memory card. The computer-readable media 1000 and 1001 can store data 1020, which, when executed by a processor system, can represent instructions that cause the processor system to perform an embodiment of a method for training and / or evaluating a multitasking neural network according to one embodiment. The computer program 1020 can be embodied as physical marks on the computer-readable medium 1000 or by magnetization of the computer-readable medium 1000. However, any other suitable embodiment is also contemplated. Furthermore, although the computer readable medium 1000 is illustrated herein as an optical disk, it will be understood that the computer readable medium 1000 may be any suitable computer readable medium, such as a hard disk, solid state memory, flash memory, etc., and may be non-recordable or recordable. The computer program 1020 includes instructions for causing a processor system to perform the above-described methods for training and / or evaluating a multitasking neural network.

[0116] FIG. 8b shows a schematic diagram of a processor system 1140 according to one embodiment. The processor system may be configured for a method of training and / or evaluating a multitasking neural network. The processor system includes one or more integrated circuits 1110. The architecture of the one or more integrated circuits 1110 is shown schematically in FIG. 8b. The circuit 1110 includes a processing unit 1120, e.g., a CPU, for executing computer program components, performing the method according to one embodiment, and / or implementing modules or units of the computer program components. The circuit 1110 includes a memory 1122 for storing programming code, data, etc. A portion of the memory 1122 may be read-only. The circuit 1110 may include a communication element 1126, e.g., an antenna, a connector, or both. The circuit 1110 may include a dedicated integrated circuit 1124 for performing some or all of the processing defined in the method. The processor 1120, memory 1122, special purpose IC 1124, and communication element 1126 may be interconnected via an interconnect 1130, such as a bus. The processor system 1110 may be configured for contact and / or contactless communication using antennas and / or connectors, respectively.

[0117] For example, in one embodiment, the processor system 1140, e.g., a neural network training device and / or a neural network evaluator, may include a processor circuit and a memory circuit, where the processor is configured to execute software stored in the memory circuit. For example, the processor circuit may be an Intel Core i7 processor, an ARM Cortex-R8, etc. The memory circuit may be a ROM circuit or a non-volatile memory, e.g., flash memory. The memory circuit may be a volatile memory, e.g., SRAM memory. In the latter case, the device may include a non-volatile software interface, e.g., a hard drive, a network interface, etc., configured to provide the software.

[0118] The memory 1122 may be considered a "non-transitory machine-readable medium." As used herein, the term "non-transitory" will be understood to include all forms of storage that do not include transitory signals, but includes both volatile and non-volatile memory.

[0119] Although apparatus 1100 is shown as including one of each described component, different components may be replicated in different embodiments. For example, processor 1120 may include multiple microprocessors, configured to independently execute the methods described herein or to perform steps or subroutines of the methods described herein such that the multiple processors cooperate to achieve the functionality described herein. Furthermore, if apparatus 1100 is implemented in a cloud computing system, the different hardware components may reside in respective separate physical systems. For example, processor 1120 may include a first processor in a first server and a second processor in a second server.

[0120] References [1]Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. In IEEE Transactions on Pattern Analysis and Machine Intelligence, volume 39(12), pages 2481-2495. IEEE, 2017. [2]Lin Baijiong, Ye Feiyang, and Zhang Yu. A closer look at loss weighting in multi-task learning. ArXiv preprint, abs / 2111.10603, 2021. [3]Rich Caruana. Multitask learning. Machine learning, 28(1):41-75, 1997. [4]Sumanth Chennupati, Ganesh Sistu, Senthil Yogamani, and Samir A Rawashdeh. Multinet++: Multi-stream feature aggregation and geometric loss strategy for multi-task learning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 0-0, 2019. [5]Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pages 801-818, 2018. [6]Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmaessan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 793-802. PMLR, 2018. [7]Zhao Chen, Jiquan Ngiam, Yanping Huang, Thang Luong, Henrik Kretzschmar, Yuning Chai, and Dragomir Anguelov. Just pick a sign: Optimizing deep multitask models with gradient sign dropout. In Hugo Larochelle, Marc’AurelioRanzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. [8]Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 3213-3223. IEEE Computer Society, 2016. [9]Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pages 248-255. IEEE Computer Society, 2009.

[10] Long Duong, Trevor Cohn, Steven Bird, and Paul Cook. Low resource dependency parsing: Cross-lingual parameter sharing in a neural network parser. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 845-850, Beijing, China, 2015. Association for Computational Linguistics.

[11] Cathrin Elich, Lukas Kirchdorfer, Jan M. Koehler, and Lukas Schott. Challenging common assumptions in multi-task learning. ArXiv preprint, abs / 2311.04698, 2023.

[12] Michelle Guo, Albert Haque, De-An Huang, Serena Yeung, and Li Fei-Fei. Dynamic task prioritization for multitask learning. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.

[13] Falk Heuer, Sven Mantowsky, Syed Saqib Bukhari, and Georg Schneider. Multitask-centernet (MCN): efficient and diverse multitask learning using an anchor free approach. In IEEE / CVF International Conference on Computer Vision Workshops, ICCVW 2021, Montreal, BC, Canada, October 11-17, 2021, pages 997-1005. IEEE, 2021.

[14] Ronghang Hu and Amanpreet Singh. Unit: Multimodal multitask learning with a unified transformer. In 2021 IEEE / CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 1419-1429. IEEE, 2021.

[15] Adrian Javaloy and Isabel Valera. Rotograd: Gradient homogenization in multitask learning. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.

[16] Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 7482-7491. IEEE Computer Society, 2018.

[17] Vitaly Kurin, Alessandro De Palma, Ilya Kostrikov, Shimon Whiteson, and Pawan Kumar Mudigonda. In defense of the unitary scalarization for deep multi-task learning. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, December, virtual, 2022.

[18] Baijiong Lin, Feiyang Ye, Yu Zhang, and Ivor W. Tsang. Reasonable effectiveness of random weighting: A litmus test for multi-task learning. ArXiv preprint, abs / 1706.05587, 2017.

[19] Baijiong Lin and Yu Zhang. LibMTL: A python library for multi-task learning. Journal of Machine Learning Research, 2023.

[20] Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. In Marc’AurelioRanzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 18878-18890, 2021.

[21] Liyang Liu, Yi Li, Zhanghui Kuang, Jing-Hao Xue, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Towards impartial multi-task learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.

[22] Shikun Liu, Edward Johns, and Andrew J. Davison. End-to-end multi-task learning with attention. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 1871-1880. Computer Vision Foundation / IEEE, 2019.

[23] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 3730-3738. IEEE Computer Society, 2015.

[24] Kevis-Kokitsi Maninis, Ilija Radosavovic, and Iasonas Kokkinos. Attentive single-tasking of multiple tasks. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 1851-1860. Computer Vision Foundation / IEEE, 2019.

[25] Yuren Mao, Zekai Wang, Weiwei Liu, Xuemin Lin, and Pengtao Xie. MetaWeighting: Learning to weight tasks in multi-task learning. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3436-3448, Dublin, Ireland, 2022. Association for Computational Linguistics.

[26] Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 3994-4003. IEEE Computer Society, 2016.

[27] Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from rgbd images. In Proceedings of the European conference on computer vision (ECCV), 2012.

[28] Aviv Navon, Aviv Shamsian, Idan Achituve, Haggai Maron, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya. Multi-task learning as a bargaining game. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 16428-16446. PMLR, 2022.

[29] Sebastian Ruder. An overview of multi-task learning in deep neural networks. ArXivpreprint, abs / 1706.05098, 2017.

[30] Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, NicoloCesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montreal, Canada, pages 525-536, 2018.

[31] Dmitry Senushkin, Nikolay Patakin, Arseny Kuznetsov, and Anton Konushin. Independent component alignment for multi-task learning. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20083-20093, 2023.

[32] Guangyuan Shi, Qimai Li, Wenlong Zhang, Jiaxin Chen, and Xiao-Ming Wu. Recon: Reducing Conflicting Gradients From the Root For Multi-Task Learning. In The Eleventh International Conference on Learning Representations, 2023.

[33] Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Marc Proesmans, Dengxin Dai, and Luc Van Gool. Multi-task learning for dense prediction tasks: A survey. In IEEE transactions on pattern analysis and machine intelligence. IEEE, 2021.

[34] Derrick Xin, Behrooz Ghorbani, Ankush Garg, Orhan Firat, and Justin Gilmer. Do current multi-task optimization methods in deep learning even help? In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, December, virtual, 2022.

[35] Dan Xu, WanliOuyang, Xiaogang Wang, and Nicu Sebe. Pad-net: Multi-tasks guided prediction-and-distillation network for simultaneous depth estimation and scene parsing. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 675-684. IEEE Computer Society, 2018.

[36] Yongxin Yang and Timothy M Hospedales. Trace norm regularised deep multi-task learning. ArXiv preprint, abs / 1606.04038, 2016.

[37] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In Hugo Larochelle, Marc’AurelioRanzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.

[38] HINTON, Geoffrey; VINYALS, Oriol; DEAN, Jeff. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.

[39] LIANG, Xuezhi, et al. Soft-margin softmax for deep classification. In: International Conference on Neural Information Processing. Cham: Springer International Publishing, 2017. S. 413-421.

[40] DE BREBISSON, Alexandre; VINCENT, Pascal. An exploration of softmax alternatives belonging to the spherical loss family. arXiv preprint arXiv:1511.05042, 2015.

Claims

1. A computer-implemented method (400) for training a multitasking neural network, comprising: The multitasking neural network receives one input (x) and produces multiple outputs (y １ , y ２ , ..., y ｋ ), the inputs of the multi-tasking neural network include dimensional data obtained from sensors; The method comprises: Obtaining 410 a training set including training pairs of a training input and corresponding training outputs; - iterating on the training set (420); Including, The iterating (420) may involve, for one training pair of the training set: Evaluating (421) the multi-task neural network against the training inputs in the training pairs, thereby obtaining a plurality of outputs of the multi-task neural network; A plurality of individual losses (L １ , L ２ , ..., L ｋ ), wherein each individual loss of the plurality of individual losses indicates a difference between an output of the multi-tasking neural network and a corresponding training output of the plurality of training outputs; ・Task weight (ω １ , ω ２ , ..., ω ｋ Calculating (423) the task weights (ω １ , ω ２ , ..., ω ｋ ) (423) includes calculating raw task weights from the plurality of individual losses and applying a normalization function to the raw task weights (423); Adjusting parameters of the multi-task neural network based on the calculated individual losses weighted by the task weights (424); A method comprising:

2. The normalization function is a probability distribution function. The training method of claim 1.

3. The normalization function includes softmax, temperature scaled softmax, soft margin softmax, Taylor softmax, or any combination thereof. The training method according to claim 1 or 2.

4. The plurality of task weights (e.g., ω ｉ ) by dividing the corresponding individual losses by a constant (e.g., ω ｉ L ｉ = 1), The training method according to any one of claims 1 to 3.

5. a gradient stopping operator is applied to the plurality of task weights; A training method according to any one of claims 1 to 4.

6. The iterations on the training set are batch processed; The multiple individual losses are calculated for each batch of training pairs and averaged over the obtained batched individual losses; The raw task weights are calculated from the batched individual losses. A training method according to any one of claims 1 to 5.

7. the input data includes at least image data; At least one of the plurality of outputs classifies objects in the image data; and / or At least one of the outputs includes an estimated depth for the image data; and / or At least one of the plurality of outputs includes an object segmentation for the image data; A training method according to any one of claims 1 to 6.

8. the input data represents a real-world environment for a mechanical agent, e.g., a robot or a vehicle, operating within the environment; the multitasking neural network is trained to infer the state of the environment and / or the machine agent; A training method according to any one of claims 1 to 7.

9. the image data includes one or more of audio data, image data, video data, lidar data, radar data, infrared data, ultrasound data, sensor data, temperature data, pressure data, electrophysiological recordings; A training method according to any one of claims 1 to 8.

10. the multi-task neural network includes a shared backbone and multiple task-specific heads; the shared backbone is configured to receive the input (x); the plurality of task-specific heads receive the output of the shared backbone; each one of the plurality of outputs is generated by a corresponding one of the plurality of task-specific heads; 10. The training method according to any one of claims 1 to 9.

11. 1. A method for a multitasking neural network, the method comprising a training phase and an inference phase; The training method according to any one of claims 1 to 10, wherein the training phase obtains a trained multitask neural network; The inference phase includes receiving an input (x) and applying the trained multi-tasking neural network to the input, thereby obtaining corresponding outputs of the multi-tasking neural network; method.

12. the input data represents a real-world environment for a mechanical agent, e.g., a robot or a vehicle, operating within the environment; the multi-tasking neural network is trained to infer the state of the environment and / or the machine agent; The inference phase includes: deriving a control function for the machine agent from one or more of the plurality of outputs of the multi-tasking neural network, the control function to be executed by the machine agent interacting with the real-world environment; The method of claim 11.

13. 1. A system including one or more processors and one or more storage devices, The one or more storage devices store instructions that, when executed by the one or more processors, cause the one or more processors to perform the operations for the method of any one of claims 1 to 12. system.

14. A transitory or non-transitory computer-readable medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform the method of any one of claims 1 to 12.

Images