Gradient Normalization System and Method for Adaptive Loss Balancing in Deep Multitask Networks

Abstract

Systems and methods for training a multi-task network are disclosed. In one aspect, training the multi-task network includes determining a gradient norm of a single-task loss adjusted by a task weight for each task relative to a network weight of the multi-task network, and a relative training rate for the task based on the single-task loss for the task. Subsequently, a gradient loss function can be determined comprising the difference between (1) the gradient norm determined for each task and (2) the corresponding target gradient norm. The gradient of the gradient loss function relative to the task weight for the task can be used to determine an updated task weight for the task and used in the next iteration of training the multi-task network.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. patent application No. 62 / 577,705, filed on October 26, 2017, U.S. patent application No. 62 / 599,693, filed on December 16, 2017, U.S. patent application No. 62 / 628,266, filed on February 8, 2018, and U.S. patent application No. 62 / 695,356, filed on July 9, 2018, each of which is entitled “Gradient Normalization System and Method for Adaptive Loss Balancing in Deep Multi-Task Networks,” and the entire contents of each of which are incorporated herein by reference in their entirety.

[0003] Copyright Notice

[0004] A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever. Technical Field

[0005] The present disclosure relates generally to systems and methods for machine learning, and more particularly to training machine learning models. Background Art

[0006] Deep neural networks (DNNs) are a type of computational machine learning method. DNNs belong to a class of artificial neural networks (NNs). Using NNs, computational graphs that simulate the characteristics of biological neural networks can be constructed. Biological neural networks include features that are important for computation and are responsible for many functions of biological systems that would otherwise be difficult to capture using other methods. In some embodiments, such networks are arranged into a sequential hierarchical structure in which connections are unidirectional. For example, the output of an artificial neuron in a particular layer can be connected to the input of an artificial neuron in a subsequent layer. A DNN can be a NN with a large number of layers (e.g., 10, 100, or more layers).

[0007] Different NNs differ from each other in various ways. For example, the topology or architecture (e.g., the number of layers and how they are interconnected) and the weights of different NNs can differ. Weights can be roughly analogous to the synaptic strength of neural connections in biological systems. Weights influence the strength of effects propagated from one layer to another. The output of an artificial neuron can be a nonlinear function of the weighted sum of its inputs. The weights of a NN can be the weights that appear in these sums. Summary of the Invention

[0008] In one example, a system for training a multi-task network may include: a non-transitory memory configured to store: executable instructions, and a multi-task network for determining outputs associated with a plurality of tasks; and a hardware processor in communication with the non-transitory memory, the hardware processor being programmed by the executable instructions to: receive a training image associated with a plurality of reference task outputs for the plurality of tasks; for each of the plurality of tasks, determine, relative to a plurality of network weights of the multi-task network, a task output of the task determined using the multi-task network with the training image as input adjusted by the task weights of the task and (2) the task output associated with the training image. a gradient norm of a single-task loss output of a corresponding reference task; and determining a relative training rate for the task based on the single-task loss of the task; determining a gradient loss function comprising (1) a difference between the gradient norm determined for each task and (2) a corresponding target gradient norm, wherein the corresponding target gradient norm is determined based on: (a) an average gradient norm of the plurality of tasks, (b) the relative training rate for the task, and (c) a hyperparameter; determining a gradient of the gradient loss function with respect to a task weight for each of the plurality of tasks; and determining an updated task weight for each of the plurality of tasks using the gradient of the gradient loss function with respect to the task weight.

[0009] In another example, a method for training a multi-task network may include: receiving a training image from a plurality of training images, each training image being associated with a plurality of reference task outputs for a plurality of tasks; determining, for each of the plurality of tasks, a gradient norm of a single-task loss adjusted by a task weight of the task relative to a plurality of network weights of the multi-task network, the single-task loss being a single-task loss of (1) a task output of the task determined using the multi-task network having the training image as input and (2) a corresponding reference task output of the task associated with the training image; and determining a relative training rate for the task based on the single-task loss of the task; determining a gradient loss function comprising a difference between (1) the determined gradient norm of each task and (2) a corresponding target gradient norm, the corresponding target gradient norm being determined based on: (a) an average gradient norm of the plurality of tasks, and (b) the relative training rate of the task; and determining an updated task weight for each of the plurality of tasks using a gradient of the gradient loss function relative to the task weight.

[0010] In yet another example, a head-mounted display system may include: a non-transitory memory configured to store: executable instructions, and a multi-task network for determining outputs associated with a plurality of tasks, wherein the multi-task network is trained using: (1) a gradient norm of a single-task loss of a task of the plurality of tasks determined using the multi-task network having a training image as input and (2) a corresponding reference task output of the task associated with the training image, relative to a plurality of network weights of the multi-task network adjusted by the task weights of the tasks, a relative training rate of the tasks determined based on the single-task loss of the tasks, comprising: (1) the determined gradient norm of the task output of the task of the plurality of tasks and (2) a gradient norm of a single-task loss of a corresponding reference task output of the task associated with the training image; a gradient loss function for determining the difference between target gradient norms of the tasks, the corresponding target gradient norm being determined based on: (a) an average gradient norm of the multiple tasks, (b) the relative training rates of the tasks, and (c) a hyperparameter, updating task weights of the tasks using the gradient of the gradient loss function with respect to the task weights of the tasks; a display; a sensor; and a hardware processor in communication with the non-transitory memory and the display, the hardware processor being programmed by the executable instructions to: receive sensor input captured by the sensor; determine a task output for each of the multiple tasks using the multi-task network; and cause the display to display information related to the determined task output to a user of the augmented reality device.

[0011] Details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. Neither this summary nor the following detailed description is intended to define or restrict the scope of the subject matter of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1A This is an example diagram of the imbalanced gradient norm across tasks when training a multi-task network.

[0013] Figure 1B Figure 1 is an example diagram of balancing gradient norms across tasks when training a multi-task network.

[0014] Figures 2A-2F Example results are shown for training a multi-task network using training tasks with similar loss functions and different loss scales. Figures 2A-2C Shown are the results of gradient normalization (GradNorm) on a 2-task system. Figures 2D-2F The results of gradient normalization on a 10-task system are shown. The network structure diagram with loss scale is shown in Figure 2A and 2D As shown in the figure, the trajectory w during trainingi (t) Figure 2B and 2E As shown in , the task-normalized test loss curve is Figure 2C and 2F A hyperparameter value of α = 0.12 was used for all example runs.

[0015] Figures 3A-3C Figure 1 is a plot of example test and training loss curves for GradNorm (hyperparameter α=1.5), an equal-weight baseline, and uncertainty weights on a large dataset. NYUv2+ keypoints (kpts) and a VGG16 backbone were used.

[0016] Figure 4 is a graph showing example grid search performance for random task weights and GradNorm.

[0017] Figures 5A-5B is an example graph showing that higher values of α tend to increase the weight w i (t) Pushing further out, which more aggressively reduces the impact of tasks that overfit or learn too quickly.

[0018] Figure 6 is an example graph showing the performance gain for various settings of the hyperparameter α.

[0019] Figures 7A-7D is to show how the value of a hyperparameter can be constant during training (e.g., Figure 7A ) or how it may change during training (e.g. Figures 7B-7D ) example diagram.

[0020] Figure 8 is a flowchart of an example process of training a multi-task network using GradNorm.

[0021] Figure 9 An example of a wearable display system that can implement embodiments of a multi-tasking network is schematically illustrated.

[0022] Reference numerals may be repeated throughout the drawings to indicate corresponding relationships between reference elements. The drawings are provided to illustrate the example embodiments described herein and are not intended to limit the scope of the present disclosure. DETAILED DESCRIPTION

[0023] Overview

[0024] A model that represents relationships and patterns in data, such as a function, algorithm, system, etc., that can accept input and produce output corresponding to the input in some way. For example, the model can be implemented as a machine learning method, such as a convolutional neural network (CNN) or a deep neural network (DNN). Deep learning is part of a broader family of machine learning methods that are based on the idea of learning data representations rather than task-specific methods, and has shown promise in solving audio-visual computing problems that are useful for augmented reality, mixed reality, virtual reality, and machine intelligence. In machine learning, convolutional neural networks (CNN or ConvNet) can include a class of deep feedforward artificial neural networks, and CNNs have been successfully applied to analyze visual images. Machine learning methods include a range of methods that provide robust and accurate solutions to a variety of problems, including eye image segmentation or eye tracking.

[0025] Disclosed herein are examples of systems and methods for training multi-task networks. Deep multi-task networks (where one neural network produces multiple prediction outputs) can provide better speed and performance than their single-task counterparts, but require appropriate training. For example, a deep multi-task network can be trained or taught to solve multiple learning tasks simultaneously while exploiting commonalities and differences between the tasks. Using shared representations between tasks, multiple tasks can be learned in parallel. For example, a multi-task neural network can share hidden layers across all tasks while providing their own task-specific output layers (this is sometimes called hard parameter sharing). As another approach, each task can have its own neural network with its own parameters. Regularization constraints can be used between the layers of the task-specific network to encourage parameter similarity (this is sometimes called soft parameter sharing).

[0026] This paper discloses an example of a gradient normalization (GradNorm) method that automatically balances training in deep multi-task models by dynamically adjusting the gradient magnitude. Compared to single-task networks, static baselines, and other adaptive multi-task loss balancing techniques, GradNorm improves accuracy and / or reduces overfitting across multiple tasks for a variety of network architectures, for regression and classification tasks, and on synthetic and real datasets. Although some implementations utilize only a single asymmetric hyperparameter α, GradNorm can match or exceed the performance of exhaustive grid search methods. Therefore, for some embodiments of GradNorm, several training runs may be required regardless of the number of tasks. Gradient manipulation can well control the training dynamics of multi-task networks and can enable a wide range of applications of multi-task learning.

[0027] Single-task learning in computer vision has been successful in deep learning, and many single-task models currently achieve or exceed human accuracy in performing a wide variety of tasks. However, the ultimate vision system for full scene understanding should be able to perform many different perception tasks simultaneously and efficiently, especially in devices such as smartphones, wearable devices (e.g., ref. Figure 9 The invention relates to a wearable display system 900 as described herein, and to an embedded system such as a robot or drone within a limited computing environment. Such a system can be implemented through multi-task learning, in which a model shares weights across multiple tasks and performs multiple inferences in a single forward pass. Not only are such networks scalable, but the shared features within these networks can lead to more robust regularization and thus improve performance. Multi-task networks trained using the methods disclosed herein can have higher efficiency and higher performance.

[0028] Multi-task networks can be difficult to train: the different tasks need to be properly balanced so that the network parameters converge to robust shared features that are useful for all tasks. In some approaches, methods in multi-task learning can find this balance by manipulating the forward pass of the network (e.g., by building explicit statistical relationships between features or optimizing the multi-task network architecture). However, task imbalances can hinder proper training because they manifest as imbalances between back-propagated gradients. A task that is too dominant during training can express this dominance, for example, by inducing gradients with relatively large magnitudes. The training methods disclosed in this article fundamentally alleviate this problem by adjusting the multi-task loss function to directly modify the gradient magnitudes.

[0029] In some embodiments, the multi-task loss function is a single-task loss L i The weighted linear combination L = ∑ i w i L i , where the sum traverses all T tasks. This paper discloses an adaptive method to i =w i (t)) Change w i This linear form of the loss function can easily achieve gradient balance, because w i is directly and linearly coupled to the back-propagated gradient magnitude of each task. The gradient normalization method disclosed in this paper can be used to normalize the gradient of each w in each training step t. i Finding good values (e.g., optimal values) to balance the contribution of each task to improve (e.g., optimal) model training. To improve (e.g., optimize) the weights w used for gradient balancing i(t), the method disclosed in this paper can penalize the network when the back-propagated gradient from any task is too large or too small. The right balance can be achieved when tasks are trained at similar rates. For example, if task i is trained relatively quickly, then its weight w i (t) should be relative to the weight w of other tasks j (t)| j≠i Lowered so that other tasks have a greater impact on training. In some embodiments, batch normalization can be implemented during training. Gradient normalization methods can normalize across tasks and use rate balance as a desirable goal to inform normalization. This gradient normalization (referred to herein as GradNorm) can improve network performance while significantly reducing overfitting.

[0030] In some embodiments, the GradNorm method can be effective for multi-task loss balancing by directly adjusting the gradient magnitude. This method can match or exceed the performance of very expensive exhaustive grid search procedures, but in some embodiments it only involves adjusting a single hyperparameter, or in some embodiments only involves adjusting two or more hyperparameters. GradNorm can enable direct gradient interaction, which can be an effective way to control multi-task learning.

[0031] In some embodiments, the gradient normalization methods disclosed herein may have applications in computer vision, natural language processing, speech synthesis, domain-specific applications (such as traffic prediction), general cross-domain applications, and curriculum learning. In some embodiments, tasks are trained jointly based on a global reward such as a reduction in total loss. Embodiments of GradNorm may be applied to training for augmented reality, mixed reality, or virtual reality (see, for example, reference Figure 9 A multi-task neural network for the augmented reality system described.

[0032] Multi-task learning is a good fit for the field of computer vision, where making multiple robust predictions is crucial for complete scene understanding. Deep networks have been used to solve various subsets of various vision tasks, from 3-task networks to larger subsets in UberNet. A single computer vision problem can be framed as a multi-task problem, such as segmentation in Mask R-CNN or object detection in YOLO-9000. Clustering methods have shown success beyond deep models, while structures such as deep relation networks and crisscross networks enable deep networks to search for meaningful relationships between tasks and learn which features are shared between tasks. Grouping between labels can be used to search for possible architectures to learn. Joint likelihood equations can be used to derive task weights based on the inherent uncertainty in each task.

[0033] Example GradNorm method

[0034] For the multi-task loss function L(t) = ∑w i (t)L i (t), these functions w can be learned by the following objectives i (t): (1) Place the gradient norms of different tasks on a common scale from which their relative magnitudes can be inferred, and (2) Dynamically adjust the gradient norms so that different tasks train at similar rates. The relevant quantities are described below, starting with the gradients to be manipulated.

[0035] W: The subset of the full network weights on which GradNorm is applied W can be the last shared layer of weights to save computational cost. In some embodiments, this choice of W can cause GradNorm to increase training time (e.g., only by ~5%).

[0036] The weighted single-task loss w relative to the chosen weight W i (t)L i The L2 norm of the gradient of (t).

[0037] The mean (or expected) gradient norm across all tasks at training time t.

[0038] The various training rates for each task i are described below.

[0039] The loss ratio of task i at time t. is a measure of the inverse training rate of task i (e.g., the lower In some embodiments, the network may have a stable initialization and may be trained directly using L i (0). When L i (0) When initialization is heavily dependent, a theoretical initial loss can be used instead. For example, for L i , CE loss between C classes, L i (0) = log(c).

[0040] Relative inverse training rate for task i.

[0041] Example of balanced gradients using GradNorm

[0042] GradNorm can establish a common scale for gradient magnitudes and / or can balance the training rates of different tasks. In one example, the common scale for gradients can be the mean gradient norm It establishes a baseline at each time step t, from which the relative gradient magnitude can be determined. The relative inverse training rate r for task i i (t) can be used to rate balance the gradient. Specifically, r i The larger the value of (t), the higher the gradient magnitude of task i, to encourage the task to train faster. Therefore, the gradient norm of each task i can be:

[0043]

[0044] where α is a hyperparameter. The hyperparameter α sets the strength of the restoring force that pulls the tasks back to a common training rate. If the tasks vary greatly in complexity, resulting in substantially different learning dynamics between tasks, higher values of α can be used to enforce a stronger training rate balance. When the tasks are more symmetric, lower values of α may be appropriate. Note that α = 0 attempts to pin the norm of the backpropagated gradients for each task to be equal at W. The hyperparameter α can be constant during training (see for example Figure 7A ), or can vary during training (e.g., α can be a function of training time t). For example, α can start out as a positive value (e.g., approximately 1.5, which can be a reference Figure 6 The effective value described above), and then decreases as training progresses (gradually (see for example Figure 7B ) or step by step (see for example Figure 7C )). This approach can advantageously provide a stronger training rate balance at the beginning of training (e.g., to train each task faster) while relaxing the training rate balance later in training. In some cases, α can be negative (e.g., for at least a portion of training), e.g., the variable α can start with a positive value and change to a relatively small negative value (e.g., about -0.1) during training toward the end of training (see for examples). Figure 7D ). Therefore, in various embodiments, the hyperparameter α can be in the range of -1 to 5, -1 to 3, 0.5 to 3, or other suitable ranges.

[0045] Equation 1 gives the objective of the gradient norm for each task i and can update the loss weight w i (t) to move the gradient norm towards the target for each task. In some embodiments, GradNorm can then be implemented as an L1 loss function L between the actual and target gradient norms for each task at each time step grad , sum up all tasks:

[0046]

[0047] Where, the sum is across all T tasks. Differentiate this loss L grad When , the target gradient norm can be Treated as a fixed constant to prevent loss of weight w i (t) drifts spuriously towards zero. Then we can calculate the i Distinguish L grad , because w i (t) Directly control the magnitude of the gradient for each task. The calculated gradient can then be applied via the update rule To update each w i (For example, Figure 1B shown).

[0048] In the following examples, the task is a computer vision task that involves identifying depth, surface normals, and key points in an image of a room. The following examples are illustrative only and not intended to be limiting. Figure 1A As shown, unbalanced gradient norms across tasks may lead to suboptimal training within the multi-task network 100a. GradNorm can calculate the gradient loss L grad (See, for example, Figure 1A ), the gradient loss adjusts the loss weight w when training the multi-task network 100b i To address this imbalance in the gradient norm. This balance can result in a balanced gradient norm. In some embodiments, relatively higher or lower gradient magnitudes can be used for certain tasks to perform better (e.g., optimal) training.

[0049] Table 1 summarizes the implementation of the GradNorm method. After each update step, the weights w i (t) is renormalized so that ∑ i w i (t) = T in order to decouple gradient normalization from the global learning rate.

[0050] Table 1. Training with GradNorm

[0051]

[0052]

[0053] Example pseudocode

[0054] Table 2 shows example pseudo-code programming language that may be used to perform embodiments of the gradient normalization techniques described herein.

[0055] Table 2. Example PyTorch implementation of an embodiment of GradNorm

[0056]

[0057]

[0058]

[0059] Example training for training tasks with similar loss functions and different loss scales

[0060] To illustrate GradNorm, we construct a common scenario for multi-task networks: training tasks with similar loss functions but different loss scales. In this case, for all loss weights w i (t), w i (t) = 1, network training may be dominated by tasks with larger loss scales, which will backpropagate larger gradients. GradNorm can solve this problem.

[0061] Consider the T regression task where the function is trained using the standard squared loss

[0062] f i (x)=σ i tanh((B+∈ i )x), Equation (3)

[0063] Among them, tanh(.) works element by element. The input is of dimension 250, the output is of dimension 100, and B and ∈ i is a constant matrix whose elements are normally distributed according to and Thus, each task shares information in B, but also contains task-specific information ∈ i σ i can be key parameters for training: they are what set the output f i A fixed scalar of scale. i Higher scales of σ result in higher expected squared loss for that task. These tasks are harder to learn because their response values vary widely, but they also backpropagate larger gradients. i This situation can lead to suboptimal training dynamics when a task dominates training across all tasks.

[0064] To train the model, a 4-layer fully connected ReLU activated network with 100 neurons per layer was used as a common backbone. The final affine transformation layer produced T final predictions (corresponding to T different tasks). To ensure valid analysis, models initialized with the same random values were used and fed with data generated from the same fixed random seed. Since the output function f i All have the same functional form, so setting the asymmetry α as low as 0.12 would thus expect minimal asymmetry between tasks.

[0065] In this example, the test time performance is judged using the task-normalized test time loss, which is the sum of the test loss ratios for each task, ∑ i L i (t) / L i (0). When there are different loss scales, a simple sum of the losses may be an insufficient performance measure for a multi-task network: tasks with higher loss scales may contribute significantly to the loss. There may not be a universal single scalar that can provide a meaningful measure of multi-task performance in all cases, but in this example, in addition to the loss scale σ i Furthermore, these tasks are statistically similar. Therefore, there is a clear measure of overall network performance, which is given by the variance of each task The normalized sum of the losses is equivalent to the sum (up to the scaling factor) of the loss ratios.

[0066] For T=2, the values (σ0, σ1)=(1.0, 100.0) were chosen. Figures 2A-2C The training results are shown in Figure 2. If equal weights w are used in training i = 1, task 1 will inhibit the learning of task 0 due to the higher loss scale of task 1. However, gradient normalization increases w0(t) to offset the larger gradient from T1, and the improved task balance leads to better test-time performance.

[0067] As the number of tasks increases, the possible benefits of gradient normalization become more apparent. For T = 10, sampling σ1 from a wide normal distribution yields Figures 2D-2F Compared to simply weighting each task equally, GradNorm significantly improves the test time performance. Similar to the T=2 case, for T=10, for smaller σ1 tasks, w i (t)Growth is greater.

[0068] For both T=2 and T=10, GradNorm is more stable and outperforms uncertainty weighting. As the loss of each task decreases, executing w i (t)~1 / L i The uncertainty weighting of (t) will result in a weight w i (t) is too large and grows too fast. Although such a network trains quickly at the beginning, the training will soon deteriorate. This problem is mainly due to the fact that the uncertainty weighting allows w to be i (t) changes (with always ensuring ∑w i (t) compared to GradNorm), which drives the global learning rate to increase rapidly as the network is trained.

[0069] During a single GradNorm run, it is observed that for each w iThe trajectory of (t) is stable and convergent. As shown below, the time-averaged weight ∑ t [w i (t)] are close to the optimal static weights, which shows that GradNorm can greatly simplify the tedious grid search process.

[0070] Example training with a larger dataset

[0071] Two variants of NYUv2 are used as the main datasets. The standard NYUv2 dataset carries depth, surface normals, and semantic segmentation labels (clustered into 13 different classes) for various indoor scenes in different room types (bathroom, living room, study, etc.). NYUv2 is relatively small (795 training, 654 test images), but contains both regression and classification labels, making it a good choice for testing the robustness of GradNorm on a variety of tasks.

[0072] The standard NYUv2 depth dataset was augmented by flipping and appending frames to each video, resulting in 90,000 images with pixel-wise depth, surface normals, and room keypoint labels (segmentation labels are not available for these additional frames). The keypoint labels were annotated by humans, while the surface normals were generated algorithmically. The entire dataset was then split by scene, with a 90 / 10 training / testing split. These two datasets are referred to in this paper as NYUv2+Segmentation (NYUv2+seg) and NYUv2+Keypoints (NYUv2+kpts).

[0073] All inputs are downsampled to 320 x 320 pixels, while outputs are 80 x 80 pixels. These resolutions allow the model to remain relatively fine-grained without compromising the semantic complexity of the ground truth output maps. The VGG-style model architecture is derived from U.S. Patent Publication No. 2018 / 0268220, entitled “Room layout estimation methods and techniques,” which is incorporated herein by reference in its entirety.

[0074] Model and general training characteristics

[0075] Two different models are studied: (1) a SegNet network with a symmetric VGG16 encoder / decoder, and (2) a fully convolutional network (FCN) network with a modified ResNet-50 encoder and a shallow ResNet decoder. The VGG SegNet reuses the maxpool index to perform upsampling, while the ResNet FCN learns all upsampling filters. The ResNet architecture is further streamlined (in terms of both its filters and activations) to contrast with the heavier and more complex VGG SegNet: stride 2 layers are moved earlier, and all 2048-filter layers are replaced by 1024-filter layers. Ultimately, the VGG SegNet has 29M parameters, while the streamlined ResNet has 15M. All model parameters are shared across all tasks until the last layer. The results show the robustness of GradNorm to the choice of underlying architecture. The aforementioned example models are for illustration only and are not intended to be limiting. GradNorm can be applied to any other type of neural network, including, for example, a recurrent neural network for “Deep Learning System for Cube Detection” as described in U.S. Patent Publication No. 2018 / 0137642 (the contents of which are hereby incorporated by reference in their entirety).

[0076] Standard pixel-wise loss functions are used for each task: cross entropy for segmentation, squared loss for depth, and cosine similarity for normals. As in US Patent Publication No. 2018 / 0268220, Gaussian heatmaps are generated for each of the 48 room keypoint types and predicted with a pixel-wise squared loss. All regression tasks use a quadratic loss (the cosine loss used for surface normal prediction is quadratic in leading order), allowing each task i to use r i (t) serves as a direct proxy for the relative inverse training rate of each task.

[0077] All runs were trained on 4 Titan X GTX 12GB GPUs with a batch size of 24 and ran at 30fps on a single GPU during inference. All NYUv2 runs started with a learning rate of 2e-5. NYUv2+keypoints (kpts) were run for 80,000 steps, with the learning rate decreasing by 0.2 every 25,000 steps. NYUv2+segments (seg) were run for 20,000 steps, with the learning rate decreasing by 0.2 every 6,000 steps. For GradNorm and uncertainty weighted baselines, w was updated at a learning rate of 0.025. i(t). Although GradNorm is insensitive to the chosen optimizer, all optimizers are Adam (a method for stochastic optimization derived from adaptive moment estimation) used to update network weights. GradNorm is implemented using TensorFlow v1.2.1.

[0078] Table 3. Test error of GradNorm and various baselines, NYUv2+seg. Lower values are better. The best performance on each task is shown in bold, and the suboptimal performance is underlined.

[0079]

[0080] Example results for NYUv2

[0081] Table 3 shows an example of GradNorm’s performance on the NYUv2+seg dataset. GradNorm α = 1.5 compares favorably to the equal-weight baseline (for all t, i, w i = 1) improves performance on all three tasks and exceeds or matches (within statistical noise) the best performance of a single network for each task. The time-averaged weights ∑ t [w i (t)], and then retrain the network with weights fixed to these values. The GradNorm static network uses the static weights derived from the GradNorm network. Therefore, GradNorm can also be used to extract good values for the static weights. As shown below, these weights are very close to the optimal weights extracted from the exhaustive grid search.

[0082] To illustrate how GradNorm performs in the presence of larger datasets, extensive experiments were conducted on the NYUv2+kpts dataset, which was augmented with a factor of 50x. The results are shown in Table 4. As with the NYUv2+seg run, the GradNorm network outperforms other multi-task methods and matches (within the noise range) or exceeds the performance of single-task networks.

[0083] Table 4. Test error of GradNorm and various baselines, NYUv2+kpts. Lower values are better. The best performance on each task is shown in bold, and the suboptimal performance is underlined.

[0084]

[0085]

[0086] Figures 3A-3CFigure 2 shows an example of test and training loss curves for GradNorm (α=1.5) and a baseline on the larger NYUv2+kpts dataset for the VGG SegNet model. Despite converging to a much higher training loss, GradNorm reduces the test-time depth error by around 5%. GradNorm achieves this by aggressively rate-balancing the network (due to the highly asymmetric α=1.5) and ultimately scaling the depth weights w depth (t) is suppressed to below 0.10. The same trend is observed for keypoint regression, a clear sign of network regularization. In contrast, uncertainty weighting always shifts the test and training errors in the same direction and is therefore not a good regularizer. Training a stripped-down ResNet FCN also yields similar results.

[0087] Gradient normalization finds optimal grid search weights in one pass

[0088] For VGG SegNet, 100 networks were trained from scratch using random task weights on NYUv2+kpts. Weights were sampled from a uniform distribution and renormalized to sum to T=3. For computational efficiency, training consisted of 15,000 iterations out of the normal 80,000 iterations. The performance of this network was then compared to the GradNormα=1.5 VGGSegNet network over the same 15,000 steps. The results are shown in Figure 4 middle. Figure 4 The grid search performance of random task weights and GradNorm, NYUv2+kpts is shown. and a set of static weights E derived from the GradNorm network t [w i (t)], plotting the L2 distance between Average performance change of a static multi-task network on three tasks. The reference line at zero performance change is shown for reference. All comparisons are performed over 15,000 training steps.

[0089] Even after training 100 networks, the grid search still cannot reach the GradNorm network. Network performance and task weight distance vs. time average GradNorm weight E t [w i (t)]. Grid search networks with an L2 distance of approximately 3 achieve nearly double the average per-task error compared to our GradNorm network. Thus, GradNorm finds the optimal grid search weights in a single training run.

[0090] Adjusting the impact of asymmetric α

[0091] In these example numerical experiments, the only hyperparameter in the method is the hyperparameter α, which is also called the asymmetry parameter as described in this paper. The optimal value of α for NYUv2 is close to α = 1.5, while α = 0.12 was used in the highly symmetric example in the previous section. This observation reinforces the property of α as an asymmetric parameter.

[0092] In these examples, adjusting α leads to improved performance, but it is found that for NYUv2, almost any value of 0 < α < 3 improves network performance over the equal-weight baseline. Figures 5A-5B is the weight w during training NYUv2+kpts i (t) shows the task weights w during training for two different values of α i How the trajectory of (t) changes. Larger values of α push the weights further apart, resulting in less symmetry between tasks. Figures 5A-5B shows that higher values of α tend to increase the weight w i (t) is pushed further out, which more aggressively reduces the impact of tasks that overfit or learn too quickly (in this example, deep). Without affecting the network performance on deep tasks, at α = 1.75 (not shown), suppressing w depth (t) to below 0.02.

[0093] Example qualitative results

[0094] Chen et al., “GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks,” in Proceedings of the 35th International Conference on Machine Learning (2018), 793–802 (hereinafter referred to as “Chen et al.”), which is incorporated herein by reference in its entirety. Figure 6 Figure 3 shows a visualization of the VGG SegNet output on the test set images for the NYUv2+seg and NYUv2+kpts datasets along with the ground truth. The ground truth labels are shown alongside the outputs of the equally weighted network, the three individual networks, and the best GradNorm network. Some of the improvements are incremental, but GradNorm produces superior visual results on the task, as evidenced by the significant quantitative improvements in Tables 3 and 4.

[0095] Chen et al. Figure 6Example visualizations at inference time are shown. NYUv2+kpts output is shown on the left, while NYUv2+seg output is shown on the right. The visualizations shown are generated from random test set images. Some improvements are gradual, but red boxes are shown around predictions where GradNorm visually improves more clearly. For the NYUv2+kpts output, GradNorm is shown to outperform the equally weighted network in normal predictions and the single network in keypoint predictions. For NYUv2+seg, there are improvements over the single network in terms of depth and segmentation accuracy. These are consistent with the numbers reported in Tables 3 and 4.

[0096] Example performance gain comparison α

[0097] The α asymmetry hyperparameter allows for adapting to various priors on the symmetry between tasks. Lower values of α can result in gradient norms having similar magnitudes across tasks, ensuring that each task has roughly the same impact on the training dynamics throughout training. Higher values of α penalize tasks whose loss decreases too quickly, instead placing more weight on tasks whose loss decreases more slowly.

[0098] For the NYUv2 experiment, we choose α = 1.5 as the optimal value of α, and increasing α can increase the task weight w i (t) further apart. For almost all positive values of α where GradNorm is numerically stable, an overall performance gain is obtained. These results are summarized in Figure 6 At large positive values of ≥ 3 in the NYUv2 case, some weights are pushed too close to zero and the GradNorm update may become unstable on this example dataset. Figure 6 Performance gains on NYUv2+kpts for various settings of α are shown. The average performance gain (defined as the average of the percentage change in test loss over an equal-weight baseline across all tasks) is plotted on NYUv2+kpts for various values of α. Results are shown for a VGG16 backbone (solid line) and a ResNet50 backbone (dashed line). Performance gains are observed for all tested values of α, although the gains appear to peak around α = 1.5. For the VGG16 backbone, no points with more than α > 2 are shown as GradNorm weights; however, this is not a limitation of GradNorm.

[0099] like Figure 6As shown, performance gains are achieved at almost all values of α. However, especially for NYUv2+kpts, the performance gains of both backbone architectures seem to peak at α = 1.5. In addition, although the ResNet architecture appears to be more robust to α than the VGG architecture by setting α appropriately, both architectures provide similar levels of gain. The consistent positive performance gains across all values of α suggest that any type of gradient balancing (even under suboptimal conditions) may be beneficial for multi-task network training.

[0100] Example performance on the Multi-Task Facial Landmarks dataset

[0101] Additional experiments were conducted on the Multi-Task Facial Landmark (MTFL) dataset. This dataset contains approximately 13,000 facial images, split into a training set of 10,000 images and a test set of 3,000 images. Each image is annotated with the (x, y) coordinates of five facial landmarks (left eye, right eye, nose, left lip, and right lip), as well as four class labels (gender, smile, glasses, and pose). Example labels from the dataset include (gender: male, smile: real, glasses: fake, pose: frontal), (gender: female, smile: real, glasses: fake, pose: left), and (gender: male, smile: fake, glasses: real, pose: left).

[0102] Since the MTFL dataset is a rich mixture of classification and regression tasks, it provides a good opportunity to test GradNorm. Experiments are conducted at two different image input resolutions: 40×40 and 160×160. For the 40×40 experiments, the same architecture used in MTFL 2014 is used to ensure a fair comparison, while for the 160×160 experiments, a deeper version of the architecture in MTFL 2014 is used: the deeper model layer stack is [CONV-5-16][POOL-2][CONV-3-32] 2 [POOL-2][CONV-3-64] 2 [POOL-2][[CONV-3-128] 2 [POOL-2]] 2 [CONV-3-128] 2 [FC-100][FC-18], where CONV-XF denotes a convolution with filter size X and F output filters, POOL-2 denotes a 2×2 pooling layer with stride 2, and FC-X is a dense layer with output X. All networks output 18 values: 10 coordinates of facial landmarks, and 4 pairs of 2 softmax scores for each classifier.

[0103] Results on the MTFL dataset are shown in Table 5. The keypoint error is the average of the L2 distance errors for all five facial landmarks, normalized to the inter-ocular distance, while the failure rate is the percentage of images for which the keypoint error exceeds 10%. For both resolutions, GradNorm outperforms the other methods on all tasks (except for glasses and pose prediction, where both methods always converge quickly to the majority classifier and reject further training). GradNorm matches the performance of MTFL 2014 on keypoints, although the latter does not attempt to optimize for classifier performance, but rather emphasizes keypoint accuracy. Overall, these results show that GradNorm significantly improves classification accuracy for gender and smile, while matching all other methods on at least all other tasks.

[0104] Table 5. Test error of GradNorm and various baselines on the Multi-Task Facial Landmarks (MTFL) dataset. Lower values are better, and the best performance on each task is indicated in bold. Experiments were performed for two different input resolutions: 40×40 and 160×160. In all cases, GradNorm shows excellent performance, especially on gender and smile classification. GradNorm also matches the performance of MTFL 2014 on keypoint prediction at 40×40 resolution, despite the latter only attempting to optimize keypoint accuracy (degrading classification accuracy in the process).

[0105]

[0106] Glasses and pose classification always converge to the majority classifier. In some embodiments, GradNorm can overcome such tasks becoming "stuck" during training by discontinuously increasing the loss weights for these tasks. GradNorm can mitigate this problem by, for example, detecting pathological tasks online and removing them from the GradNorm update equation.

[0107] GradNorm still provides excellent performance on this dataset. All loss weights are initialized to w i (0) = 1. Uncertainty weighting tends to increase the loss weight of keypoints relative to the classifier loss, while GradNorm actively reduces the relative keypoint loss weight. For GradNorm training runs, w kpt (t) converges to a value ≤ 0.01, which shows that the keypoint task is properly trained without losing accuracy even when the gradient is reduced by two orders of magnitude compared to the uncertainty weighted or equal-weighted methods.

[0108] GradNorm correctly identifies that the classification tasks in the MTFL dataset are relatively untrained and need improvement. In contrast, uncertainty weighting makes the opposite decision by placing more relative emphasis on keypoint regression and generally performs poorly on classification (especially for higher resolution inputs). Therefore, these experiments highlight GradNorm's ability to identify and benefit from tasks that require more attention during training.

[0109] Additional Observations about GradNorm

[0110] As described in this paper, gradient normalization can act as a good model regularizer by operating directly on the gradients in the network and leads to excellent performance in multi-task networks. GradNorm can be driven by an attractive and simple rate-balancing heuristic and can adapt to problems with varying complexity within the same unified model, and in some embodiments, uses only a single hyperparameter representing task asymmetry. GradNorm networks can also be used to quickly extract optimal fixed-task weights, eliminating the need for exhaustive grid search methods, which are exponentially expensive as the number of tasks increases.

[0111] Embodiments of GradNorm can provide an effective method for adjusting loss weights in multi-task learning settings by balancing the training rates of different tasks. For both synthetic and real-world datasets, GradNorm improves multi-task test-time performance in a variety of scenarios and can adapt to various levels of asymmetry between tasks via the hyperparameter α. Results show that GradNorm offers superior performance to other multi-task adaptive weighting methods and can match or exceed the performance of exhaustive grid search while being significantly less time-intensive.

[0112] In some embodiments, GradNorm can have applications beyond multi-task learning. GradNorm can be applied to class balancing and sequence-to-sequence models, all situations where conflicting gradient signals can degrade model performance. GradNorm embodiments can not only provide a robust new approach for multi-task learning, but also demonstrate that gradient regularization can be advantageously used to train large, efficient models on complex tasks.

[0113] Example process of training a multi-task network

[0114] Figure 8800 is a flow chart of an example process for training a multi-task network. The multi-task network can be used to determine outputs associated with multiple tasks. The multi-task network can include multiple shared layers and an output layer including multiple task-specific filters. The output layer of the multi-task network can include an affine transformation layer. A computing system, such as a computing system having non-transitory memory and a hardware processor, can implement process 800 to train the multi-task network. The non-transitory memory of the computing system can store or be configured to store executable instructions. The hardware processor can communicate with the non-transitory memory and can be programmed with the executable instructions to perform process 800 to train the multi-task network using an embodiment of GradNorm.

[0115] Process 800 begins at block 804, where a computing system receives training images associated with a plurality of reference task outputs for a plurality of tasks. The plurality of tasks may include regression tasks, classification tasks, or a combination thereof. The plurality of tasks may include, for example, perception tasks such as facial recognition, visual search, gesture identification or recognition, semantic segmentation, object detection, room layout estimation, cuboid detection, lighting detection, simultaneous localization and mapping, object or avatar relocalization, or speech processing tasks such as speech recognition or natural language processing, or a combination thereof.

[0116] At block 808 , the computing system may determine a single-task loss L between (1) the task output of each task and (2) the corresponding reference task output of the task adjusted by the task weight. i The gradient norm of (t) The gradient norm can be determined relative to multiple network weights of the multi-task network. The corresponding reference task output of the task can be associated with the training image. The gradient norm of the single-task loss adjusted by the task weight can be the L2 norm of the single-task loss adjusted by the task weight.

[0117] The computing system may determine a single-task loss between (1) a task output of each task determined using a multi-task network having a training image as input and (2) a corresponding task output of the task associated with the training image. To determine the single-task loss, the computing system may use a loss function associated with the task from among a plurality of loss functions to determine a single-task loss between (1) a task output of each task determined using a multi-task network having a training image as input and (2) a corresponding task output of the task associated with the training image. The computing system may average the gradient norms of the plurality of tasks Determined as the mean gradient norm.

[0118] At block 812, the computing system may determine a relative training rate for the task based on the single-task loss of the task. To determine the relative training rate for the task based on the single-task loss of the task, the computing system may determine a relative training rate for the task based on the single-task loss of the task.i (t) and another single-task loss for the task (such as L i (0)) to determine the inverse of the relative training rate of the task i To determine the inverse of the relative rate of a task, the computing system may determine the inverse of the relative training rate as a ratio of the loss ratio of the task and an average of the loss ratios of the plurality of tasks.

[0119] At block 816, the computing system may determine the gradient norm including (1) determining for each task The gradient loss function L is the difference between the target gradient norm corresponding to (2) grad The corresponding target gradient norm can be determined based on: (a) the average gradient norm of multiple tasks, (b) the relative training rate of the tasks, and (c) the hyperparameter α, e.g. The gradient loss function may be an L1 loss function. The corresponding target gradient norm may be determined based on: (a) the average gradient norm of multiple tasks, (b) the inverse of the relative training rate of the task, and (c) a hyperparameter. The corresponding target gradient norm is determined by multiplying the average gradient norm of multiple tasks by the power of the hyperparameter of the inverse relative training rate of the task. The hyperparameter α may be constant during training (see for an explanation). Figure 7A ), or can be changed during training (see Figures 7B-7D As described herein, in various embodiments, the hyperparameter α can be in the range of -1 to 3, 0.5 to 3, or some other range.

[0120] At block 820 , the computing system may determine the gradient of the gradient loss function with respect to the task weight of each task in the plurality of tasks. To determine the gradient of the gradient loss function, the computing system may determine the gradient of the gradient loss function with respect to a task weight of each of the plurality of tasks while maintaining a target gradient norm for the task unchanged.

[0121] At block 824, the computing system may use the gradient of the gradient loss function with respect to the task weights To determine the updated task weight w for each task in multiple tasks i (t+1). The computing system may normalize the updated weights of the multiple tasks. To normalize the updated weights of the multiple tasks, the computing system may normalize the updated weights of the multiple tasks to the number of tasks in the multiple tasks.

[0122] The computing system may determine a multi-task loss function including a single-task loss adjusted by a task weight of each task, determine a gradient of the multi-task loss function with respect to all network weights of the multi-task network, and determine updated network weights of the multi-task network based on the gradient of the multi-task loss function.

[0123] Example NN layer

[0124] A layer of a neural network (NN), such as a deep neural network (DNN), can apply a linear or nonlinear transformation to its input to generate its output. A deep neural network layer can be a normalization layer, a convolutional layer, a soft-sign layer, a rectified linear layer, a cascade layer, a pooling layer, a recursive layer, an inception layer, or any combination thereof. A normalization layer can normalize the brightness of its input to generate its output, for example, by L2 normalization. A normalization layer can, for example, normalize the brightness of multiple images relative to each other at once to generate multiple normalized images as its output. Non-limiting examples of methods for normalizing brightness include local contrast normalization (LCN) or local response normalization (LRN). Local contrast normalization can nonlinearly normalize the contrast of an image by normalizing a local region of an image on a per-pixel basis to have a mean of zero and a variance of 1 (or other values of the mean and variance). Local response normalization can normalize an image over a local input region to have a mean of zero and a variance of 1 (or other values of the mean and variance). Normalization layers can speed up the training process.

[0125] A convolutional layer may apply a set of kernels that convolve its input to generate its output. A soft sign layer may apply a soft sign function to its input. The soft sign function (softsign(x)) may be, for example, (x / (1+|x|)). The soft sign layer may ignore the effects of per-element outliers. A rectified linear layer may be a rectified linear unit (ReLU) or a parameterized rectified linear unit (PReLU). A ReLU layer may apply a ReLU function to its input to generate its output. The ReLU function ReLU(x) may be, for example, max(0,x). A PReLU layer may apply a PReLU function to its input to generate its output. The PReLU function PReLU(x) may be, for example, x (if x≥0) and ax (if x<0), where a is a positive number. A cascade layer may concatenate its input to generate its output. For example, a cascade layer may concatenate four 5×5 images to generate a 20×20 image. A pooling layer may apply a pooling function that downsamples its input to generate its output. For example, a pooling layer may downsample a 20x20 image to a 10x10 image. Non-limiting examples of pooling functions include max pooling, average pooling, or min pooling.

[0126] At time point t, the recursive layer can calculate the hidden state s(t), and the recursive connection can provide the hidden state s(t) at time t to the recursive layer as input at a subsequent time point t+1. The recursive layer can calculate the output at time t+1 based on the hidden state s(t) at time t. For example, the recursive layer can apply a soft sign function to the hidden state s(t) at time t to calculate its output at time t+1. The hidden state of the recursive layer at time t+1 takes the hidden state s(t) of the recursive layer at time t+1 as its input. The recursive layer can calculate the hidden state s(t+1) by, for example, applying a ReLU function to its input. The class initialization layer may include one or more of a normalization layer, a convolutional layer, a soft sign layer, a rectified linear layer such as a ReLU layer and a PReLU layer, a cascade layer, a pooling layer, or any combination thereof.

[0127] In different embodiments, the number of layers in a NN can vary. For example, the number of layers in a DNN can be 50, 100, 200, or more. In different embodiments, the input types of a deep neural network layer can vary. For example, a layer can receive the outputs of multiple layers as its input. The input of a layer can include the outputs of five layers. As another example, the input of a layer can include 1% of the NN layer. The output of a layer can be the input of multiple layers. For example, the output of a layer can be used as the input of five layers. As another example, the output of a layer can be used as the input of 1% of the NN layer.

[0128] The input or output size of a layer can be very large. The input or output size of a layer can be n×m, where n is the width and m is the height of the input or output. For example, n or m can be 11, 21, 31, or larger. In different embodiments, the channel size of the input or output of a layer can vary. For example, the channel size of the input or output of a layer can be 4, 16, 32, 64, 128, or larger. In different embodiments, the kernel size of a layer can vary. For example, the kernel size can be n×m, where n is the width of the kernel and m is the height of the kernel. For example, n or m can be 5, 7, 9, or larger. In different embodiments, the stride size of a layer can vary. For example, a deep neural network layer can have a stride size of 3, 5, 7, or larger.

[0129] In some embodiments, NN may refer to a plurality of NNs that together compute the output of the NN. Different NNs in the plurality of NNs may be trained for different tasks. A processor (e.g., referring to Figure 9 The processor of the local data processing module 924 described above can calculate the output of the NN in the plurality of NNs to determine the output of the NN. For example, the output of the NN in the plurality of NNs can include a likelihood score. The processor can determine the output of the NN including the plurality of NNs based on the likelihood scores of the outputs of different NNs in the plurality of NNs.

[0130] Example Wearable Display System

[0131] In some embodiments, the user device may be or may be included in a wearable display device, which may advantageously provide a more immersive virtual reality (VR), augmented reality (AR), or mixed reality (MR) experience in which a digitally reproduced image, or portion thereof, is presented to the wearer in a manner that appears or can be perceived as real.

[0132] Without being limited by theory, it is believed that the human eye can generally interpret a finite number of depth planes to provide depth perception. Therefore, a highly convincing simulation of perceived depth can be achieved by providing the eye with a different presentation of an image corresponding to each of these finite number of depth planes. For example, a display comprising a stack of waveguides can be configured to be worn in front of the eyes of a user or viewer. By using multiple waveguides to direct light from an image injection device (e.g., the output of a discrete display or a multiplexed display that transmits image information through one or more fiber optic conduits) to the viewer's eyes at specific angles (and divergences) corresponding to the depth planes associated with specific waveguides, a stack of waveguides can be used to provide the eye / brain with a three-dimensional perception.

[0133] In some embodiments, two stacks of waveguides can be utilized to provide a different image to each eye, one stack for each eye of the viewer. As an example, an augmented reality scene can cause a wearer of AR technology to see a real-world park-like setting featuring people, trees, buildings in the background, and a concrete platform. In addition to these objects, the wearer of AR technology may also perceive that he "sees" a robotic statue standing on the real-world platform, and that a cartoon-like avatar character flying past him appears to be a personification of a bumblebee, even though the robotic statue and bumblebee do not exist in the real world. The stack of waveguides can be used to generate a light field corresponding to the input image, and in some embodiments, the wearable display includes a wearable light field display. Examples of wearable display devices and waveguide stacks for providing light field images are described in U.S. Patent Publication No. 2015 / 0016777, the entire contents of which are incorporated herein by reference.

[0134] Figure 9An example of a wearable display system 900 is shown, which can be used to present a VR, AR, or MR experience to a display system wearer or viewer 904. The wearable display system 900 can be programmed to perform any application or embodiment described herein (e.g., executing a CNN, reordering the values of an input activation map or kernel, eye image segmentation, or eye tracking). The display system 900 includes a display 908 and various mechanical and electronic modules and systems that support the functionality of the display 908. The display 908 can be coupled to a frame 912, which can be worn by the display system wearer or viewer 904 and is configured to position the display 908 in front of the wearer's 904 eyes. The display 908 can be a light field display. In some embodiments, a speaker 916 is coupled to the frame 912 and positioned near the user's ear canal, and another speaker (not shown) is positioned near the user's other ear canal to provide stereo sound / shapeable sound control. The display system 900 can include an outward-facing imaging system 944 (e.g., one or more cameras) that can capture images (e.g., still images or video) of the environment surrounding the wearer 904. The images obtained by the externally facing imaging system 944 can be referenced in the environment surrounding the wearer 904 Figure 8 An embodiment of a method 800 for training a multi-task network is described for analysis.

[0135] The display 908 can be operably coupled 920 to a local data processing module 924, for example, via a wired lead or a wireless connection, which can be mounted in various configurations, such as fixedly attached to the frame 912, fixedly attached to a helmet or hat worn by the user, embedded in headphones, or otherwise removably attached to the user 904 (e.g., in a backpack configuration, a belt-coupled configuration).

[0136] The local processing and data module 924 can include a hardware processor and non-transitory digital memory (e.g., non-volatile memory (e.g., flash memory)), both of which can be used to assist in processing, caching, and storing data. The data can include: (a) data captured from sensors (which can, for example, be operably coupled to the frame 912 or otherwise attached to the wearer 904), such as image capture devices (e.g., cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, radios, and / or gyroscopes; and / or (b) data acquired and / or processed using the remote processing module 928 and / or remote data repository 932, possibly after such processing or retrieval, and transferred to the display 908. The local processing and data module 924 can be operably coupled to the remote processing module 928 and the remote data repository 932 via communication links 936, 940 (e.g., via wired or wireless communication links), such that these remote modules 928, 932 are operably coupled to each other and can serve as resources for the local processing and data module 924. The image capture device may be used to capture eye images for use in eye image segmentation or eye tracking processes.

[0137] In some embodiments, the remote processing module 928 may include one or more processors configured to analyze and process data and / or image information, such as video information captured by an image capture device. Video data may be stored locally in the local processing and data module 924 and / or in a remote data repository 932. In some embodiments, the remote data repository 932 may include a digital data storage facility that may be accessed via the Internet or other network configurations in a "cloud" resource configuration. In some embodiments, all data is stored and all computations are performed in the local processing and data module 924, allowing for fully autonomous use from the remote module.

[0138] In some embodiments, the local processing and data module 924 and / or the remote processing module 928 are programmed to perform embodiments of reordering the values of the input activation map or kernel, eye image segmentation, or eye tracking as disclosed herein. For example, the local processing and data module 924 and / or the remote processing module 928 can be programmed to use a reference Figure 8The method 800 described herein trains a multi-task network to perform task prediction and determination. An image capture device can capture video of a specific application (e.g., augmented reality (AR), human-computer interaction (HCI), self-driving cars, drones, or robotics in general). The video can be analyzed using a CNN by one or both of the processing modules 924, 928. In some cases, reordering the values of the input activation map or kernel, offloading at least some of the eye image segmentation or eye tracking to a remote processing module (e.g., in the "cloud") can improve computational efficiency or speed. The parameters of the CNN (e.g., weights, bias terms, subsampling factors for pooling layers, the number and size of kernels in different layers, the number of feature maps, etc.) can be stored in a data module.

[0139] The result of the task prediction or determination (e.g., reference Figure 8 The output of the multi-tasking network 800 described above can be used for additional operations or processing by one or both of the processing modules 924, 928. For example, the processing modules 924, 928 of the wearable display system 900 can be programmed to execute additional applications (such as applications in augmented reality, human-computer interaction (HCI), self-driving cars, drones, or general robotics) based on the output of the multi-tasking network.

[0140] Additional tasks

[0141] Embodiments of GradNorm are not limited to computer vision tasks, where multi-task networks are trained on images or visual data. In other embodiments, training samples may include non-image data captured by sensors, such as audio data, acceleration data, positioning data, temperature data, radio frequency data, or optical tracking data. Examples of sensors include audio sensors (e.g., microphones), inertial measurement units (IMUs), accelerometers, compasses, gyroscopes, temperature sensors, motion sensors, depth sensors, global positioning system (GPS) units, and radio equipment. In other embodiments, training samples for medical-related tasks may include measurements such as gender, age, heart rate, body temperature, white blood cell count, disease state, disease progression, symptoms, disease diagnosis, etc. For example, for tasks related to speech recognition or natural language processing, training samples may include audio datasets (or audio data accompanying videos) or electronic representations or embeddings (e.g., n-grams) of words, sentences, paragraphs, or text. Tasks may include, for example, part-of-speech (POS) tagging, chunk dependency parsing, semantic relevance, or text meaning.

[0142] The tasks can be related to medical treatment or medical decision making. For example, the task can include determining which laboratory tests or risk assessments should be performed on patients with a particular disease. For example, the training data can include measurements (e.g., gender, age, heart rate, temperature, white blood cell count, etc.) of patients with a particular disease or condition (e.g., pneumonia). Embodiments of GradNorm can be used to train a multi-task network for predicting a patient's disease risk assessment or laboratory test results.

[0143] Therefore, the training data used by GradNorm can include images, videos, audio, electronic records, databases, etc., which can be associated with corresponding training labels, categories or signals.

[0144] Additional aspects

[0145] In a first aspect, a system for training a multi-task network is disclosed. The system includes: a non-transitory memory configured to store: executable instructions; and a multi-task network for determining outputs associated with a plurality of tasks; and a hardware processor in communication with the non-transitory memory, the hardware processor being programmed by the executable instructions to: receive a training image associated with a plurality of reference task outputs for the plurality of tasks; for each of the plurality of tasks, determine, relative to a plurality of network weights of the multi-task network, a task output of the task determined by the task weights adjusted for the task, (1) using the multi-task network having the training image as input and (2) a corresponding reference task output of the task associated with the training image. a gradient norm of a single-task loss of the task; and determining a relative training rate for the task based on the single-task loss of the task; determining a gradient loss function comprising (1) a difference between the gradient norm determined for each task and (2) a corresponding target gradient norm, wherein the corresponding target gradient norm is determined based on: (a) an average gradient norm of the plurality of tasks, (b) the relative training rate for the task, and (c) a hyperparameter; determining a gradient of the gradient loss function with respect to a task weight for each of the plurality of tasks; and determining an updated task weight for each of the plurality of tasks using the gradient of the gradient loss function with respect to the task weight.

[0146] In aspect 2, the system of aspect 1, wherein the hardware processor is further programmed by the executable instructions to: determine a single-task loss between (1) a task output of each task determined using the multi-task network having the training image as input and (2) the corresponding task output of the task associated with the training image.

[0147] In a third aspect, the system of aspect 2, wherein the non-transitory memory is configured to further store: a plurality of loss functions associated with the plurality of tasks.

[0148] In a fourth aspect, the system of aspect 3, wherein, to determine the single-task loss, the hardware processor is further programmed by the executable instructions to: use a loss function associated with the task from the plurality of loss functions to determine the single-task loss of (1) a task output of each task determined using the multi-task network having the training image as input and (2) the corresponding task output of the task associated with the training image.

[0149] In a fifth aspect, the system of any one of aspects 1-4, wherein the hardware processor is further programmed by the executable instructions to: determine a multi-task loss function comprising the single-task loss adjusted by the task weight of each task; determine the gradient of the multi-task loss function with respect to all network weights of the multi-task network; and determine updated network weights of the multi-task network based on the gradient of the multi-task loss function.

[0150] In a 6th aspect, the system of any one of aspects 1-5, wherein the gradient norm of the single-task loss adjusted by the task weight is the L2 norm of the single-task loss adjusted by the task weight.

[0151] In a 7th aspect, the system of any one of aspects 1-6, wherein the gradient loss function is an L1 loss function.

[0152] In an eighth aspect, the system of any one of aspects 1-7, wherein the hardware processor is further programmed by the executable instructions to: determine an average of the gradient norms of the plurality of tasks as the average gradient norm.

[0153] In aspect 9, the system of any one of aspects 1-8, wherein the corresponding target gradient norm is determined based on: (a) an average gradient norm of the plurality of tasks, (b) an inverse of the relative training rate of the task, and (c) a hyperparameter.

[0154] In aspect 10, the system of aspect 9, wherein the hardware processor is further programmed by the executable instructions to: determine the corresponding target gradient norm by multiplying the average gradient norm of the plurality of tasks by a power of the hyperparameter of the inverse relative training rate of the task.

[0155] In an 11th aspect, the system of any of aspects 9-10, wherein, to determine the relative training rate of the task based on the single-task loss of the task, the hardware processor is further programmed by the executable instructions to: determine an inverse of the relative training rate of the task based on a loss ratio of the single-task loss of the task to another single-task loss of the task.

[0156] In a 12th aspect, the system of aspect 11, wherein, to determine the inverse of the relative rate of the task, the hardware processor is further programmed by the executable instructions to: determine a ratio of the loss ratio of the task to an average of the loss ratios of the plurality of tasks as the inverse of the relative training rate.

[0157] In a 13th aspect, the system of any one of aspects 1-12, wherein, to determine the gradient of the gradient loss function, the hardware processor is further programmed by the executable instructions to: determine the gradient of the gradient loss function with respect to a task weight of each of the plurality of tasks while maintaining the target gradient norm of the task unchanged.

[0158] In a 14th aspect, the system of any of aspects 1-13, wherein the hardware processor is further programmed by the executable instructions to: normalize the updated weights of the plurality of tasks.

[0159] In a 15th aspect, the system of aspect 14, wherein, to normalize the updated weights of the plurality of tasks, the hardware processor is further programmed by the executable instructions to: normalize the updated weights of the plurality of tasks to the number of the plurality of tasks.

[0160] In a 16th aspect, the system of any one of aspects 1-15, wherein the plurality of tasks comprises a regression task, a classification task, or a combination thereof.

[0161] In a 17th aspect, the system of aspect 16, wherein the classification task comprises perception, facial recognition, visual search, gesture recognition, semantic segmentation, object detection, room layout estimation, cuboid detection, lighting detection, simultaneous localization and mapping, relocalization, speech processing, voice recognition, natural language processing, or a combination thereof.

[0162] In an 18th aspect, the system of any one of aspects 1-17, wherein the multi-task network comprises a plurality of shared layers and an output layer comprising a plurality of task-specific filters.

[0163] In a 19th aspect, the system of aspect 18, wherein the output layer of the multi-task network comprises an affine transformation layer.

[0164] In aspect 20, a method for training a multi-task network is disclosed. The method is under the control of a hardware processor and includes: receiving training data from a plurality of training data, each training data being associated with a plurality of reference task outputs for a plurality of tasks; determining, for each task in the plurality of tasks, a gradient norm of a single-task loss adjusted by a task weight of the task relative to a plurality of network weights of the multi-task network, the single-task loss being a single-task loss of (1) a task output of the task determined using the multi-task network having the training data as input and (2) a corresponding reference task output of the task associated with the training data; and determining a relative training rate for the task based on the single-task loss of the task; determining a gradient loss function comprising a difference between (1) the determined gradient norm of each task and (2) a corresponding target gradient norm, the corresponding target gradient norm being determined based on: (a) an average gradient norm of the plurality of tasks, and (b) the relative training rate of the task; and determining an updated task weight for each task in the plurality of tasks using the gradient of the gradient loss function relative to the task weight.

[0165] In aspect 21, the method of aspect 20, wherein the plurality of training data comprises a plurality of training images, and wherein the plurality of tasks comprises computer vision tasks, speech recognition tasks, natural language processing tasks, medical diagnosis tasks, or a combination thereof.

[0166] In aspect 22, the method according to any one of aspects 20-21 further includes: determining a single-task loss of (1) a task output of each task determined using the multi-task network having the training image as input and (2) the corresponding task output of the task associated with the training image.

[0167] In aspect 23, the method of aspect 22, wherein determining the single-task loss comprises using a loss function associated with the task from the plurality of loss functions to determine the single-task loss of (1) a task output of each task determined using the multi-task network having the training image as input and (2) the corresponding task output of the task associated with the training image.

[0168] In aspect 24, the method according to any one of aspects 20-23 further includes: determining a multi-task loss function including the single-task loss adjusted by the task weight of each task; determining the gradient of the multi-task loss function relative to all network weights of the multi-task network; and determining the updated network weights of the multi-task network based on the gradient of the multi-task loss function.

[0169] In a 25th aspect, the method according to any one of aspects 20-24, wherein the gradient norm of the single-task loss adjusted by the task weight is the L2 norm of the single-task loss adjusted by the task weight.

[0170] In a 26th aspect, the method of any one of aspects 20-25, wherein the gradient loss function is an L1 loss function.

[0171] In a 27th aspect, the method according to any one of aspects 20-26 further comprises: determining an average value of the gradient norms of the plurality of tasks as the average gradient norm.

[0172] In aspect 28, the method according to any one of aspects 20-27 further includes: the corresponding target gradient norm is determined based on: (a) the average gradient norm of the multiple tasks, (b) the relative training rate of the tasks, and (c) hyperparameters.

[0173] In aspect 29, the method according to any one of aspects 20-27 further includes: the corresponding target gradient norm is determined based on: (a) the average gradient norm of the multiple tasks, (b) the inverse of the relative training rate of the task, and (c) hyperparameters.

[0174] In aspect 30, the method according to aspect 29 further comprises: determining the corresponding target gradient norm as the average gradient norm of the plurality of tasks multiplied by a power of the hyperparameter of the inverse relative training rate of the task.

[0175] In aspect 31, the method of any one of aspects 29-30, wherein determining the relative training rate of the task based on the single-task loss of the task comprises determining an inverse of the relative training rate of the task based on a loss ratio of the single-task loss of the task to another single-task loss of the task.

[0176] In aspect 32, the method of aspect 31, wherein determining the inverse of the relative rate of the task comprises determining a ratio of the loss ratio of the task to an average of the loss ratios of the plurality of tasks as the inverse of the relative training rate.

[0177] In a 33rd aspect, the method of any one of aspects 20-32 further comprises determining a gradient of the gradient loss function with respect to a task weight of each task in the plurality of tasks.

[0178] In aspect 34, the method of aspect 33, wherein determining the gradient of the gradient loss function comprises determining the gradient of the gradient loss function with respect to a task weight of each of the plurality of tasks while maintaining the target gradient norm of the task unchanged.

[0179] In a 35th aspect, the method according to any one of aspects 20-34 further comprises normalizing the updated weights of the plurality of tasks.

[0180] In a 36th aspect, the method of aspect 35, wherein normalizing the updated weights of the plurality of tasks comprises normalizing the updated weights of the plurality of tasks to the number of the plurality of tasks.

[0181] In a 37th aspect, the method according to any one of aspects 20-36, wherein the plurality of tasks comprises a regression task, a classification task, or a combination thereof.

[0182] In a 38th aspect, the method of aspect 37, wherein the classification task comprises perception, facial recognition, visual search, gesture recognition, semantic segmentation, object detection, room layout estimation, cuboid detection, lighting detection, simultaneous localization and mapping, relocalization, speech processing, voice recognition, natural language processing, or a combination thereof.

[0183] In a 39th aspect, the method of any one of aspects 20-38, wherein the multi-task network comprises a plurality of shared layers and an output layer comprising a plurality of task-specific filters.

[0184] In a 40th aspect, the method of aspect 39, wherein the output layer of the multi-task network comprises an affine transformation layer.

[0185] In a 41st aspect, a head-mounted display system is disclosed. The system comprises: a non-transitory memory configured to store: executable instructions; and a multi-task network according to any one of aspects 1-40; a display; a sensor; and a hardware processor in communication with the non-transitory memory and the display, the hardware processor being programmed by the executable instructions to: receive sensor data captured by the sensor; determine a task output for each of a plurality of tasks using the multi-task network having the sensor data as input; and cause the display to display information related to the determined task output to a user of an augmented reality device.

[0186] In a 42nd aspect, a head-mounted display system is disclosed. The system includes: a non-transitory memory configured to store: executable instructions; and a multi-task network for determining outputs associated with a plurality of tasks, wherein the multi-task network is trained using: (1) a gradient norm of a single-task loss of a task output of a task in the plurality of tasks determined using the multi-task network having training data as input and (2) a corresponding reference task output of the task associated with the training data, relative to a plurality of network weights of the multi-task network adjusted by the task weights of the task, and a relative training rate for the task determined based on the single-task loss of the task, including a gradient norm between (1) the determined gradient norm of the task output and (2) the corresponding target gradient norm. a gradient loss function of the difference between the values of the gradient loss function and ...

[0187] In a 43rd aspect, the system of aspect 42, wherein the sensor comprises an inertial measurement unit, an external-facing camera, a depth-sensing camera, a microphone, an eye-imaging camera, or a combination thereof.

[0188] In a 44th aspect, the system of any one of aspects 42-43, wherein the plurality of tasks comprises one or more perception tasks, one or more regression tasks, one or more classification tasks, speech recognition tasks, natural language processing tasks, medical diagnosis tasks, or a combination thereof.

[0189] In a 45th aspect, the system of aspect 44, wherein the perception task comprises facial recognition, visual search, gesture identification, semantic segmentation, object detection, lighting detection, simultaneous localization and mapping, relocalization, or a combination thereof.

[0190] In a 46th aspect, the system of any one of aspects 44-45, wherein the classification task comprises perception, facial recognition, visual search, gesture recognition, semantic segmentation, object detection, room layout estimation, cuboid detection, lighting detection, simultaneous localization and mapping, relocalization, speech processing, voice recognition, natural language processing, or a combination thereof.

[0191] In aspect 47, the system of any one of aspects 42-46, wherein the multi-task network is trained by: receiving training data from a plurality of training data, each training data being associated with a plurality of reference task outputs for a plurality of tasks; determining, for each task in the plurality of tasks, a gradient norm of a single-task loss adjusted by a task weight of the task relative to a plurality of network weights of the multi-task network, the single-task loss being a single-task loss of (1) a task output of the task determined using the multi-task network having the training data as input and (2) a corresponding reference task output of the task associated with the training data; and determining a relative training rate for the task based on the single-task loss of the task; determining a gradient loss function comprising a difference between (1) the determined gradient norm for each task and (2) a corresponding target gradient norm, the corresponding target gradient norm being determined based on: (a) an average gradient norm of the plurality of tasks, and (b) the relative training rate for the task; and determining an updated task weight for each task in the plurality of tasks using a gradient of the gradient loss function relative to the task weight.

[0192] In a 48th aspect, the system of aspect 47, wherein the plurality of training data comprises a plurality of training images, and wherein the plurality of tasks comprises computer vision tasks.

[0193] In aspect 49, the system of any one of aspects 47-48, wherein the multi-task network is trained by determining a single-task loss between (1) a task output for each task determined using the multi-task network having the training image as input and (2) the corresponding task output for the task associated with the training image.

[0194] In aspect 50, the system of aspect 49, wherein determining the single-task loss comprises using a loss function associated with the task from the plurality of loss functions to determine the single-task loss between (1) a task output of each task determined using the multi-task network having the training image as input and (2) the corresponding task output of the task associated with the training image.

[0195] In aspect 51, the system of any one of aspects 47-50, wherein the multi-task network is trained by: determining a multi-task loss function comprising the single-task loss adjusted by a task weight of each task; determining the gradient of the multi-task loss function with respect to all network weights of the multi-task network; and determining updated network weights of the multi-task network based on the gradient of the multi-task loss function.

[0196] In a 52nd aspect, the system of any one of aspects 47-51, wherein the gradient norm of the single-task loss adjusted by the task weight is the L2 norm of the single-task loss adjusted by the task weight.

[0197] In a 53rd aspect, the system of any one of aspects 47-52, wherein the gradient loss function is an L1 loss function.

[0198] In a 54th aspect, the system of any one of aspects 47-53, wherein the multi-task network is trained by determining an average of the gradient norms of the plurality of tasks as the average gradient norm.

[0199] In a 55th aspect, the system of any one of aspects 47-54, wherein the multi-task network is trained by the following steps: determining the corresponding target gradient norm based on (a) the average gradient norm of the multiple tasks, (b) the relative training rates of the tasks, and (c) hyperparameters.

[0200] In a 56th aspect, the system of any one of aspects 47-54, wherein the multi-task network is trained by: determining the corresponding target gradient norm based on (a) an average gradient norm of the plurality of tasks, (b) an inverse of the relative training rates of the tasks, and (c) a hyperparameter.

[0201] In a 57th aspect, the system of aspect 56 trains the multi-task network by determining the corresponding target gradient norm by multiplying the average gradient norm of the plurality of tasks by a power of the hyperparameter of the inverse relative training rate of the task.

[0202] In aspect 58, the system of any one of aspects 56-57, wherein determining the relative training rate of the task based on the single-task loss of the task comprises determining an inverse of the relative training rate of the task based on a loss ratio of the single-task loss of the task to another single-task loss of the task.

[0203] In a 59th aspect, the system of aspect 58, wherein determining the inverse of the relative rate of the task comprises determining a ratio of the loss ratio of the task to an average of the loss ratios of the plurality of tasks as the inverse of the relative training rate.

[0204] In a 60th aspect, the system of any one of aspects 47-59, wherein the multi-task network is trained by determining a gradient of the gradient loss function with respect to a task weight for each of the plurality of tasks.

[0205] In a 61st aspect, the system of aspect 60, wherein determining the gradient of the gradient loss function comprises determining the gradient of the gradient loss function with respect to a task weight of each of the plurality of tasks while maintaining the target gradient norm for the task constant.

[0206] In a 62nd aspect, the system of any one of aspects 47-61, wherein the multi-task network is trained by normalizing the updated weights of the plurality of tasks.

[0207] In a 63rd aspect, the system of aspect 62, wherein normalizing the updated weights of the plurality of tasks comprises normalizing the updated weights of the plurality of tasks to the number of the plurality of tasks.

[0208] In a 64th aspect, the system of any one of aspects 47-63, wherein the multi-task network comprises a plurality of shared layers and an output layer comprising a plurality of task-specific filters.

[0209] In a 65th aspect, the system of aspect 64, wherein the output layer of the multi-task network comprises an affine transformation layer.

[0210] In a 66th aspect, a method for training a multi-task neural network for determining outputs associated with a plurality of tasks is disclosed. The method is under the control of a hardware processor and includes: receiving a set of training samples associated with a plurality of reference task outputs for the plurality of tasks; computing a multi-task loss function based at least in part on a weighted combination of single-task loss functions, wherein the weights in the weighted multi-task loss function can vary at each training step; during the training, determining a weight for each of the single-task loss functions such that each of the plurality of tasks is trained at a similar rate; and outputting a trained multi-task neural network based at least in part on the training.

[0211] In a 67th aspect, the method of aspect 66, wherein the task comprises a computer vision task, a speech recognition task, a natural language processing task, or a medical diagnosis task.

[0212] In a 68th aspect, the method of any one of aspects 66-67, wherein the multi-task loss function is a linear combination of the weights and the single-task loss functions.

[0213] In aspect 69, the method of any one of aspects 66-68, wherein determining the weight of each of the single-task loss functions comprises penalizing the multi-task neural network when back-propagated gradients starting from a first task in the plurality of tasks are substantially different from back-propagated gradients starting from a second task in the plurality of tasks.

[0214] In aspect 70, the method of any one of aspects 66-69, wherein determining the weight of each of the single-task loss functions comprises reducing a first weight of a first task of the plurality of tasks relative to a second weight of a second task of the plurality of tasks when a first training rate of the first task exceeds a second training rate of the second task of the plurality of tasks.

[0215] In aspect 71, the method of any one of aspects 66-70, wherein determining the weight of each of the single-task loss functions comprises: evaluating the gradient norm of the weighted single-task loss function for each of the plurality of tasks relative to the weight during training; evaluating the average gradient norm over all tasks during training; calculating a relative inverse training rate for each of the plurality of tasks; and calculating the gradient loss function based at least in part on a difference between the gradient norm of each of the weighted single-task loss functions and a function of the average gradient norm multiplied by the relative inverse training rate.

[0216] In a 72nd aspect, the method of aspect 71, wherein the gradient loss function comprises an L-1 loss function.

[0217] In a 73rd aspect, the method of any one of aspects 71-72, wherein the function of the relative inverse training rate comprises a power law function.

[0218] In a 74th aspect, the method of aspect 73, wherein the power law function has a power law exponent in the range of -1 to 3.

[0219] In a 75th aspect, the method of aspect 73, wherein the power law function has a power law exponent that varies during the training.

[0220] Additional Notes

[0221] Each process, method, and algorithm described herein and / or depicted in the accompanying drawings may be embodied in a code module comprised of one or more physical computing systems, hardware computer processors, dedicated circuits, and / or electronic hardware configured to execute specific and particular computer instructions and may be automated in whole or in part. For example, a computing system may include a general-purpose computer (e.g., a server) or a dedicated computer, dedicated circuit, or the like programmed with specific computer instructions. The code module may be compiled and linked into an executable program, installed in a dynamic link library, or may be written in an interpreted programming language. In some embodiments, specific operations and methods may be performed by circuits specific to a given function.

[0222] In addition, certain implementations of the functionality of the present disclosure are sufficiently mathematically, computationally, or technologically complex that specialized hardware or one or more physical computing devices (utilizing appropriate specialized executable instructions) may be required to perform the functionality, e.g., due to the amount or complexity of the computations involved, or to provide results in substantially real time. For example, a video may include many frames, each with millions of pixels, and specially programmed computer hardware is required to process this video data to provide the desired image processing task or application in a commercially reasonable amount of time. As another example, training a deep multi-task network using embodiments of the GradNorm method described herein is computationally challenging and may be implemented on a graphics processing unit (GPU), an application specific integrated circuit (ASIC), or a floating point gate array (FPGA).

[0223] Code modules or any type of data may be stored on any type of non-transitory computer-readable medium, such as physical computer storage devices including hard drives, solid-state memory, random access memory (RAM), read-only memory (ROM), optical disks, volatile or non-volatile storage devices, combinations thereof, and / or the like. Methods and modules (or data) may also be sent as generated data signals (e.g., as part of a carrier wave or other analog or digital propagation signal) over various computer-readable transmission media (including wireless-based and wired / cable-based media) and may take various forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). The results of the disclosed processes or process steps may be stored permanently or otherwise in any type of non-transitory tangible computer storage device, or may be communicated via computer-readable transmission media.

[0224] Any process, block, state, step or function in the flowcharts described herein and / or depicted in the accompanying drawings should be understood to potentially represent a code module, code segment or code portion comprising one or more executable instructions for implementing a specific function (such as logic or arithmetic) or step in the process. Various processes, blocks, states, steps or functions can be combined, rearranged, added, deleted, modified or otherwise changed with the illustrative examples provided herein. In some embodiments, additional or different computing systems or code modules can perform some or all of the functions described herein. The methods and processes described herein are also not limited to any particular order, and the blocks, steps or states associated therewith can be performed in other appropriate orders, such as serially, in parallel or in some other manner. Tasks or events can be added to or deleted from the disclosed example embodiments. In addition, the separation of the various system components in the embodiments described herein is for illustrative purposes and should not be understood as requiring such separation in all embodiments. It should be understood that the program components, methods and systems described can generally be integrated together in a single computer product or packaged into multiple computer products. Many variations of the embodiments are possible.

[0225] The processes, methods, and systems can be implemented in a network (or distributed) computing environment. Network environments include enterprise-wide computer networks, intranets, local area networks (LANs), wide area networks (WANs), personal area networks (PANs), cloud computing networks, crowd-sourced computing networks, the Internet, and the World Wide Web. The network can be a wired or wireless network or any other type of communication network.

[0226] The systems and methods of the present disclosure each have several innovative aspects, none of which is solely responsible for or requires the desired properties disclosed herein. The various features and processes described herein can be used independently of each other, or can be combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of the present disclosure. Various modifications to the embodiments described in this disclosure will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the claims are not intended to be limited to the embodiments shown herein, but should be given the widest scope consistent with the present disclosure, principles and novel features disclosed herein.

[0227] Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable subcombination. Moreover, although features may be described above as functioning in certain combinations and even initially claimed as such, in some cases one or more features in a claimed combination may be removed from that combination, and the claimed combination may be used in subcombinations or variations of the subcombinations. No single feature or set of features is essential or necessary for every embodiment.

[0228] Conditional language used herein, particularly words such as "may," "could," "might," "can," "for example," etc., unless expressly stated otherwise or understood otherwise in the context, is generally intended to convey that certain embodiments include and other embodiments do not include certain features, elements, and / or steps. Therefore, such conditional language is generally not intended to imply that features, elements, and / or steps are in any way necessary for one or more embodiments, or that one or more embodiments necessarily include logic for deciding (whether or not there is author input or prompting) whether these features, elements, and / or steps will be included or performed in any particular embodiment. The terms "include," "comprise," "have," etc. are synonyms and are used inclusively in an open-ended manner and do not exclude additional elements, features, actions, operations, etc. Similarly, the term "or" is used in its inclusive sense (rather than in its exclusive sense), so that, for example, when used to connect a list of elements, the term "or" means one, some, or all of the elements in the list. In addition, "a," "an," and "the" used in this application and the appended claims should be interpreted as meaning "one or more" or "at least one," unless otherwise stated.

[0229] As used herein, a phrase referring to "at least one" of a list of items refers to any combination of those items, including individual members. For example, "at least one of A, B, or C" is intended to encompass: A, B, C, A and B, A and C, B and C, and A, B, and C. Unless otherwise specifically stated, conjunctions such as the phrase "at least one of X, Y, and Z" should be understood in conjunction with the context that is typically used to convey that an item, term, etc. can be at least one of X, Y, or Z. Thus, such conjunctions are generally not intended to imply that certain embodiments require the presence of at least one of X, at least one of Y, and at least one of Z.

[0230] Similarly, although operations may be depicted in the figures in a particular order, it should be understood that such operations need not be performed in the particular order or sequence shown, or that all illustrated operations need not be performed to achieve the desired result. In addition, the figures may schematically depict one or more example processes in the form of flow charts. However, other operations not shown may be incorporated into the schematically illustrated example methods and processes. For example, one or more additional operations may be performed before, after, simultaneously with, or between any of the illustrated operations. In addition, in other embodiments, the operations may be rearranged or reordered. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the above-described embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged into multiple software products. In addition, other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result.

Claims

1. A system for training a multi-task network, comprising: Non-transitory memory configured to store: executable instructions, and Multi-task networks, used to determine outputs associated with multiple tasks; as well as a hardware processor in communication with the non-transitory memory, the hardware processor being programmed by the executable instructions to: receiving training images associated with a plurality of reference task outputs for the plurality of tasks; For each task in the plurality of tasks, determining, with respect to a plurality of network weights of the multi-task network, a gradient norm of a single-task loss of (1) a task output of the task determined using the multi-task network having the training image as input and (2) a corresponding reference task output of the task associated with the training image, adjusted by the task weights of the task; and determining a relative training rate for the task based on the single-task loss for the task; determining a gradient loss function comprising a difference between (1) a gradient norm determined for each task and (2) a corresponding target gradient norm, the corresponding target gradient norm being determined based on: (a) an average gradient norm of the plurality of tasks, (b) the relative training rates of the tasks, and (c) a hyperparameter; determining a gradient of the gradient loss function with respect to a task weight of each of the plurality of tasks; as well as An updated task weight for each of the plurality of tasks is determined using the gradient of the gradient loss function with respect to the task weight.

2. The system according to claim 1, wherein: The hardware processor is further programmed by the executable instructions to determine a single-task loss between (1) a task output for each task determined using the multi-task network having the training image as input and (2) the corresponding task output for the task associated with the training image.

3. The system according to claim 2, wherein: The non-transitory memory is configured to further store: a plurality of loss functions associated with the plurality of tasks.

4. The system according to claim 3, wherein: To determine the single-task loss, the hardware processor is further programmed by the executable instructions to: use a loss function associated with the task from the multiple loss functions to determine the single-task loss between (1) a task output of each task determined using the multi-task network having the training image as input and (2) the corresponding task output of the task associated with the training image.

5. The system according to claim 1, wherein: The hardware processor is further programmed by the executable instructions to: determining a multi-task loss function comprising the single-task loss adjusted by a task weight of each task; determining the gradient of the multi-task loss function with respect to all network weights of the multi-task network; and Based on the gradient of the multi-task loss function, an updated network weight of the multi-task network is determined.

6. The system according to claim 1, wherein: The gradient norm of the single-task loss adjusted by the task weight is an L2 norm of the single-task loss adjusted by the task weight.

7. The system according to claim 1, wherein: The gradient loss function is an L1 loss function.

8. The system according to claim 1, wherein: The hardware processor is further programmed by the executable instructions to determine an average of the gradient norms of the plurality of tasks as the average gradient norm.

9. The system according to claim 1, wherein: The corresponding target gradient norm is determined based on: (a) an average gradient norm of the plurality of tasks, (b) an inverse of the relative training rate of the task, and (c) a hyperparameter.

10. The system according to claim 9, wherein: The hardware processor is further programmed by the executable instructions to determine the corresponding target gradient norm by multiplying the average gradient norm of the plurality of tasks by a power of the hyperparameter of the inverse of the relative training rate of the task.

11. The system according to claim 9, wherein: To determine the relative training rate of the task based on the single-task loss of the task, the hardware processor is further programmed by the executable instructions to: determine an inverse of the relative training rate of the task based on a loss ratio of the single-task loss of the task to another single-task loss of the task.

12. The system according to claim 11, wherein To determine the inverse of the relative training rate for the task, the hardware processor is further programmed by the executable instructions to: determine a ratio of the loss ratio for the task to an average of the loss ratios of the plurality of tasks as the inverse of the relative training rate.

13. The system according to claim 1, wherein: To determine the gradient of the gradient loss function, the hardware processor is further programmed by the executable instructions to: determine the gradient of the gradient loss function with respect to a task weight of each of the plurality of tasks while maintaining the corresponding target gradient norm of the task unchanged.

14. The system according to claim 1, wherein: The hardware processor is further programmed by the executable instructions to normalize the updated task weights of the plurality of tasks.

15. The system according to claim 14, wherein: To normalize the updated task weights of the plurality of tasks, the hardware processor is further programmed by the executable instructions to normalize the updated task weights of the plurality of tasks to the number of the plurality of tasks.

16. The system of claim 1, wherein: The multiple tasks include regression tasks, classification tasks, or a combination thereof.

17. The system according to claim 16, wherein: The classification tasks include perception, facial recognition, visual search, gesture recognition, semantic segmentation, object detection, room layout estimation, cuboid detection, lighting detection, simultaneous localization and mapping, relocalization, speech processing, voice recognition, natural language processing, or a combination thereof.

18. The system of claim 1, wherein: The multi-task network includes a plurality of shared layers and an output layer including a plurality of task-specific filters.

19. The system according to claim 18, wherein: The output layer of the multi-task network includes an affine transformation layer.

20. A method for training a multi-task network, comprising: Under the control of the hardware processor receiving training data from a plurality of training data, each training data being associated with a plurality of reference task outputs for a plurality of tasks; For each task in the plurality of tasks, determining, with respect to a plurality of network weights of the multi-task network, a gradient norm of a single-task loss adjusted by the task weights of the task, the single-task loss being a single-task loss between (1) a task output of the task determined using the multi-task network having the training data as input and (2) a corresponding reference task output of the task associated with the training data; as well as determining a relative training rate for the task based on the single-task loss for the task; determining a gradient loss function comprising a difference between (1) the determined gradient norm for each task and (2) a corresponding target gradient norm, the corresponding target gradient norm being determined based on: (a) an average gradient norm of the plurality of tasks, and (b) the relative training rates of the tasks; as well as determining an updated task weight for each of the plurality of tasks using a gradient of a gradient loss function with respect to the task weight, Wherein, the plurality of training data comprises a plurality of training images, and wherein the plurality of tasks comprises computer vision tasks.

21. The method according to claim 20, wherein The corresponding target gradient norm is determined based on: (a) an average gradient norm of the plurality of tasks, (b) the relative training rates of the tasks, and (c) hyperparameters.

22. The method according to claim 20, further comprising: A gradient of the gradient loss function with respect to a task weight of each of the plurality of tasks is determined.

23. A head-mounted display system comprising: Non-transitory memory configured to store: executable instructions, and A multi-task network for determining outputs associated with a plurality of tasks, wherein the multi-task network is trained using: a gradient norm of a single-task loss of (1) a task output of a task of the plurality of tasks determined using the multi-task network having a training image as input and (2) a corresponding reference task output of the task associated with the training image, adjusted by the task weights of the task, relative to a plurality of network weights of the multi-task network, a relative training rate for the task determined based on the single-task loss for the task, a gradient loss function comprising a difference between (1) the determined gradient norm of the task and (2) a corresponding target gradient norm, the corresponding target gradient norm being determined based on: (a) an average gradient norm of the plurality of tasks, (b) the relative training rates of the tasks, and (c) hyperparameters, and updating a task weight for the task using a gradient of the gradient loss function with respect to the task weight for the task; monitor; Sensors; and a hardware processor in communication with the non-transitory memory and the display, the hardware processor being programmed by the executable instructions to: receiving sensor data captured by the sensor; determining a task output for each of the plurality of tasks using the multi-task network having the sensor data as input; and The display is caused to display information related to the determined task output to a user of the head mounted display system.

24. The system of claim 23, wherein: The plurality of tasks includes a plurality of perception tasks.

25. The system of claim 24, wherein: The plurality of perception tasks include facial recognition, visual search, gesture identification, semantic segmentation, object detection, lighting detection, simultaneous localization and mapping, relocalization, or a combination thereof.

26. The system of claim 23, wherein: The sensor includes an inertial measurement unit, an external-facing camera, a depth-sensing camera, a microphone, an eye-imaging camera, or a combination thereof.

27. A method for training a multi-task neural network for determining outputs associated with a plurality of tasks, the method comprising: Under the control of the hardware processor: receiving a set of training samples associated with a plurality of reference task outputs for the plurality of tasks; computing a multi-task loss function based at least in part on a weighted combination of the single-task loss functions, wherein weights in the weighted multi-task loss function can vary at each training step; During the training, determining a weight for each of the single-task loss functions so that each of the plurality of tasks is trained at a similar rate; and outputting a trained multi-task neural network based at least in part on the training, The tasks include computer vision tasks, speech recognition tasks, natural language processing tasks, or medical diagnosis tasks.

28. The method according to claim 27, wherein The multi-task loss function is a linear combination of the weights and the single-task loss function.

29. The method according to claim 27, wherein Determining a weight for each of the single-task loss functions includes penalizing the multi-task neural network when a back-propagated gradient from a first task in the plurality of tasks is different from a back-propagated gradient from a second task in the plurality of tasks.

30. The method of claim 27, wherein: Determining the weight of each of the single-task loss functions includes reducing a first weight of a first task in the plurality of tasks relative to a second weight of a second task in the plurality of tasks when a first training rate of the first task exceeds a second training rate of the second task in the plurality of tasks.

31. The method of claim 27, wherein: Determining the weight of each of the single-task loss functions includes: evaluating a gradient norm of a weighted single-task loss function for each of the plurality of tasks relative to the weights during training; Evaluate the mean gradient norm over all tasks during the training; calculating a relative inverse training rate for each of the plurality of tasks; and A gradient loss function is calculated based at least in part on a difference between a gradient norm of each of the weighted single-task loss functions and a function of the average gradient norm multiplied by the relative inverse training rate.

32. The method according to claim 31, wherein The gradient loss function includes an L1 loss function.

33. The method according to claim 31, wherein The function of the relative inverse training rate comprises a power law function.

34. The method according to claim 33, wherein The power law function has a power law exponent in the range of -1 to 3.

35. The method of claim 33, wherein: The power law function has a power law exponent that varies during the training.

Images