Meta-learning for Multi-task Learning of Neural Networks

Abstract

Methods and systems for meta-learning for automatically learning subtasks by a single neural network are described. The order in which a neural network learns tasks can affect the performance of the network, and the meta-learning method can use a task-level curriculum for multi-task training. The task-level curriculum can be learned by monitoring the trajectory of the loss function during training. The meta-learning method can learn to adapt task-loss balancing weights during training to improve performance on multiple tasks on real-world datasets. Advantageously, learning to dynamically balance weights between different task losses can lead to higher performance than using static weights determined by expensive random search or heuristic methods. Embodiments of the meta-learning method can be used for computer vision tasks or natural language processing tasks, and the trained neural network can be used by augmented or virtual reality devices.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to U.S. patent application No. 62 / 586,154, filed on November 14, 2017, entitled “Meta-learning for Multi-task Learning,” which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates generally to systems and methods for machine learning, and more particularly to using meta-learning techniques to train machine learning models. Background Art

[0004] A deep neural network (DNN) is a computational machine learning method. DNNs belong to a class of artificial neural networks (NNs). Using NNs, a computational graph can be constructed that mimics the characteristics of biological neural networks. Biological neural networks include features that are important for computation and are responsible for many functions of biological systems that might otherwise be difficult to capture using other methods. In some embodiments, such networks are arranged into a sequential layered structure in which connections are unidirectional. For example, the output of an artificial neuron in a particular layer can be connected to the input of an artificial neuron in a subsequent layer. A DNN can be a NN with a large number of layers (e.g., 10, 100, or more layers).

[0005] Different NNs differ from each other in different ways. For example, the topology or architecture (e.g., the number of layers and how they are interconnected) and weights of different NNs can differ. Weights can be roughly analogous to the synaptic strength of neural connections in biological systems. Weights influence the strength of the effect propagated from one layer to another. The output of an artificial neuron can be a nonlinear function of the weighted sum of its inputs. The weights of a NN can be the weights that appear in these sums. Summary of the Invention

[0006] A meta-learning method for automatically learning multiple tasks with a single neural network and associated methods and systems are described. The order in which a neural network learns tasks may affect the performance of the network, and the meta-learning method can use a task-level curriculum for multi-task training. The task-level curriculum can be learned by monitoring the trajectory of the loss function during training. The meta-learning method can learn to adapt task-loss balancing weights during training to improve performance of multiple tasks on real-world datasets. Advantageously, learning to dynamically balance weights between different task losses can lead to higher performance than using static weights determined by expensive random search or heuristics. Embodiments of the meta-learning method can be used for computer vision tasks or natural language processing tasks, and the trained neural network can be used by augmented or virtual reality devices.

[0007] Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. Neither this summary nor the following detailed description is intended to define or restrict the scope of the subject matter of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figures 1A-1 to 1C-2 Results are shown for training multiple tasks in a deep neural network in different orders on the NYUv2 dataset. Tasks 1, 2, and 3 are semantic segmentation, depth prediction, and surface normal estimation, respectively. Figure 1A-1 、 1B-1 and 1C-1 show examples of the balancing weights (alpha 1, alpha 2, and alpha 3) of the corresponding tasks during training. Figure 1A-2 、 1B-2 Figures 1C-2 and 1C-2 show examples of the total training loss (top) and the test-training loss (bottom).

[0009] Figure 2 is a block diagram of an example meta-learning system including a meta-network and child networks.

[0010] Figures 3A to 3F An example of numerical results is shown that compares the performance of an embodiment of a meta-learning meta-network with the performance of an embodiment of grid search and equal-weighted multi-task learning. The three tasks are computer vision tasks for scene understanding: segmentation, depth, and surface normals. Figure 3A An example meta-learning trajectory is shown, Figure 3B shows an example of converged task weights for three tasks, and Figures 3C-3F Shows the overall task ( Figure 3C ) and examples of training loss (top) and test loss (bottom) for each individual task: Split ( Figure 3D ),depth( Figure 3E ) and surface normal ( Figure 3F ).

[0011] Figure 4 is a flowchart of an example process of meta-learning for multi-task learning.

[0012] Figure 5 Schematically illustrated are examples of wearable display systems that can implement embodiments of multi-task networks trained using the meta-learning techniques described herein.

[0013] Reference numerals may be repeated throughout the drawings to indicate corresponding relationships between reference elements.The drawings are provided to illustrate example embodiments described herein and are not intended to limit the scope of the present disclosure. DETAILED DESCRIPTION

[0014] Overview

[0015] Learning to learn, or meta-learning, may have substantial advantages for training DNNs. Training a DNN for a specific task (e.g., semantic segmentation) can be accomplished through careful design of the network architecture. However, training a single DNN to solve multiple difficult problems poses challenges. For example, balancing learning objectives when there are multiple objectives is one such challenge. This multi-objective problem can become even more difficult when each objective has different properties (e.g., cross entropy, squared loss, cosine loss, etc.) and has varying scales.

[0016] In some cases, if the network has been trained with more than one objective function, the network may be performing multi-task learning. Implicit multi-task learning can be performed in convolutional neural networks (CNNs) such as Fast / Faster R-CNN by jointly optimizing region proposals and the final detection goal. Single-shot object detector methods such as YOLO / SSD (Look Only Once / Single Shot Multi-box Detection) further eliminate the region proposal structure by optimizing the anchor classification loss and directly predict the location and bounding box output in one forward pass. These types of models may minimize more than one loss function at the same time even if the final goal is only a single task.

[0017] Some embodiments of implicit or explicit multi-task learning require manual tuning or grid searching for suboptimal balance weights (ratios) between different tasks during training. Grid searching can involve performing several (often expensive) training iterations and scales poorly as the number of tasks increases. Furthermore, fixing the balance weights throughout training often leads to suboptimal training, as some tasks may need to be trained earlier than others.

[0018] Therefore, the present disclosure provides various embodiments of meta-learning methods to automatically learn multiple tasks using a single deep network. The learning goal can be to improve or maximize the learning efficiency of all tasks and their respective performance. Numerical experiments show that the test performance of different tasks may be affected by the order in which the tasks are learned (e.g., see Figure 1A-1 C), it may be necessary to dynamically balance multiple objectives throughout the learning iterations to improve or optimize performance. Meta-learning embodiments can be successfully utilized for loss balancing, and some such embodiments can outperform statically balancing losses using heuristics or grid search.

[0019] Some embodiments disclosed herein train a meta-network to discover a task-level curriculum for jointly training multiple tasks using a single network. Some embodiments involve training a meta-network that balances multiple objectives for training a multi-task network. Some embodiments of the meta-network learn balancing weights in such a way that some tasks are learned in a task-level curriculum prior to others.

[0020] In some embodiments, the meta-network predicts a trade-off between the losses for each training iteration. By observing the trajectories of these losses, the meta-network can learn to discover a task-level curriculum for jointly training multiple tasks using a single network. The task-level curriculum can correspond to inter-task learning dynamics, such as that some tasks (e.g., those that are likely to be more stable / unchanging) should be learned earlier than other tasks (e.g., those that are likely to be more unstable / changing).

[0021] Some embodiments provide learning schedules for different tasks that are difficult or resource-intensive when using a grid search approach based on human intuition. Meta-learning methods can be applied to many real-world datasets, including datasets for computer vision, speech recognition, or natural language processing. For example, some applications described herein are for computer vision tasks such as scene understanding in room layout (e.g., semantic segmentation, depth prediction, or surface normal estimation).

[0022] Numerical experiments jointly training semantic segmentation, depth prediction, and surface normal estimation in an exemplary meta-network show that the meta-network first learns depth, then gradually focuses on learning surface normals, and finally learns semantic segmentation. This learning sequence discovered by the meta-network follows a pattern found during the development of the infant visual cortex system.

[0023] Applicants have discovered that, in some embodiments, task-level curricula can be advantageous for multi-task learning. Embodiments of a meta-network are able to capture this multi-task dynamics in a meta-learning framework and determine non-trivial learning schedules for different tasks that cannot be found using human intuition or exhaustive grid search methods.

[0024] Task-level course learning

[0025] One approach for training deep neural networks for multiple tasks is to create manually designed algorithms / network architectures to automatically learn a set of tasks. However, adjustments to algorithms based on human experience or manual design may not be optimal for certain tasks. An example of a manually designed learning algorithm is curriculum learning. Instead of training examples in a random order, curriculum learning focuses on re-weighting the training examples using a meaningful order based on a manually designed difficulty metric (e.g., the order trains progressively more examples and progressively more complex examples). Continuous methods include optimization strategies for dealing with minimizing non-convex criteria. These methods aim to optimize the model by first using easy training examples and then gradually moving on to difficult training examples. A basic paradigm is to first optimize a smooth objective and gradually consider reducing the smoothness, with the intuition that the smoothed version of the problem will reveal the global situation.

[0026] Another example training method, which can be called task-level curriculum learning, deals with learning the sequence of tasks in a multi-task neural network. These types of methods can use easier tasks to pre-train a sub-network, and can train harder tasks on top of the pre-trained sub-network.

[0027] Here, systems and methods are described for training a meta-network to predict a task-level learning curriculum. In some embodiments, the meta-network predicts a trade-off between losses for each training iteration. By observing the trajectories of these losses, the meta-network can learn to discover a task-level curriculum for jointly training multiple tasks using a single network. The task-level curriculum can correspond to inter-task learning dynamics, e.g., certain tasks (e.g., tasks that are likely to be more stable / unchanging) should be learned earlier than other tasks (e.g., tasks that are likely to be more unstable / changing).

[0028] Training multiple tasks with one network via grid search

[0029] One technique that can be used to solve multi-task learning is to find the optimal balancing weights α for K different tasks by using grid search k Once the weight α is chosen k After that, the weights can remain constant throughout the training process. In some embodiments, the meta-network can first determine whether the order of the tasks being trained is important. To address this issue, ablation studies are performed by rearranging the order of the tasks being trained in a single network to determine how the training order affects learning.

[0030] Figure 1A-1C shows example observations obtained by training a single deep network on multiple tasks in different orders. The multiple tasks learned by the deep network are labeled tasks 1, 2, and 3, and involve semantic segmentation, depth prediction, and surface normal estimation, respectively. The DNN is trained on the NYUv2 dataset (described below).

[0031] Figure 1A-1 The balance weights alpha 1 (α1) and alpha 2 (α2) corresponding to tasks 1 and 2 during training are shown. Figure 1A-2 Shows the use Figure 1A-1 The balance weights shown are derived from the total training loss and test loss for training tasks 1 and 2. Figure 1B-1 The balance weights alpha 2 (α2) and alpha 3 (α3) corresponding to tasks 2 and 3 during training are shown. Figure 1B-2 Shows the use Figure 1B-1 The balanced weights shown are from the total training and test losses for training tasks 2 and 3. Figure 1C-1 The balance weights alpha 1 (α1) and alpha 3 (α3) corresponding to tasks 1 and 3 during training are shown. Figure 1C-2 Shows the use Figure 1C-1 The balance weights shown are from the total training and test losses for training tasks 1 and 3.

[0032] exist Figure 1A-1 In C, the symbol "task i -> task j" indicates that the training starts with a high task i weight α i Assign a high value to task i, and then gradually change to a high task j weight α j At each training step, the task weights are constrained to sum to one. Figure 1A-1 、 1B-1 and 1C-1 show the weight changes according to the training steps taken during training.

[0033] like Figure 1A-1 and 1A-2 As shown in Figure 2, the network converges to roughly the same training loss value when training from task 1 (segmentation) to task 2 (depth) or vice versa. However, the test loss reveals a very different behavior: training from task 1 produces a much lower overall test loss. This task order-specific network training behavior can also be seen in Figure 1B-2 and 1C-2 Note that in these experiments, this behavior occurs for the same amount of training time, and the task weights sum to one at each training step. In other words, each task receives the same amount of training time.

[0034] In short, Figure 1A-1C demonstrates the difference in examples between training with different task emphases. For example, for Figure 1A-2 、 1B-2 As in each of the graphs 1C-2, the networks converge to roughly the same training loss value. However, even with the same training time and balanced weights summing to one, the test loss exhibits very different behaviors. Therefore, the test performance of different tasks may be affected by the order in which the tasks are learned during training.

[0035] Multi-task learning with examples via meta-learning

[0036] In various embodiments, there are K different tasks and their associated weights α k The loss function L k Multi-task learning is performed in the form of a weighted linear sum of:

[0037]

[0038] This exemplary multi-task network is labeled as a child network as an introduction to how to use a meta network to learn such a child network in a meta-learning framework.

[0039] In some embodiments of meta-learning, the goal is to perform all K tasks as well as possible on the training set χ. This goal can be represented by the meta-network objective function with its parameters φ To capture the optimal sub-network parameters It can be a function of the meta-network parameters φ and the loss function in question. Given a loss function distribution, we can formulate the expected loss as:

[0040]

[0041] in, is the expectation operator of the sub-loss function.

[0042] Given the current loss value of the sub-network and the parameters φ of the meta-network, the meta-network m generates adaptive weights α. The sub-network can then use this updated set of α to Compute the updated sub-network parameters θ. In some embodiments, a recurrent model such as a long short-term memory (LSTM) module can be used as the meta-network m with hidden states denoted by h. The output of equation (2) provides the actual learning signal at the end of the sub-network's training session; therefore, for a training window of T steps, it may be convenient to have an objective that depends on the optimization trajectory,

[0043]

[0044] in

[0045] θt+1 =θ t +g t

[0046]

[0047]

[0048] Given the latest learning trajectory and LSTM hidden state h t , at each training iteration t, the meta-network m generates a new set of balanced weights α k，t .

[0049] In some embodiments, gradient descent on φ may be used to reduce or minimize the meta-network loss function Instead of searching for the optimal α through human intuition or heuristic methods k , some embodiments may learn the loss in a data-driven manner with respect to φ The meta-network may consume many optimization curves and learn to adjust its parameters to dynamically produce the optimal relationship coefficient α between all tasks. This approach is different from directly optimizing the weight α. method, because the objective function It does not have access to the history of the training trajectory and can only let the optimizer find the best weight α based on the current target value.

[0050] Figure 2 is a block diagram of an example meta-learning system 200 including a meta-network 204 and a sub-network 208. As described with reference to equations (1) to (4), the meta-network loss function It can depend on the parameters φ of the meta-network and the sub-network loss function α may depend on the parameters θ of the sub-network, which may optionally depend on the training iteration t. The value of the meta-network loss function may be reduced or minimized to provide a set of adaptive weights for the sub-network. If the sub-network comprises a multi-task network, the adaptive weights may be weights for each task α. In some embodiments, the meta-network may include hidden states 212. The meta-network 204 may include a recurrent model, such as a long short-term memory (LSTM) module. Thus, the sub-network may utilize the updated set of adaptive weights from its loss function Calculate the updated sub-network parameters θ. As described above, the meta-learning system 200 can iteratively optimize the trajectory to minimize the loss function.

[0051] Example meta-learning problem and meta-network

[0052] One of the most common and natural multi-task learning problems in computer vision is to capture scene geometry and recognize objects in the scene. Object recognition has a wide range of applications, such as real-time three-dimensional (3D) reconstruction for augmented reality and inferring supporting relationships between objects for autonomous driving.

[0053] The embodiment of the meta-learning technique described with respect to equations (1) to (4) is applied to the real-world scene understanding dataset NYUv2. The NYUv2 dataset consists of 1449 densely labeled aligned color (RGB) and depth image pairs of indoor scenes with a resolution of 640x480. The sub-network and meta-network are trained on 795 training examples, and the test results are calculated on 654 separate test examples. In the numerical experiments, the images are downsampled to a resolution of 128x128 to reduce the computational burden.

[0054] The NYUv2 dataset has three task suites: semantic segmentation, depth prediction, and surface normal estimation, which are good candidates for multi-task meta-learning scenarios.

[0055] Semantic segmentation involves understanding images at the pixel level. Meta-learning was evaluated on a set of semantic classes with 13 semantic labels described in the NYUv2 dataset. A cross-entropy loss function was used for semantic segmentation to learn pixel-wise classification probabilities, and the test set accuracy was reported using the average of the intersection over union (mIoU) metric.

[0056] For depth prediction, the L1 loss function is used to perform pixel-wise supervised depth regression training for depth prediction. In the numerical experiments, the depth is measured in meters (from 0 to 10 meters) for training and testing, where the ground truth depth is captured from an RGBD sensor.

[0057] For surface normal training, the pixel-wise cosine distance (e.g., 1-|cosθ|) is used, where θ measures the angle between the ground truth surface normal vector and the predicted surface normal vector. Each surface normal vector consists of three components representing the x, y, and z directions. The NYUv2 toolbox is used to generate the surface normal ground truth based on the depth ground truth and camera parameters.

[0058] In the numerical experiments, the meta-network architecture uses two layers of LSTM, each with 20 hidden units. The output of the meta-network is passed through a softmax function to ensure that the generated weights α kThe sum is unity and greater than zero. The meta-network is trained by minimizing Equation (3) using the ADAM optimization algorithm (Adaptive Moment Estimation) with a learning rate of 0.001 using backpropagation through time (BPTT). In the experiments, gradient clipping with a standard value of 1 is used to improve training. The method is found to be relatively insensitive to the number of hidden units (20), which is kept constant throughout the experiments.

[0059] In the experiments, the multi-task subnetwork is a deep fully convolutional network consisting of six 3×3 convolutional layers with stride 1, except for the first two convolutional layers which have stride 2. Each convolutional layer has 16 filters, where the nonlinearity is a rectified linear unit (ReLU) activation followed by batch normalization. At each training step, the meta-network generates a new set of task-balanced weights α for all three trained tasks. k The subnetwork uses these adaptive weights to compute the gradient with respect to its own subnetwork parameters θ. The subnetwork is trained using standard stochastic gradient descent with a momentum of 0.9, a learning rate of 0.01, and a batch size of 10. All parameters are initialized using an iid Gaussian distribution.

[0060] The results from the meta-learned meta-network are compared with two commonly used multi-task learning techniques: equal weighting and grid search. The equal weighting technique uses the same α value of 0.333 for all three tasks. For grid search, for each α k , using a step size of 0.02 from the search space with the constraint that the weights sum to 1 and are all positive. Figures 3A-3F and Table 1 summarizes the best results from 1200 grid search runs.

[0061] Figure 3A Examples of learning trajectories showing how an example meta-network learns a set of three tasks: semantic segmentation (Task 1), depth detection (Task 2), and surface normal estimation (Task 3). Figure 3A The learning trajectory is shown to converge to a certain loss value for 0, 100, 200, 300, and 400 training iterations. Figure 3B The final converged task weight values α for three tasks of one learning trajectory for an example meta-network are shown.

[0062] Figures 3C-3F Shown is a comparison between multi-task learning controlled by equally weighted, grid search and an embodiment of a meta-learning meta-network on the NYUv2 dataset for learning how to learn semantic segmentation (task 1), depth detection (task 2), and surface normal estimation (task 3). Figure 3C The overall training loss and test loss for all three tasks are shown. Figure 3D 、 3E, 3F shows the training loss and test loss for each individual task: split ( Figure 3D ),depth( Figure 3E ) and surface normal ( Figure 3F ).

[0063] The meta-network embodiment dynamically adjusts the balancing weight α for different tasks throughout the training process and leads to better convergence speed and lower test loss compared to equal-weighted and grid search methods.

[0064] Table 1 shows a summary of experimental results comparing the meta-learning techniques described in this paper with equal weighting and grid search. Table 1 also shows a comparison with a recently developed uncertainty weighting method, which computes the homoscedastic uncertainty of the output of each task based on the Gaussian likelihood assumption and jointly learns the uncertainty-task balancing weights with the original task objectives.

[0065] Table 1

[0066]

[0067]

[0068] As can be seen from Table 1, when using embodiments of meta-learning multi-task learning techniques (e.g., see Equations (1) to (4)) to train on single-task models, weighted losses, grid search, and uncertainty weighting, training performance can be improved.

[0069] Example meta-learning process for multi-task learning

[0070] Figure 4 800 is a flow chart of an example process 800 for training a subnetwork using meta-learning and a meta-network. The subnetwork may include a multi-task network for determining outputs associated with multiple tasks (e.g., scene understanding). The subnetwork may include multiple shared layers and an output layer including multiple task-specific filters. A computing system, such as a computing system having non-transitory memory and a hardware processor, may perform process 800 to train the subnetwork. The non-transitory memory of the computing system may store or be configured to store executable instructions. The hardware processor may communicate with the non-transitory memory and be programmed with the executable instructions to perform process 800 to train the subnetwork using an embodiment of meta-learning.

[0071] At block 804, process 800 accesses training data. The training data may include training images associated with a plurality of reference task outputs of the subnetwork. The plurality of tasks may include regression tasks, classification tasks, or a combination thereof. The plurality of tasks may include, for example, perception tasks such as face recognition, visual search, gesture identification or recognition, scene understanding, semantic segmentation, depth estimation, surface normal estimation, object detection, room layout estimation, cuboid detection, lighting detection, simultaneous localization and mapping, object or avatar relocalization, or speech processing tasks (such as speech recognition or natural language processing), or a combination thereof.

[0072] At block 808, process 800 accesses a sub-network. The sub-network may be a multi-task neural network. The sub-network may be associated with a sub-network loss function, which may depend on the parameters θ of the sub-network. For example, the sub-network loss function Can include losses for K different tasks and its associated weight α k The sub-network parameters θ may depend on the training iteration time t.

[0073] At block 812, process 800 accesses the meta-network. The meta-network can be compared to the meta-network loss function Related, network loss function The association may depend on a parameter φ of the meta-network.In various embodiments, the meta-network loss function may be formulated as described above with reference to equation (2) or (3).

[0074] At block 816, process 800 trains the meta-network to provide adaptive weights for the sub-loss functions. The adaptive weights may be weights α associated with the tasks of the multi-task sub-network. k . Given the current loss value of the sub-network and the parameters φ of the meta-network, the trained meta-network thereby produces adaptive weights α. The meta-network may include a recursive model, such as a long short-term memory (LSTM) module. The meta-network may be trained using an embodiment of the ADAM optimization algorithm (e.g., adaptive moment estimation) by stochastic gradient descent or backpropagation through time (BPTT). Process 800 may train the meta-network based on the optimization trajectory of a T-step training window using equations (3) and (4). In some such embodiments, at each training iteration t, given the sub-network The latest learning trajectory and the hidden state h of the meta-network t , the meta-network produces a new set of balanced weights.

[0075] At block 820, process 800 trains the sub-network based on the adaptive weights learned at block 816. For example, the sub-network can use these adaptive weights to compute the gradient of its own sub-network parameters θ, such as by stochastic gradient descent. Blocks 816 and 820 can be iterated until sufficient convergence is achieved.

[0076] At block 824, process 800 outputs the trained sub-network. The trained sub-network can be stored in a memory of an AR, VR, or MR display device and used to perform tasks associated with the sub-network (see, for example, reference to FIG. Figure 5 description of the wearable system described herein).

[0077] Example NN layer

[0078] A model that represents data relationships and patterns, such as a function, algorithm, system, etc., can accept input and produce output corresponding to the input in some way. For example, the model can be implemented as a machine learning method, such as a convolutional neural network (CNN) or a deep neural network (DNN). Deep learning is part of a broader family of machine learning methods that are based on the idea of learning data representations rather than task-specific methods, and has shown promise in solving audio-visual computing problems that are useful for augmented reality (AR), mixed reality (MR), virtual reality (VR), and machine intelligence. In machine learning, convolutional neural networks (CNN or ConvNet) can include a class of deep and feed-forward artificial neural networks, and CNNs have been successfully applied to analyze visual images. Machine learning methods include a range of methods that can provide robust and accurate solutions to a variety of problems, including eye image segmentation or eye tracking.

[0079] A layer of a neural network (NN), such as a deep neural network (DNN), can apply a linear or nonlinear transformation to its input to generate its output. A deep neural network layer can be a normalization layer, a convolutional layer, a soft-sign layer, a rectified linear layer, a cascade layer, a pooling layer, a recursive layer, an inception layer, or any combination thereof. A normalization layer can normalize the brightness of its input to generate its output, for example, by L2 normalization. A normalization layer can, for example, normalize the brightness of multiple images relative to each other at once to generate multiple normalized images as its output. Non-limiting examples of methods for normalizing brightness include local contrast normalization (LCN) or local response normalization (LRN). Local contrast normalization can nonlinearly normalize the contrast of an image by normalizing a local region of an image on a per-pixel basis to have a mean of zero and a variance of 1 (or other values of the mean and variance). Local response normalization can normalize an image over a local input region to have a mean of zero and a variance of 1 (or other values of the mean and variance). Normalization layers can speed up the training process.

[0080] A convolutional layer may apply a set of kernels that convolve its input to generate its output. A soft sign layer may apply a soft sign function to its input. The soft sign function (softsign(x)) may be, for example, (x / (1+|x|)). The soft sign layer may ignore the effects of per-element outliers. A rectified linear layer may be a rectified linear unit (ReLU) or a parameterized rectified linear unit (PReLU). A ReLU layer may apply a ReLU function to its input to generate its output. The ReLU function ReLU(x) may be, for example, max(0,x). A PReLU layer may apply a PReLU function to its input to generate its output. The PReLU function PReLU(x) may be, for example, x (if x≥0) and ax (if x<0), where a is a positive number. A cascade layer may concatenate its input to generate its output. For example, a cascade layer may concatenate four 5×5 images to generate a 20×20 image. A pooling layer may apply a pooling function that downsamples its input to generate its output. For example, a pooling layer may downsample a 20x20 image to a 10x10 image. Non-limiting examples of pooling functions include max pooling, average pooling, or min pooling.

[0081] At time point t, the recursive layer can calculate the hidden state s(t), and the recursive connection can provide the hidden state s(t) at time t to the recursive layer as input at a subsequent time point t+1. The recursive layer can calculate the output at time t+1 based on the hidden state s(t) at time t. For example, the recursive layer can apply a soft sign function to the hidden state s(t) at time t to calculate its output at time t+1. The hidden state of the recursive layer at time t+1 takes the hidden state s(t) of the recursive layer at time t+1 as its input. The recursive layer can calculate the hidden state s(t+1) by, for example, applying a ReLU function to its input. The class initialization layer may include one or more of a normalization layer, a convolutional layer, a soft sign layer, a rectified linear layer such as a ReLU layer and a PReLU layer, a cascade layer, a pooling layer, or any combination thereof.

[0082] In different embodiments, the number of layers in a NN can vary. For example, the number of layers in a DNN can be 50, 100, 200, or more. In different embodiments, the input types of a deep neural network layer can vary. For example, a layer can receive the outputs of multiple layers as its input. The input of a layer can include the outputs of five layers. As another example, the input of a layer can include 1% of the NN layer. The output of a layer can be the input of multiple layers. For example, the output of a layer can be used as the input of five layers. As another example, the output of a layer can be used as the input of 1% of the NN layer.

[0083] The input or output size of a layer can be very large. The input or output size of a layer can be n×m, where n is the width and m is the height of the input or output. For example, n or m can be 11, 21, 31, or larger. In different embodiments, the channel size of the input or output of a layer can vary. For example, the channel size of the input or output of a layer can be 4, 16, 32, 64, 128, or larger. In different embodiments, the kernel size of a layer can vary. For example, the kernel size can be n×m, where n is the width of the kernel and m is the height of the kernel. For example, n or m can be 5, 7, 9, or larger. In different embodiments, the stride size of a layer can vary. For example, a deep neural network layer can have a stride size of 3, 5, 7, or larger.

[0084] In some embodiments, NN may refer to a plurality of NNs that together compute the output of the NN. Different NNs in the plurality of NNs may be trained for different tasks. A processor (e.g., referring to Figure 5 The processor of the local data processing module 924 described above can calculate the output of the NN in the plurality of NNs to determine the output of the NN. For example, the output of the NN in the plurality of NNs can include a likelihood score. The processor can determine the output of the NN including the plurality of NNs based on the likelihood scores of the outputs of different NNs in the plurality of NNs.

[0085] Example Wearable Display System

[0086] In some embodiments, the user device may be or may be included in a wearable display device, which may advantageously provide a more immersive virtual reality (VR), augmented reality (AR), or mixed reality (MR) experience in which a digitally reproduced image, or portion thereof, is presented to the wearer in a manner that appears or can be perceived as real.

[0087] Without being limited by theory, it is believed that the human eye can generally interpret a finite number of depth planes to provide depth perception. Therefore, a highly convincing simulation of perceived depth can be achieved by providing the eye with a different presentation of an image corresponding to each of these finite number of depth planes. For example, a display comprising a stack of waveguides can be configured to be worn in front of the eyes of a user or viewer. By using multiple waveguides to direct light from an image injection device (e.g., the output of a discrete display or a multiplexed display that transmits image information through one or more fiber optic conduits) to the viewer's eyes at specific angles (and divergences) corresponding to the depth planes associated with specific waveguides, a stack of waveguides can be used to provide the eye / brain with a three-dimensional perception.

[0088] In some embodiments, two stacks of waveguides can be utilized to provide a different image to each eye, one stack for each eye of the viewer. As an example, an augmented reality scene can cause a wearer of AR technology to see a real-world park-like setting featuring people, trees, buildings in the background, and a concrete platform. In addition to these objects, the wearer of AR technology may also perceive that he "sees" a robotic statue standing on the real-world platform, and that a cartoon-like avatar character flying past him appears to be a personification of a bumblebee, even though the robotic statue and bumblebee do not exist in the real world. The stack of waveguides can be used to generate a light field corresponding to the input image, and in some embodiments, the wearable display includes a wearable light field display. Examples of wearable display devices and waveguide stacks for providing light field images are described in U.S. Patent Publication No. 2015 / 0016777, the entire contents of which are incorporated herein by reference.

[0089] Figure 5An example of a wearable display system 900 is shown, which can be used to present a VR, AR, or MR experience to a display system wearer or viewer 904. The wearable display system 900 can be programmed to perform any application or embodiment described herein (e.g., executing a CNN or DNN, reordering the values of an input activation map or kernel, eye image segmentation, or eye tracking). The display system 900 includes a display 908 and various mechanical and electronic modules and systems that support the functionality of the display 908. The display 908 can be coupled to a frame 912, which can be worn by the display system wearer or viewer 904 and is configured to position the display 908 in front of the wearer's 904 eyes. The display 908 can be a light field display. In some embodiments, a speaker 916 is coupled to the frame 912 and positioned near the user's ear canal, and another speaker (not shown) is positioned near the user's other ear canal to provide stereo sound / shapeable sound control. The display system 900 can include an outward-facing imaging system 944 (e.g., one or more cameras) that can capture images (e.g., still images or video) of the environment surrounding the wearer 904. The images obtained by the externally facing imaging system 944 can be referenced in the environment surrounding the wearer 904 Figure 2 and Figure 4 An embodiment of a meta-learning method 900 for training a multi-task network is described for analysis.

[0090] The display 908 can be operably coupled 920 to a local data processing module 924, for example, via a wired lead or a wireless connection, which can be mounted in various configurations, such as fixedly attached to the frame 912, fixedly attached to a helmet or hat worn by the user, embedded in headphones, or otherwise removably attached to the user 904 (e.g., in a backpack configuration, a belt-coupled configuration).

[0091] The local processing and data module 924 can include a hardware processor and non-transitory digital memory (e.g., non-volatile memory (e.g., flash memory)), both of which can be used to assist in processing, caching, and storing data. The data can include: (a) data captured from sensors (which can, for example, be operably coupled to the frame 912 or otherwise attached to the wearer 904), such as image capture devices (e.g., cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, radios, and / or gyroscopes; and / or (b) data acquired and / or processed using the remote processing module 928 and / or remote data repository 932, possibly after such processing or retrieval, and transferred to the display 908. The local processing and data module 924 can be operably coupled to the remote processing module 928 and the remote data repository 932 via communication links 936, 940 (e.g., via wired or wireless communication links), such that these remote modules 928, 932 are operably coupled to each other and can serve as resources for the local processing and data module 924. The image capture device may be used to capture eye images for use in eye image segmentation or eye tracking processes.

[0092] In some embodiments, the remote processing module 928 may include one or more processors configured to analyze and process data and / or image information, such as video information captured by an image capture device. Video data may be stored locally in the local processing and data module 924 and / or in a remote data repository 932. In some embodiments, the remote data repository 932 may include a digital data storage facility that may be accessed via the Internet or other network configurations in a "cloud" resource configuration. In some embodiments, all data is stored and all computations are performed in the local processing and data module 924, allowing for fully autonomous use from the remote module.

[0093] In some embodiments, the local processing and data module 924 and / or the remote processing module 928 are programmed to perform embodiments of reordering the values of the input activation map or kernel, eye image segmentation, or eye tracking as disclosed herein. For example, the local processing and data module 924 and / or the remote processing module 928 can be programmed to use a reference Figure 2 and 4Embodiments of a multi-task network trained by method 90 for performing task prediction and determination are described. An image capture device may capture video of a particular application (e.g., augmented reality (AR), human-computer interaction (HCI), self-driving cars, drones, or robotics in general). The video may be analyzed using a CNN by one or both of processing modules 924, 928. In some cases, reordering the values of the input activation maps or kernels, offloading at least some of the eye image segmentation or eye tracking to a remote processing module (e.g., in the "cloud") may improve computational efficiency or speed. Parameters of the CNN or DNN (e.g., weights, bias terms, subsampling factors for pooling layers, the number and size of kernels in different layers, the number of feature maps, etc.) may be stored in data modules 924 and / or 932. For example, data modules 924, 932 may store parameters that have been learned from training using the meta-learning and meta-network described herein.

[0094] The result of the task prediction or determination (e.g., reference Figure 2 and Figure 4 The output of the meta-learning multi-task network 900 described above can be used for additional operations or processing by one or both of the processing modules 924, 928. For example, the processing modules 924, 928 of the wearable display system 900 can be programmed to execute additional applications (such as applications in augmented reality, human-computer interaction (HCI), self-driving cars, drones, or general robotics) based on the output of the multi-task network.

[0095] Additional tasks

[0096] Embodiments of meta-learning are not limited to computer vision tasks, where the meta-network is trained on images or visual data. In other embodiments, the training samples may include non-image data captured by sensors, such as audio data, acceleration data, positioning data, temperature data, radio frequency data, and optical tracking data. Examples of sensors include audio sensors (e.g., microphones), inertial measurement units (IMUs), accelerometers, compasses, gyroscopes, temperature sensors, motion sensors, depth sensors, global positioning system (GPS) units, and radios. In other embodiments, the training samples may include measurements such as gender, age, heart rate, body temperature, white blood cell count, disease state, disease progression, symptoms, disease diagnosis, etc. For example, for tasks related to speech recognition or natural language processing, the training samples may include audio datasets (or audio data accompanying videos) or electronic representations or embeddings (e.g., n-grams) of words, sentences, paragraphs, and text. Tasks may include, for example, part-of-speech (POS) tagging, chunk dependency parsing, semantic relevance, or text meaning.

[0097] The tasks that can be learned by the multi-network may include regression tasks, classification tasks, or a combination thereof. The tasks may include, for example, perception tasks such as face recognition, visual search, gesture identification or recognition, semantic segmentation, object detection, room layout estimation, cuboid detection, lighting detection, simultaneous localization and mapping, relocalization of objects or avatars, or speech processing tasks (such as speech recognition or natural language processing), or a combination thereof.

[0098] The tasks that can be learned by multiple networks can be related to medical treatment or medical decision-making. For example, the task can include determining which laboratory tests or risk assessments should be performed on patients with a particular disease. For example, the training data can include measurements (e.g., gender, age, heart rate, body temperature, white blood cell count, etc.) of patients with a particular disease or condition (e.g., pneumonia). Embodiments of meta-learning can be used to train a multi-task network for predicting a patient's risk assessment or laboratory test results.

[0099] Therefore, the training data used for meta-learning can include images, videos, audio, electronic records, databases, etc., which can be associated with corresponding training labels, classifications or signals.

[0100] In some embodiments, the meta-learning methods disclosed herein may have applications in computer vision, natural language processing, speech synthesis, domain-specific applications (e.g., traffic prediction), general cross-domain applications, or curriculum learning. In some embodiments, tasks are jointly trained based on a global reward such as a reduction in total loss. Embodiments of meta-learning can be applied to training multi-task neural networks for augmented reality, mixed reality, or virtual reality (see, e.g., reference Figure 5 (describes an augmented reality system that can store and utilize such meta-learned trained neural networks for AR-related tasks such as computer vision or language processing).

[0101] Additional aspects

[0102] In a first aspect, a system for training a multi-task neural network using meta-learning is disclosed. The system includes: a non-transitory storage device configured to store: a multi-task neural network configured to determine outputs associated with a plurality of tasks, each of the plurality of tasks being associated with a task loss function, the multi-task neural network associated with the multi-task loss function comprising a combination of task weights and task loss functions for each of the plurality of tasks; a meta-network configured to output task weights associated with each of the plurality of tasks, the meta-network associated with a meta-network loss function comprising an expected loss based at least in part on the multi-task loss function; a hardware processor in communication with the non-transitory storage device, the hardware processor being programmed to: access training data associated with a plurality of reference task outputs for the plurality of tasks; reduce or minimize the meta-network loss function to determine an estimated task weight associated with each of the plurality of tasks; reduce or minimize the multi-task loss function based at least in part on the estimated task weights; and output the trained multi-task neural network.

[0103] In a second aspect, the system according to aspect 1, wherein the tasks of the multi-task neural network are associated with scene understanding.

[0104] In a third aspect, the system of any one of aspects 1 or 2, wherein the task comprises at least one of semantic segmentation, depth, or surface normals.

[0105] In a 4th aspect, the system according to any one of aspects 1-3, wherein the multi-task loss function comprises a linear combination of task weights for all tasks in the plurality of tasks multiplied by the task loss function.

[0106] In a fifth aspect, the system of any one of aspects 1-4, wherein the meta-network comprises a long short-term memory (LSTM) module.

[0107] In a sixth aspect, the system of any one of aspects 1-5, wherein the meta-network loss function comprises the sum of the multi-task loss function over the optimization trajectory of the training window.

[0108] In a 7th aspect, the system of any of aspects 1-6, wherein to reduce or minimize the meta-network loss function, the hardware processor is programmed to apply backpropagation through time using adaptive moment estimation.

[0109] In an eighth aspect, the system of any one of aspects 1-7, wherein to reduce or minimize the multi-task loss function, the hardware processor is programmed to apply stochastic gradient descent.

[0110] In a 9th aspect, the system of any one of aspects 1-8, wherein the multi-task neural network comprises a convolutional neural network.

[0111] In aspect 10, the system of any one of aspects 1-9, wherein the hardware processor is further programmed to: monitor the values of the multi-task loss function and the meta-network loss function at each training time; and determine a task-level curriculum for jointly training the multiple tasks.

[0112] In an eleventh aspect, a method for training a sub-neural network using meta-learning and a meta-network is described. The method includes: accessing a meta-network configured to output adaptive weights for a sub-neural network; training the meta-network based at least in part on a sub-loss function for the sub-neural network, the sub-loss function depending at least in part on the adaptive weights; outputting updated adaptive weights via the meta-network; training the sub-neural network using the updated adaptive weights; and outputting the trained sub-network. The method can be executed under the control of a hardware processor. The hardware processor can communicate with a non-transitory memory configured to store the meta-network and the sub-neural network.

[0113] In a twelfth aspect, the method according to aspect 11 further comprises accessing a training dataset for training the meta-network and the sub-network.

[0114] In a 13th aspect, the method of any one of aspects 11 or 12, wherein the meta-network comprises a recurrent neural network.

[0115] In a 14th aspect, the method of aspect 13, wherein the recurrent neural network comprises a long short-term memory (LSTM) module.

[0116] In a 15th aspect, the method of any one of aspects 11-14, wherein the sub-neural network comprises a multi-task neural network.

[0117] In a 16th aspect, the method of aspect 15, wherein the multi-task neural network comprises a multi-task loss function comprising a combination of adaptive weights associated with respective tasks and a task loss function.

[0118] In a 17th aspect, the method according to any one of aspects 11-16 further comprises determining a task-level curriculum associated with jointly training the respective tasks.

[0119] In an 18th aspect, the method according to any one of aspects 11-17, wherein training the meta-network comprises monitoring a trajectory of the meta-network loss function over a training window.

[0120] In a 19th aspect, a head-mounted display system is described. The head-mounted display system includes: a non-transitory memory configured to store: executable instructions; and a multi-task network for determining outputs associated with a plurality of tasks, wherein the multi-task network is trained using: a meta-network configured to output adaptive weights for the plurality of tasks of the multi-task network; a multi-task loss function for the multi-task neural network, the multi-task loss function being at least partially dependent on the adaptive weights learned by the meta-network; a display; a sensor; and a hardware processor in communication with the non-transitory memory, the display, and the sensor, the hardware processor being programmed by the executable instructions to: receive sensor data captured by the sensor; determine a task output for each of the plurality of tasks using the multi-task network with the sensor data as input; and cause the display to display information related to the determined task output to a user of the head-mounted display system.

[0121] In a 20th aspect, in the system of aspect 19, wherein the plurality of tasks comprises a plurality of perception tasks.

[0122] In a 21st aspect, in the system of aspect 20, wherein the plurality of perception tasks comprises face recognition, visual search, gesture identification, semantic segmentation, depth estimation, surface normal estimation, scene recognition, object detection, lighting detection, simultaneous localization and mapping, relocalization, or a combination thereof.

[0123] In a 22nd aspect, the system of any of aspects 19-21, wherein the sensor comprises an inertial measurement unit, an external-facing camera, a depth-sensing camera, a microphone, an eye-imaging camera, or a combination thereof.

[0124] In a 23rd aspect, a system for training a neural network to learn a set of tasks is described. The system includes: a non-transitory memory configured to store: executable instructions; and a sub-network for learning a plurality of tasks, wherein the sub-network is associated with a loss function for the plurality of tasks and assigns a task weight to each of the plurality of tasks; a hardware processor in communication with the non-transitory memory, the hardware processor being programmed by the executable instructions to: determine a first sub-network loss associated with the loss function for the sub-network; determine an updated task weight for each of the plurality of tasks based on the first sub-network loss; determine an updated sub-network based on the updated task weight for each of the plurality of tasks; determine a second sub-network loss associated with the loss function for the updated sub-network; and determine a second updated task weight for each of the plurality of tasks based on at least the second sub-network loss.

[0125] In a 24th aspect, the system of aspect 23, wherein the hardware processor is further programmed to learn a relationship between the first sub-network loss and the second sub-network loss.

[0126] In a 25th aspect, the system of any of aspects 23 or 24, wherein the hardware processor is programmed to determine the second updated task weight based at least on the relationship between the first sub-network loss and the second sub-network loss.

[0127] In a 26th aspect, the system of any of aspects 23-25, wherein the plurality of tasks comprises at least semantic segmentation, depth prediction, and surface normal estimation.

[0128] In a 27th aspect, the system according to any one of aspects 23-26, wherein the plurality of tasks comprises regression tasks, classification tasks, or a combination thereof.

[0129] In a 28th aspect, the system of aspect 27, wherein the classification task comprises perception, facial recognition, visual search, gesture recognition, semantic segmentation, object detection, room layout estimation, cuboid detection, lighting detection, simultaneous localization and mapping, relocalization, speech processing, voice recognition, natural language processing, or a combination thereof

[0130] Additional Notes

[0131] Each process, method, and algorithm described herein and / or depicted in the accompanying drawings may be embodied in a code module comprised of one or more physical computing systems, hardware computer processors, dedicated circuits, and / or electronic hardware configured to execute specific and particular computer instructions and may be automated in whole or in part. For example, a computing system may include a general-purpose computer (e.g., a server) or a dedicated computer, dedicated circuit, or the like programmed with specific computer instructions. The code module may be compiled and linked into an executable program, installed in a dynamic link library, or may be written in an interpreted programming language. In some embodiments, specific operations and methods may be performed by circuits specific to a given function.

[0132] Furthermore, certain implementations of the functionality of the present disclosure are sufficiently mathematically, computationally, or technologically complex that specialized hardware or one or more physical computing devices (utilizing appropriate specialized executable instructions) may be required to perform the functionality, e.g., due to the amount or complexity of the computations involved, or to provide results in substantially real time. For example, a video may include many frames, each with millions of pixels, and specially programmed computer hardware may be required to process this video data to provide the desired image processing task or application in a commercially reasonable amount of time. As another example, training a deep multi-task network using embodiments of the meta-learning methods described herein is computationally challenging and may be implemented on a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or a floating-point gate array (FPGA).

[0133] Code modules or any type of data may be stored on any type of non-transitory computer-readable medium, such as physical computer storage devices including hard drives, solid-state memory, random access memory (RAM), read-only memory (ROM), optical disks, volatile or non-volatile storage devices, combinations thereof, and / or the like. Methods and modules (or data) may also be sent as generated data signals (e.g., as part of a carrier wave or other analog or digital propagation signal) over various computer-readable transmission media (including wireless-based and wired / cable-based media) and may take various forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). The results of the disclosed processes or process steps may be stored permanently or otherwise in any type of non-transitory tangible computer storage device, or may be communicated via computer-readable transmission media.

[0134] Any process, block, state, step or function in the flowcharts described herein and / or depicted in the accompanying drawings should be understood to potentially represent a code module, code segment or code portion comprising one or more executable instructions for implementing a specific function (such as logic or arithmetic) or step in the process. Various processes, blocks, states, steps or functions can be combined, rearranged, added, deleted, modified or otherwise changed with the illustrative examples provided herein. In some embodiments, additional or different computing systems or code modules can perform some or all of the functions described herein. The methods and processes described herein are also not limited to any particular order, and the blocks, steps or states associated therewith can be performed in other appropriate orders, such as serially, in parallel or in some other manner. Tasks or events can be added to or deleted from the disclosed example embodiments. In addition, the separation of the various system components in the embodiments described herein is for illustrative purposes and should not be understood as requiring such separation in all embodiments. It should be understood that the program components, methods and systems described can generally be integrated together in a single computer product or packaged into multiple computer products. Many variations of the embodiments are possible.

[0135] The processes, methods, and systems can be implemented in a network (or distributed) computing environment. Network environments include enterprise-wide computer networks, intranets, local area networks (LANs), wide area networks (WANs), personal area networks (PANs), cloud computing networks, crowd-sourced computing networks, the Internet, and the World Wide Web. The network can be a wired or wireless network or any other type of communication network.

[0136] The systems and methods of the present disclosure each have several innovative aspects, none of which is solely responsible for or requires the desired properties disclosed herein. The various features and processes described herein can be used independently of each other, or can be combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of the present disclosure. Various modifications to the embodiments described in this disclosure will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the claims are not intended to be limited to the embodiments shown herein, but should be given the widest scope consistent with the present disclosure, principles and novel features disclosed herein.

[0137] Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable subcombination. Moreover, although features may be described above as functioning in certain combinations and even initially claimed as such, in some cases one or more features in a claimed combination may be removed from that combination, and the claimed combination may be used in subcombinations or variations of the subcombinations. No single feature or set of features is essential or necessary for every embodiment.

[0138] Conditional language used herein, particularly words such as "may," "could," "might," "can," "for example," etc., unless expressly stated otherwise or understood otherwise in the context, is generally intended to convey that certain embodiments include and other embodiments do not include certain features, elements, and / or steps. Therefore, such conditional language is generally not intended to imply that features, elements, and / or steps are in any way necessary for one or more embodiments, or that one or more embodiments necessarily include logic for deciding (whether or not there is author input or prompting) whether these features, elements, and / or steps will be included or performed in any particular embodiment. The terms "include," "comprise," "have," etc. are synonyms and are used inclusively in an open-ended manner and do not exclude additional elements, features, actions, operations, etc. Similarly, the term "or" is used in its inclusive sense (rather than in its exclusive sense), so that, for example, when used to connect a list of elements, the term "or" means one, some, or all of the elements in the list. In addition, "a," "an," and "the" used in this application and the appended claims should be interpreted as meaning "one or more" or "at least one," unless otherwise stated.

[0139] As used herein, a phrase referring to "at least one" of a list of items refers to any combination of those items, including individual members. For example, "at least one of A, B, or C" is intended to encompass: A, B, C, A and B, A and C, B and C, and A, B, and C. Unless otherwise specifically stated, conjunctions such as the phrase "at least one of X, Y, and Z" should be understood in conjunction with the context that is typically used to convey that an item, term, etc. can be at least one of X, Y, or Z. Thus, such conjunctions are generally not intended to imply that certain embodiments require the presence of at least one of X, at least one of Y, and at least one of Z.

[0140] Similarly, although operations may be depicted in the figures in a particular order, it should be understood that such operations need not be performed in the particular order or sequence shown, or that all illustrated operations need not be performed to achieve the desired result. In addition, the figures may schematically depict one or more example processes in the form of flow charts. However, other operations not shown may be incorporated into the schematically illustrated example methods and processes. For example, one or more additional operations may be performed before, after, simultaneously with, or between any of the illustrated operations. In addition, in other embodiments, the operations may be rearranged or reordered. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the above-described embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged into multiple software products. In addition, other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result.

Claims

1. A system for training a multi-task neural network using meta-learning, the system comprising: A non-transitory storage device configured to store: a multi-task neural network configured to determine outputs associated with a plurality of tasks, each task in the plurality of tasks being associated with a task loss function, the multi-task neural network associated with the multi-task loss function comprising a combination of a task weight for each task in the plurality of tasks and the task loss function; a meta-network configured to output the task weight associated with each of the plurality of tasks, the meta-network being associated with a meta-network loss function including an expected loss based at least in part on the multi-task loss function; a hardware processor in communication with the non-transitory storage device, the hardware processor being programmed to: accessing training data associated with a plurality of reference task outputs for the plurality of tasks, wherein the training data comprises a plurality of digital images; reducing or minimizing the meta-network loss function to determine an estimated task weight associated with each task in the plurality of tasks; reducing or minimizing the multi-task loss function based at least in part on the estimated task weights; and Output of the trained multi-task neural network.

2. The system according to claim 1, wherein: The tasks of the multi-task neural network are associated with scene understanding.

3. The system according to claim 2, wherein: The tasks include at least one of semantic segmentation, depth, or surface normals.

4. The system according to claim 1, wherein: The multi-task loss function includes a linear combination of the task weights for all tasks in the plurality of tasks multiplied by the task loss function.

5. The system according to claim 1, wherein: The meta-network includes a long short-term memory (LSTM) module.

6. The system according to claim 1, wherein: The meta-network loss function includes the sum of the multi-task loss functions over the optimization trajectory of the training window.

7. The system according to claim 1, wherein: To reduce or minimize the meta-network loss function, the hardware processor is programmed to apply backpropagation through time using adaptive moment estimation.

8. The system according to claim 1, wherein: To reduce or minimize the multi-task loss function, the hardware processor is programmed to apply stochastic gradient descent.

9. The system according to claim 1, wherein: The multi-task neural network includes a convolutional neural network.

10. The system according to claim 1, wherein: The hardware processor is also programmed to: Monitoring the values of the multi-task loss function and the meta-network loss function at each training time; and A mission-level curriculum is determined for jointly training the plurality of missions.

11. A method for training a child neural network using meta-learning and a meta-network, the method comprising: accessing a meta-network configured to output adaptive weights for the child neural network; accessing a training dataset for training the meta-network and the sub-network, wherein the training dataset comprises a plurality of digital images; training the meta-network based at least in part on a sub-loss function for the sub-neural network, the sub-loss function depending at least in part on the adaptive weights; Outputting updated adaptive weights via the meta-network; training the sub-neural network using the updated adaptive weights; and Output the trained subnetwork.

12. The method according to claim 11, wherein The meta-network includes a recurrent neural network.

13. The method according to claim 12, wherein: The recurrent neural network includes a long short-term memory (LSTM) module.

14. The method according to claim 12, wherein: The sub-neural network includes a multi-task neural network.

15. The method according to claim 14, wherein The multi-task neural network includes a multi-task loss function, which includes a combination of the adaptive weights associated with each task and a task loss function.

16. The method according to claim 15, further comprising: A mission-level curriculum associated with each of the missions described in the joint training is determined.

17. The method according to claim 12, wherein: Training the meta-network includes monitoring a trajectory of a meta-network loss function over a training window.

18. A head-mounted display system comprising: Non-transitory memory configured to store: executable instructions, and A multi-task network for determining outputs associated with a plurality of tasks, wherein the multi-task network is trained using: a meta-network configured to output adaptive weights for a plurality of tasks of the multi-task network; a multi-task loss function for the multi-task network, the multi-task loss function depending at least in part on the adaptive weights learned by the meta-network; monitor; Sensors; and a hardware processor in communication with the non-transitory memory, the display, and the sensor, the hardware processor being programmed by the executable instructions to: receiving sensor data captured by the sensor; determining a task output for each of the plurality of tasks using the multi-task network having the sensor data as input; and The display is caused to display information related to the determined task output to a user of the head mounted display system.

19. The system according to claim 18, wherein: The plurality of tasks includes a plurality of perception tasks.

20. The system of claim 19, wherein: The plurality of perception tasks include face recognition, visual search, gesture identification, semantic segmentation, depth estimation, surface normal estimation, scene recognition, object detection, lighting detection, simultaneous localization and mapping, relocalization, or a combination thereof.

21. The system of claim 20, wherein: The sensor includes an inertial measurement unit, an external-facing camera, a depth-sensing camera, a microphone, an eye-imaging camera, or a combination thereof.

22. A system for training a neural network to learn a set of tasks, the system comprising: Non-transitory memory configured to store: executable instructions; as well as a subnetwork for learning a plurality of tasks, wherein the subnetwork is associated with a loss function for the plurality of tasks and a task weight is assigned to each of the plurality of tasks; a hardware processor in communication with the non-transitory memory, the hardware processor being programmed by the executable instructions to: accessing a training dataset for training the sub-network, wherein the training dataset comprises a plurality of digital images; determining a first sub-network loss associated with the loss function of the sub-network; determining, for each of the plurality of tasks, an updated task weight based on the first sub-network loss; determining an updated sub-network based on the updated task weight of each task in the plurality of tasks; determining a second sub-network loss associated with the loss function of the updated sub-network; and A second updated task weight is determined for each task in the plurality of tasks based on at least the second sub-network loss.

23. The system of claim 22, wherein: The hardware processor is further programmed to learn a relationship between the first sub-network loss and the second sub-network loss.

24. The system of claim 23, wherein: The hardware processor is programmed to determine the second updated task weight based on at least the relationship between the first sub-network loss and the second sub-network loss.

25. The system of claim 22, wherein: The multiple tasks include at least semantic segmentation, depth prediction, and surface normal estimation.

26. The system of claim 22, wherein: The multiple tasks include regression tasks, classification tasks, or a combination thereof.

27. The system of claim 26, wherein: The classification tasks include perception, facial recognition, visual search, gesture recognition, semantic segmentation, object detection, room layout estimation, cuboid detection, lighting detection, simultaneous localization and mapping, relocalization, speech processing, voice recognition, natural language processing, or a combination thereof.

Images