Generating task neural networks from larger neural networks using projections

Abstract

Methods, systems, and apparatuses, including computer programs encoded on computer storage media, for generating a task neural network for performing a specific task from a larger ("first") neural network. In particular, the described techniques include obtaining a task input specifying a task to be performed by the task neural network, processing the task input to generate, for each of a plurality of weight matrices of a first neural network, a set of projection matrices, and generating a task neural network with fewer parameters than the first neural network using the projection matrices. By using the described techniques, the system generates a task neural network without fine-tuning the first neural network that has comparable or better performance than the first neural network for the specialized task.

Description

[0001] Attorney Docket No.45288-0487WO1 GENERATING TASK NEURAL NETWORKS FROM LARGER NEURAL NETWORKS USING PROJECTIONS CROSS-REFERENCE TO RELATED APPLICATION This application claims priority to U.S. Application No.63 / 656,066, filed June 4, 2024, the disclosure of which is incorporated herein by reference. BACKGROUND This specification relates to processing inputs using neural networks. Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current value inputs of a respective set of parameters. SUMMARY This specification describes a system implemented as computer programs on one or more computers in one or more locations that generates a task neural network for performing a specific task from a larger neural network (also referred to as a “first” neural network). Generally, the task neural network is a smaller neural network (i.e., a neural network that has fewer parameters than the first neural network) that is specialized for performing the specific task. Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages. By training a neural network across a broad data distribution belonging to many modalities (e.g., images, audio, and natural language text) and tasks (e.g., image processing or generation tasks, audio processing or generation tasks, natural language text processing or generation task, and so on) the neural network can perform a variety of machine learning tasks (i.e., the trained neural network can function as a ‘foundation model neural network’). Although the generality of tasks the neural network can perform is advantageous, when only a narrow set of specialized tasks are of interest, the neural network’s ability to perform a generality of tasks introduces unnecessary complexity, resource use, and potential performance trade-offs for different tasks. For example, if the computational resources available in a deployment environment is insufficient for the neural network, the neural Attorney Docket No.45288-0487WO1 network cannot be deployed, rendering the neural network’s ability to perform a generality of tasks irrelevant due to the practical computational cost barriers to use the first neural network. In other words, the major challenges of using a large neural network that is a foundation model are that, in some cases, there is a need to perform only a specific task well (instead of a generality of tasks well) and it is too costly to use (with the acceptable cost being different for different contexts of use of the neural network). One technique to address the need to perform only a specific task well is to fine-tune the neural network to perform the task (i.e., further training the neural network on a subset of data relevant to specific task). Such techniques require storing a set of fine-tuned parameters for each task, which can be memory intensive for a large neural network. Another technique to address the need to reduce the cost of using the neural network is to train a smaller version of the neural network. Such a technique, for a collection of smaller neural networks of various sizes, would require independently training each smaller neural network from scratch and storing each respective set of learned parameters, which is costly to do in terms of time and memory. Another technique to address both the need to perform only a specific task and the cost of using a neural network to perform task is to train a smaller version of the neural network for the specific task. Such a technique, for a set of tasks and a collection of various sized neural networks, would require independent training and storing of learned parameters for each task and size, which would be impractical in terms of time needed and memory required. This specification describes techniques that can address the aforementioned challenges by generating a smaller task specialized neural network from a larger general neural network. That is, this specification describes techniques that enable learning a projection of weight matrices of the larger neural network to generate the smaller task specialized neural network by processing a task input using a projection neural network system to generate an output that defines sets of projection matrices and generating the task neural network with fewer parameters than the larger neural network by applying the sets of projection matrices to respective weight matrixes of the larger neural network. By processing a task input using the projection neural network system to generate projection matrices, and using these matrices to generate projected matrices from a larger neural network weight matrices, the described techniques enable the system to generate and provide a task neural network for use in performing a task that has lower cost of use (e.g., Attorney Docket No.45288-0487WO1 memory cost, compute resource cost, latency cost, etc.) than the larger neural network but maintains comparable or better performance in performing the task as the larger neural network has. Thus, the described techniques provide a method of task specialization that does not require fine-tuning training for the task and that can improve the cost of performing a particular task through the generation and use of a task neural network. Additionally, because the generated task neural network of the described techniques is a smaller neural network (in terms of memory cost, number of parameters, etc.) relative to the larger neural network, the described techniques provide a method of “compressing” a larger neural network into any of a variety of different sizes. For example, the described techniques allow a larger neural network to be adapted for deployment in a resource constrained environment, e.g., an edge device, where the larger neural network cannot be deployed through the generation and use of a task specific neural network. As a particular example, for a resource constrained environment such as a user smartphone with limited memory constraints, a task specific neural network (e.g., one that performs image recognition) can be generated from a larger neural network and deployed on the user smartphone. According to a first aspect, there is provided a method performed by one or more computers, the method comprising: obtaining a task input specifying a task to be performed by a neural network; processing the task input using a projection neural network system to generate an output that defines, for each of a plurality of weight matrices of a first neural network, a set of projection matrices; generating a task neural network for performing the task, wherein the task neural network has fewer parameters than the first neural network, and wherein the generating comprises, for each of the plurality of weight matrices of the first neural network, applying the projection matrices to the weight matrix to generate a projected weight matrix having a smaller size than the weight matrix; and providing data specifying the task neural network for performing the task. In some implementations, providing data specifying the task neural network for performing the task comprises: providing the task neural network for deployment on a user device. Attorney Docket No.45288-0487WO1 In some implementations, obtaining a task input specifying a task to be performed by a neural network comprises: receiving the task input from the user device; or generating the task input based on data received from the user device. In some implementations, the first neural network is maintained on a user device and wherein providing data specifying the task neural network for performing the task comprises deploying the task neural network on the user device. In some implementations the method further comprises: receiving a new input for the task; and processing the new input using the task neural network to generate an output for the task. In some implementations, for a given weight matrix of a first neural network that has size dn x dn, the set of projection matrices includes a first projection matrix that has size sdn x dn and a second projection matrix that has size dn x sdn. In some implementations, applying the projection matrices to the weight matrix to generate a projected weight matrix having a smaller size than the weight matrix comprises, for the given weight matrix, generating a projected weight matrix by computing a product of (i) the first projection matrix, (ii) the given weight matrix, and (iii) the second projection matrix. In some implementations, each projection matrix in the set is defined by a respective vector and wherein the output comprises the respective vector for each of the projection matrices. In some implementations, the projection neural network system comprises a respective projection neural network for each of the plurality of weight matrices that is configured to process the task input to generate an output that defines the set of projection matrices for the weight matrix. In some implementations, the respective projection neural networks are multi-layer perceptrons (MLPs). In some implementations, the respective projection neural networks share at least some parameters. In some implementations, the projection neural network system has been trained on a training data set that includes, for each of a plurality of training tasks, a respective set of training examples. In some implementations, the task is one of the training tasks. In some implementations, the projection neural network system has been trained on an objective that measures, for each training task, a performance on the training examples Attorney Docket No.45288-0487WO1 for the training task of a training task neural network for the training task that has been generated from the first neural network using sets of projection matrices generated by the projection neural network system by processing training task data specifying the training task. In some implementations, the first neural network is also trained as part of the training of the projection neural network. In some implementations, the first neural network has been pre-trained prior to being trained as part of the training of the projection neural network. According to a second aspect, there is provided a method performed by one or more computers, the method comprising: receiving a new input for a task; and processing the new input using a task neural network for the task to generate an output for the task, wherein the task neural network has been generated from a first neural network by performing the respective operations of any of the above methods. According to a third aspect, there is provided a method performed by one or more computers, the method comprising: obtaining a set of training data that includes, for each of one or more training tasks, a respective set of training examples, each training example comprising a respective training input for the task; for each of the one or more tasks: processing a task input specifying the task using a projection neural network system to generate an output that defines, for each of a plurality of weight matrices of a first neural network, a set of projection matrices; generating a task neural network for performing the task, wherein the task neural network has fewer parameters than the first neural network, and wherein the generating comprises, for each of the plurality of weight matrices of the first neural network, applying the projection matrices to the weight matrix to generate a projected weight matrix having a smaller size than the weight matrix; and for each training example for the task, processing the respective training input in the training example using the task neural network to generate a task output for the training input; and training the projection neural network system on an objective that measures, for each training task, a performance on the training examples for the training task of the task output for the task. Attorney Docket No.45288-0487WO1 In some implementations, the method further comprises: for each task and for one or more of the training examples for the task: processing the respective training input in the training example using the first neural network to generate another task output for the training input; and training the first neural network on an objective that measures, for each training task, a performance on the training examples for the training task of the other task outputs. In some implementations, the first neural network has a decoder-only architecture. In some implementations, the first neural network has an encoder-decoder architecture. In some implementations, the first neural network has an encoder-only architecture. In some implementations, the first neural network comprises one or more self-attention layers. In some implementations, the first neural network comprises one or more cross-attention layers. In some implementations, the first neural network is a denoising neural network. In some implementations, the task is an image processing task. In some implementations, the task is a video processing task. In some implementations, the task is an audio processing task. In some implementations, the task is a multi-modal task that requires processing two or more of an image, video, audio, or text. According to a fourth aspect, there is provided a system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one more computers to perform the operations of any of the above methods. According to a fifth aspect, there is provided one or more computer storage media storing instructions that when executed by one or more computers cause the one more computers to perform the operations of any of the above methods. The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS FIG.1A shows a neural network generation system. FIG.1B shows an example of the operation of the neural network generation system when the first neural network includes a self-attention layer block. FIG.2A is a flow diagram of an example process for generating a task neural network for performing a specific task from a first neural network. Attorney Docket No.45288-0487WO1 FIG.2B shows an example of the projection of a weight matrix of the first neural network. FIG.3 is a flow diagram of an example process for training a projection neural network system. FIG.4 is an example of the performance of the described techniques. FIG.5 is an example of the performance of the described techniques. FIG.6 is an example of the performance of the described techniques. FIG.7 is an example of the performance of the described techniques. FIG.8 is an example of the performance of the described techniques. Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION FIG.1A shows an example neural network generation system 100. The system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented. The neural network generation system 100 generates a task neural network 114 for performing a specific task from a first neural network 110, where the first neural network 110 is already trained (e.g., pre-trained to perform a variety of tasks). In particular, the system 100 obtains a task input 102 specifying a task to be performed by a neural network. The task input 102 can be any appropriate data characterizing a task, e.g., a task identifier for the task, a text prompt that describes the task in natural language text, an example input for the task, and so on. The system 100 processes the task input 102 using a projection neural network system 104 to generate an output that defines, for each of a plurality of weight matrices 108 of the first neural network 110, a set of projection matrices 106. The projection neural network system 104 can be a single neural network or a collection of multiple different neural networks. The system 100 then generates a task neural network 114 for performing the task that has fewer parameters than the first neural network 110. To generate a task neural network 114, for each of the plurality of weight matrices 108 of the first neural network 110, the system 100 applies the projection matrices 106 for the weight matrix (that are defined by the output of the projection neural network system 104) to the weight matrix 108 Attorney Docket No.45288-0487WO1 to generate a projected weight matrix 112 having a smaller size than the weight matrix 108, where the size of a matrix is the total number of entries in the matrix (i.e., a product of the number of rows of the matrix and the number of columns in the matrix). Thus, the task neural network 114 has the same architecture as the first neural network 110 but with each of the plurality of weight matrices replaced by a corresponding smaller matrix, thereby greatly reducing the number of parameters relative to the first neural network 110. Additionally, the task neural network 114 can perform the task without any additional training. The system 100 then provides data 116 specifying the task neural network 114 for performing the task. The first neural network 110 and task neural network 114 (collectively referred to below as “the neural network”) can have any of a variety of neural network architectures. Examples of such neural networks include convolutional neural networks, encoder-only Transformer neural networks, decoder-only Transformer neural networks, encoder-decoder Transformer neural networks, state space models, recurrent neural networks, residual fully- connected neural networks and so on. In some situations, the neural network can be referred to as an auto-regressive neural network when the neural network auto-regressively generates an output sequence of tokens. More specifically, the auto-regressively generated output is created by generating each particular token in the output sequence conditioned on a current input sequence that includes any tokens that precede the particular token in the output sequence, i.e., the tokens that have already been generated for any previous positions in the output sequence that precede the particular position of the particular token. For example, the neural network can be an auto-regressive Transformer-based neural network that includes (i) a plurality of attention blocks that each apply a self-attention operation and (ii) an output subnetwork that processes an output of the last attention block to generate the score distribution. In this example, the neural network can have any of a variety of Transformer-based neural network architectures. Examples of such architectures include those described in J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language models, arXiv preprint arXiv:2203.15556, 2022; J.W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, H. F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young, E. Rutherford, T. Hennigan, J. Menick, A. Cassirer, R. Powell, G. van den Driessche, L. A. Hendricks, M. Attorney Docket No.45288-0487WO1 Rauh, P. Huang, A. Glaese, J. Welbl, S. Dathathri, S. Huang, J. Uesato, J. Mellor, I. Higgins, A. Creswell, N. McAleese, A.Wu, E. Elsen, S. M. Jayakumar, E. Buchatskaya, D. Budden, E. Sutherland, K. Simonyan, M. Paganini, L. Sifre, L. Martens, X. L. Li, A. Kuncoro, A. Nematzadeh, E. Gribovskaya, D. Donato, A. Lazaridou, A. Mensch, J. Lespiau, M. Tsimpoukelli, N. Grigorev, D. Fritz, T. Sottiaux, M. Pajarskas, T. Pohlen, Z. Gong, D. Toyama, C. de Masson d’Autume, Y. Li, T. Terzi, V. Mikulik, I. Babuschkin, A. Clark, D. de Las Casas, A. Guy, C. Jones, J. Bradbury, M. Johnson, B. A. Hechtman, L. Weidinger, I. Gabriel, W. S. Isaac, E. Lockhart, S. Osindero, L. Rimell, C. Dyer, O. Vinyals, K. Ayoub, J. Stanway, L. Bennett, D. Hassabis, K. Kavukcuoglu, and G. Irving. Scaling language models: Methods, analysis & insights from training gopher. CoRR, abs / 2112.11446, 2021; Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019; Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. Towards a human-like open-domain chatbot. CoRR, abs / 2001.09977, 2020; and Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020. Generally, to apply the self-attention operation, each attention block uses one or more attention heads. Each attention head generates a set of queries, a set of keys, and a set of values, and then applies any of a variety of variants of query-key-value (QKV) attention, e.g., a dot product attention function or a scaled dot product attention function, using the queries, keys, and values to generate an output. Each query, key, value can be a vector that includes one or more vector elements. When there are multiple attention heads, the attention block then combines the outputs of the multiple attention heads, e.g., by concatenating the outputs and, optionally, processing the concatenated outputs through a linear layer. The neural network can generally be configured to perform any of a variety of tasks. That is, the neural network can perform any kind of machine learning task, i.e., can be configured to receive any kind of digital data input and to generate any kind of score, classification, or regression output based on the input. In some cases, the neural network system is configured to perform an image processing task, i.e., receive an input image and to process the input image, i.e., to Attorney Docket No.45288-0487WO1 process intensity values of the pixels of the image, to generate a network output for the input image. The input image may be a real-world image. The input image may be an image of a real-world environment captured by a camera. For example, the task may be image classification and the output generated by the neural network for a given image may be scores for each of a set of object categories, with each score representing an estimated likelihood that the image contains an image of an object belonging to the category. As another example, the task can be image embedding generation and the output generated by the neural network can be a numeric embedding of the input image. As yet another example, the task can be object detection and the output generated by the neural network can identify locations in the input image at which particular types of objects are depicted. As yet another example, the task can be image segmentation and the output generated by the neural network can assign each pixel of the input image to a category from a set of categories. As another example, the task can be a depth prediction task. In a depth prediction task, the output generated by the neural network identifies, for each pixel in the image, a predicted depth of the scene at the pixel. As yet another example, the task can be a surface normal prediction task. In a surface normal prediction task, the output generated by the neural network identifies, for each pixel in the image, a predicted surface normal of the scene at the pixel. As another example, the neural network can be configured to perform a video processing task, where the neural network receives a video that includes a sequence of input images and processes the input images, i.e., process the intensity values of the pixels of the images, to generate a network output for the video. For example, the network output can be a classification output that includes a respective score for each of multiple categories, where the categories represent, e.g., topics of the video, object categories, or action categories that each correspond to possible actions that may be being performed by entities in the video, and each score represents an estimated likelihood that the video belongs to the category. As another example, the network output can identify optical flow between pixels of the images in the video. As another example, the network output can be one or more predicted images that are predicted to follow the last image in the sequence. As another example, if the inputs to the neural network system are Internet resources (e.g., web pages), documents, or portions of documents or features extracted from Internet resources, documents, or portions of documents, the task can be to classify the resource or document, i.e., the output generated by the neural network system for a given Internet resource, document, or portion of a document may be a score for each of a Attorney Docket No.45288-0487WO1 set of topics, with each score representing an estimated likelihood that the Internet resource, document, or document portion is about the topic. As another example, if the inputs to the neural network system are features of an impression context for a particular advertisement, the output generated by the neural network system may be a score that represents an estimated likelihood that the particular advertisement will be clicked on. As another example, if the inputs to the neural network system are features of a personalized recommendation for a user, e.g., features characterizing the context for the recommendation, e.g., features characterizing previous actions taken by the user, the output generated by the neural network system may be a score for each of a set of content items, with each score representing an estimated likelihood that the user will respond favorably to being recommended the content item. As one example, the task may be a neural machine translation task. For example, if the input to the neural network system is a sequence of text, e.g., a sequence of words, phrases, characters, or word pieces, in one language, the output generated by the neural network system may be a translation of the sequence of text into another language, i.e., a sequence of text in the other language that is a translation of the input sequence of text. As a particular example, the task may be a multi-lingual machine translation task, where a single neural network is configured to translate between multiple different source language – target language pairs. In this example, the source language text may be augmented with an identifier that indicates the target language into which the neural network system should translate the source language text. As another example, the task may be an audio processing task. For example, if the input to the neural network system is a sequence representing a spoken utterance, e.g., a spectrogram or a waveform or features of the spectrogram or waveform, the output generated by the neural network system may be a piece of text that is a transcript for the utterance. As another example, if the input to the neural network system is a sequence representing a spoken utterance, the output generated by the neural network system can indicate whether a particular word or phrase (“hotword”) was spoken in the utterance. As another example, if the input to the neural network system is a sequence representing a spoken utterance, the output generated by the neural network system can identify the natural language in which the utterance was spoken. As another example, the task can be a natural language processing or understanding task, e.g., an entailment task, a paraphrase task, a textual similarity task, a Attorney Docket No.45288-0487WO1 sentiment task, a sentence completion task, a grammaticality task, and so on, that operates on a sequence of text in some natural language. As another example, the task can be a text to speech task, where the input is text in a natural language or features of text in a natural language and the network output is a spectrogram, a waveform, or other data defining audio of the text being spoken in the natural language. As another example, the task can be a health prediction task, where the input is a sequence derived from electronic health record data for a patient and the output is a prediction that is relevant to the future health of the patient, e.g., a predicted treatment that should be prescribed to the patient, the likelihood that an adverse health event will occur to the patient, or a predicted diagnosis for the patient. As another example, the task can be a text generation task, where the system receives a conditioning input and generates as output a sequence of text. For example, the conditioning input can be another sequence of text, e.g., so that the output sequence is a completion of the input sequence of text, a response to a question posed in the input sequence, or a sequence of text that is about a topic specified by the first sequence of text. As another example, the input to the text generation task can be an input other than text, e.g., an image, and the output sequence can be text that describes the input. As another example, the task can be an image generation task, where the input is a conditioning input and the output is a sequence of intensity values for the pixels of an image. As another example, the task can be a computer vision task, where the input is an image or a point cloud and the output is a computer vision output for the image or point cloud, e.g., a classification output that includes a respective score for each of a plurality of categories, with each score representing the likelihood that the image or point cloud includes an object belonging to the category. When the input is an image or point cloud, the neural network system can include an embedding subnetwork that generates a respective embedding for each multiple patches of the image or point cloud, and the input to the first block of the neural network system can be a sequence that includes the respective embeddings (and, optionally, one or more additional embeddings, e.g., at a predetermined position that will later be used to generate the output). Each patch includes the intensity values of the pixels in a different region of the input image. As another example, the task can be a multi-modal task that requires processing two or more of the modalities described above to generate an output. Attorney Docket No.45288-0487WO1 As another example, the task can be an agent control task, where the input is a sequence of observations or other data characterizing states of an environment and the output defines an action to be performed by the agent in response to the most recent data in the sequence. The agent can be, e.g., a real-world or simulated robot, a control system for an industrial facility, or a control system that controls a different kind of agent. As another example, the task can be a genomics task, where the input is a sequence representing a fragment of a DNA sequence or other molecule sequence and the output is either an embedding of the fragment for use in a downstream task, e.g., by making use of an unsupervised learning technique on a data set of DNA sequence fragments, or an output for the downstream task. Examples of downstream tasks include promoter site prediction, methylation analysis, predicting functional effects of non-coding variants, and so on. FIG.1B shows an example 150 of the operation of the system 100 when the first neural network 110 (“source model”) includes two or more self-attention layer blocks. Each self-attention layer block has a respective set of weight matrices 108, e.g., ^^^^,^^ொ^ ,^^^^and so on for one of the self-attention layer blocks, that define the operations performed by the self-attention layer and the multi-layer perceptron (MLP) of the layer block. The projection neural network system 104 generates, for each of the weight matrices of each of the self-attention layer blocks, one or more projection matrices 106 depend on the task input t 102. The system 100 then uses the respective sets of one or more projection matrices 106 to generate a respective projected weight matrix 112 for each weight matrix 108 of the self-attention blocks to be included in the task neural network 114 (“projected model”), e.g., the projected weight matrices ^^^^,^^ொ^ ,^^^^for the weight matrices ^^^^,^^ொ^ ,^^^^. Because the projected weight matrices 112 have a smaller dimensionality relative to the corresponding weight matrices 108, the task neural network 114 has a similar but more computationally efficient architecture relative to the first neural network, i.e., because the task neural network 114 has fewer parameters. FIG.2A is a flow diagram of an example process 200 for generating a task neural network for performing a specific task from a first neural network. For convenience, the process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a neural network generation system, e.g., the Attorney Docket No.45288-0487WO1 neural network generation system 100 of FIG.1A, appropriately programmed in accordance with this specification, can perform the process 200. The system obtains a task input specifying a task to be performed by a neural network (step 202). As described above, the task can be any of a variety of types of tasks. Some examples include image processing tasks (e.g., captioning an image, classifying an image, identifying objects in an image, segmenting the image, transforming the image, in-filling an image, editing an image, predicting additional pixels from current pixels, denoising an image and so on), video processing tasks (e.g., summarizing the content in a video, classifying a video, identifying content within a video, tracking an object in a video, predicting new video frames from previous video frames, transforming a video, dividing a video into chapters, and so on), and audio processing tasks (e.g., identifying speech content, identifying speech language, identifying the speaker, removing particular audio contributions like background noise, editing parts of audio data, enhancing audio quality such as upsampling, compression audio data, and so on). Other examples of tasks are multi-modal tasks that require processing two or more of an image, video, audio, or text. For example, the task can be to identify objects in image according to their provided text description. As another example, the task can be to process audio and video data to generate closed caption text for the audio and video data. As another example, the task can be to process audio and video data to generate an action for agent interacting an in environment that generated the audio and video data to perform. The system can obtain the task input from any source, e.g., from a user or another system, and, as described above, the task input can be any appropriate data characterizing a task, e.g., a task identifier for the task, a text prompt that describes the task in natural language, an example input for the task, and so on. For example, the task input can be a task identifier represented as a finite multi- dimensional vector ^^. For example, the system can directly obtain the finite multi- dimensional vector ^^, e.g., from a user, or can receive data representing the task input that the system then processes to generate the finite multi-dimensional vector ^^. For example, the system can obtain task input options where each option is mapped to a numeric entry of the finite multi-dimensional vector ^^ and, ^^ can be such that each unique value of ^^ has a one-to-one correspondence with a unique task. For example, the task Attorney Docket No.45288-0487WO1 of generating an transformed image with a particular background can be represented as task identifiers ^^, where each entry of ^^ corresponds to parameter of the image transformation. As another particular example, ^^ can be an encoded representation of a task. For example, ^^ can be an output of an encoder neural network, e.g., an image encoding neural network that processes images and generates respective embedding vectors that represents the images. For example, the task of generating an image can be represented as task identifier ^^, where ^^ is the encoded representation of an image generated by an image encoder (e.g., SIMCLRv2) and represents the image to be generated. As another example, the task input can be natural language text that describes a task. For example, the natural language text can be one that describes the task of generating computer programming code, generating a language translation, generating a text summary. For example, the task input can be “Write a python function that can accept any number of float type arguments and returns their product.” As another example, the task input can be an example input for the task. For example, the task input can be a “few-shot prompt” with examples of input-output-pairs for a classification task (i.e., a natural language text input that describes the task through examples). As a particular example, the classification task may be to classify the sport associated with a body of text and an input can be “(body of text: a touchdown is 6 points, sport: football); (body of text: a lap on the track is 400 M sport: track & field);”. The listing of examples, provide the information needed for the task so, for example, a task input of “(body of text: the swimming pool should be 50 meters in length, sport:” will properly be processed to generate an output for the input of a task of, e.g., “swimming”. In some cases, the system obtains the task input by receiving the task input from a user device. In some other cases, the system obtains the task input by generating the task input based on data received from the user device. Some examples of user devices include a smart phone, a tablet, or a laptop computer. In some cases, the system receives the task input (or data that will determine the task input) from a user of the user device. For example, a user of a smart phone, tablet, or laptop, can provide the task input (or data that will determine the task input) to the user device through the user device interface (i.e., the touch screen, mic, or camera for the smart phone and tablet, and the keyboard or mic for the laptop). Then, the system can receive the task input (or data that determines the task input) from the user device over a communication Attorney Docket No.45288-0487WO1 network (e.g., cloud-based network, the internet, a local area network, wireless network, and so on). The system processes the task input using a projection neural network system to generate an output that defines, for each of a plurality of weight matrices of a first neural network, a set of projection matrices (step 204). The system generates a task neural network for performing the task (step 206). As described above, the task neural network has fewer parameters than the first neural network. To generate the task neural network, the system, for each of the plurality of weight matrices of the first neural network, applies the projection matrices to the weight matrix to generate a projected weight matrix having a smaller size than the weight matrix. As described above, the first neural network can have any of a variety of neural network architectures. That is, the first neural network can have any appropriate architecture in any appropriate configuration that can process an input for a task to generate an output for the task, including fully connected layers, convolutional layers, recurrent layers, attention-based layers, and so on, as is appropriate. For example, the first neural network can include one or more self-attention layers, one or more cross-attention layers, or both. As a particular example, the first neural network can have a transformer based architecture as described in arXiv:1706.03762, where the transformer encoder includes a self-attention layer, and the transformer decoder includes a self-attention layer and a cross-attention layer. As another example, the first neural network can have an encoder-decoder architecture. As a particular example, the first neural network can have the transformer encoder-decoder architecture described above, or the T5 transformer encoder-decoder architecture as described in arXiv:1910.10683. As another particular example, the first neural network can have a variational autoencoder (i.e., an encoder-decoder) architecture, e.g., as described in arXiv:1312.6114. As another example, the first neural network can have a decoder-only architecture. As a particular example, the first neural network can belong to the Gemma family of neural networks as described in arXiv:2403.08295. As another example, the first neural network can have an encoder-only architecture. As a particular example, the first neural network can be BERT as described in arXiv:1810.04805v2. As another example, the first neural network can be a denoising neural network. As a particular example, the first neural network can be a denoising neural network based on Attorney Docket No.45288-0487WO1 the U-Net architecture, e.g., as described in arXiv:2102.09672, or arXiv:2006.11239. As another particular example, the first neural network can be a denoising neural network based on the U-ViT architecture as described in arXiv:2209.12152. As described above, the projection neural network system can be single neural network or a collection of different neural networks, and each of the projection neural network(s) can have any of a variety of neural network architectures. That is, each projection neural network in the projection neural network system can have any appropriate architecture in any appropriate configuration, including fully connected layers, convolutional layers, recurrent layers, attention-based layers, and so on, as is appropriate. Generally, the projection matrices generated by the projection neural network system are those that when applied to their corresponding weight matrices generate corresponding projected weight matrices that have smaller sizes than their corresponding weight matrices. For example, the projection matrices can be those that linearly transform the weight matrices. For example, for a given weight matrix ^^^of a first neural network associated ^ with the linear operator ^^ ^ defined as ^ ^^ ^^^^^:ൌ ^^^ ^^ ^ ^^^, as other subscripts signify matrix element indices. In some cases, for the previous example, the set of projection matrices also linearly transforms the bias ^^^. For example, the set of projection matrices can include ^^^and ^^^which generates a projected bias ^^^as ^^^^^^:ൌ ^^^^^^^^^^^^ ^ ^^^^where ^^^^^^is the kth is the ith element of ^^^^^^. It will be understood that a bias vector can be incorporated into a weight matrix by augmenting the weight matrix with an additional column that is the bias vector and augmenting the operand the weight matrix is applied to by an additional row with a constant (e.g., a scalar constant 1) and is considered to be a part of the weight matrix. Thus, the set of projection matrices, when applied to their corresponding weight matrices, generate Attorney Docket No.45288-0487WO1 corresponding projected weight matrices that have smaller sizes than their corresponding weight matrices, and simultaneously handles and includes the bias vector. In some cases, for a given weight matrix ^^^of a first neural network that has size dn x dn, the set of projection matrices includes a first projection matrix ^^ that has size sdn x dn and a second projection matrix ^^ that has size dn x sdn. When sdn is less then dn (e.g., dn is an integer greater than or equal to 1 and s is a fraction less than 1 such that sdn is an integer value and dn > sdn) then the projection matrices can be used to generate a projected matrix ^^^with a smaller size than the weight matrix ^^^. So, for example, the system can compute a projected matrix ^^^that has a smaller size than the weight matrix ^^^by computing a product of (i) the first projection matrix ^^, (ii) the given weight matrix ^^^, and (iii) thesecond projection matrix ^^, i.e., ^^^ ൌ ^^^^^^^. That is, the dimensions of ^^^ are sdn x sdnwhich is a total of (sdn)2matrix elements which is less than the dn2elements that the matrix ^^^with dimensions dn x dn has. In one example, the first neural network is a transformer neural network with embedding size d that defines the sizes of all self-attention and MLP weight matrices. In a further example, for a given weight matrix ^^^of the first neural network that has size dn x dn, where n is a positive integer, the set of projection matrices includes a first projection matrix ^^ that has size sdn x dn and a second projection matrix ^^ that has size dn x sdn. In an example, s=2-kwith integer k≥0, where it is assumed that d / 2kis integer for all models of interest. It will be understood that any given weight matrix ^^^of the first neural network is not required to be square (e.g., the above described example in which a weight matrix is augmented to accommodate a bias).

[0002] Attorney Docket No.45288-0487WO1 So, in some cases, for a given weight matrix ^^^of a first neural network that has size dn x dm, the set of projection matrices includes a first projection matrix ^^ that has size sdn x dn and a second projection matrix ^^ that has size dm x sdm. When sdn is less then dn and sdn is less then dm (e.g., s is a fraction less than 1 such that sdn and sdm are integer values and dn > sdn and dm > sdm) then the projection matrices can be used to generate a projected matrix ^^^with a smaller size than the weight matrix ^^^. So, for example, the system can compute a projected matrix ^^^that has a smaller size than the weight matrix ^^^by computing a product of (i) the first projection matrix ^^, (ii) the given weight matrix ^^^,and (iii) the second projection matrix ^^, i.e., ^^^ ൌ ^^^^^^^. That is, the dimensions of ^^^ aresdn x sdm which is a total of (sdn* sdm) matrix elements which is less than the dn* dm elements that the matrix ^^^with dimensions dn x dm has. In some cases, each projection matrix in the set is defined by a respective vector and the output of the projection neural network system includes the respective vector for each of the projection matrices. For example, the projection matrices ^^ and ^^ for the weight matrix ^^^described above can be defined respectively by the projection neural network system generatedvectors ^^ and ^^ through relationships such as ^^^^ ൌ ^^^^^^,⌊^^⌋ and ^^^^ ൌ ^^^^^^,⌊^^⌋ where theterms ⌊^^^^⌋ and ⌊^^^^⌋ are outputs of “floor functions” used because ^^ is a fraction less than 1 and ^^^^ / ^^^^ are not necessarily integers, ^^ is the Kronecker delta function, and ^^ and ^^ are matrix or vector indices. In some implementations, the projection neural network system includes a respective projection neural network for each of the plurality of weight matrices that is configured to process the task input to generate an output that defines the set of projection matrices for the weight matrix. For example, a set of projection neural networks, one for each of the plurality of weight matrices of the first neural network, can each process a task input ^^ to generate two respective vectors ^^^^^^ and ^^^^^^ (where the task input ^^ here denotes that the vectors ^^ and ^^ are functions of ^^) which in turn can determine the projection matrices ^^ and ^^ describedabove as ^^^^ ൌ ^^^^^^^^^^,⌊^^⌋ and ^^^^ ൌ ^^^^^^^^^^,⌊^^⌋ where the terms ⌊^^^^⌋ and ⌊^^^^⌋ are outputsof “floor functions” and ^^ is the function. Attorney Docket No.45288-0487WO1 An example of this is shown in FIG.2B. In particular, FIG.2B shows an example 250 of a projection ^^^of a weight matrix ^^^of the first neural network. As can be seenfrom the example 250, the projection ^^^ ^^^^ ൌ ^^^^^^^^^^^^^^^ is a 3x3 matrix with 9 totalentries whereas ^^^is a 6x6 matrix with 36 total entries. P and Q are defined by outputs of the projection neural network system, and the dependency on t is introduced because the projection neural network system processes t to generate the values that define P and Q, respectively. More specifically, in the example of FIG.2B, the projection neural network system generates a vector p(t) that includes entries ^^^through ^^^that specify the diagonal elements of the matrix P and a vector q(t) that includes entries ^^^through ^^^that specify the diagonal elements of the matrix Q. In some cases, the respective projection neural networks in the projection neural network system are multi-layer perceptrons (MLPs). For example, the projection neural networks of the previous example can be MLPs. Other examples of neural networks that the respective projection neural networks in the projection neural network system can be are convolutional neural networks, encoder- only Transformer neural networks, decoder-only Transformer neural networks, encoder- decoder Transformer neural networks, recurrent neural networks, residual fully-connected neural networks and so on. In some cases, the respective projection neural networks for each of the weight matrices of a first neural network included in a projection neural network system share at least some parameters. For example, the initial layers of the projection neural networks can be shared among all the projection neural networks while each neural network has a separate output head. The respective projection neural networks sharing some parameters helps reduce the number of parameters of the projection neural network system and can help learn shared representations of the task input. Prior to using the projection neural network system to process the task input and generate an output that defines a set of projection matrices for each of a plurality of weight matrices of a first neural network, the system or another system trains the projection neural network system. In some cases, the system trains the projection neural network system specifically for use with the first neural network, i.e., the system trains the projection neural network system to generate sets of projection matrices for the respective weight matrices of the first neural network and for a variety of tasks for the first neural network. Attorney Docket No.45288-0487WO1 In order to train the projection neural network system to generate task specific output for various tasks, in some cases, the system trains the projection neural network system on a training data set that includes a set of training examples for each of a plurality of training tasks. That is, in some cases, the projection neural network system has been trained on a training data set that includes, for each of a plurality of training tasks, a respective set of training examples. In some cases, each training example belonging to a respective training task includes an appropriate training input for the training task that can be processed by a neural network (e.g., a training task neural network, a first neural network, or both). So, for example, if the training task were an image processing task, then the training input for this training task would be an image. Likewise, as another example, if the training task were a multi-modal processing task (e.g., processing text instructions to identify objects of an image by generating a version of the image with bounding boxes around the identified objects), then the training input can be natural language text description of objects and the image to locate them in. In some cases, each training example belonging to a respective training task includes an appropriate training output for the training task (e.g., the output that that a neural network is expected to generate when used to process the respective training input of the training example for the training task). For example, if the task is a classification task, e.g., classify if an image is a dog or not, the training output can be classification output, e.g., a numeric value of 1 or 0 representing ‘dog’ or ‘not a dog’, or a numeric value between 0 and 1 that represents a likelihood of the image being of a dog. In some cases, the training input of the training example can also serve as the training output. For example, when the task is a next token prediction of a sequence task, each token belonging to sequence of tokens that is a training input is also the expected output for the preceding tokens of the sequence. For example, if a training input is a sequence of word tokens for the phrase “How are you?”, then the word token “are” can be the expected output for the token sequence “How” and the word token “you” can be the expected output for the sequence “How”, “are”. In some cases, the task specified by the task input that the system obtains to generate a task neural network is included in the training tasks. That is, the task is one of the training tasks. In other cases, the task specified by the task input that the system obtains to generate a task neural network is not included in the training tasks. That is, the task is not one of the Attorney Docket No.45288-0487WO1 training tasks, and therefore, the tasks the task neural network can perform generalizes to new tasks after training. In some cases, in order to train the projection neural network system, the system uses, for each training task, a training task neural network for the training task, where the system generates the training task neural network from the first neural network using sets of projection matrices generated by the projection neural network system by processing training task data specifying the training task. Moreover, the system trains the projection neural network system on an objective that measures, for each training task, the respective training task neural network performance on the training examples for the training task. For example, the system can train the projection neural network system on an objective such as: ^^^క,௧^∼^^∗ℒ^ೞ^௧^^^^^ , ^^ the training task input, ^^ represents a training example that a and the respective training output ^^ for the training input ^^, ^^^క,௧^∼^^∗represents an expectation over training tasks and respective training examples for task that are included in the training data set ^^∗, ^^^^^^^ represents a training task neural network for the training task ^^, and ℒ^ೞ^௧^^^^^ represents the performance of the training task neural network ^^^^^^^ on a training example ^^ as a loss (i.e., loss function). For the previous example, the loss ℒ of ℒ^ೞ^௧^^^^^ can be any of a variety of appropriate types of loss functions for the that can measure a performance of the training task neural network ^^^^^^^ on a training example ^^ for the training task ^^. For example, if the training task were classification task or next token prediction task (e.g., next word prediction, next pixel prediction) then the loss can be the log loss (i.e., cross entropy loss, i.e., negative log likelihood loss) between the training output ^^ and generated output. As another example, the loss ℒ can be mean squared error between the training output y and the generated output for tasks like predictions of continuous values (e.g., time series forecasting of real world events), denoising noisy images, and so on. For the previous example, the trainable parameters of the projection neural network system are present in the objective through the training task neural network ^^^^^^^ because the values of the trainable parameters of the projection neural network system define the training task neural network, i.e., define how the first neural network is “projected” to generate the training task neural network. So, the system can update the trainable parameters of the projection neural network system by, e.g., performing backpropagation Attorney Docket No.45288-0487WO1 using a gradient descent on the objective to updates trainable parameters through ^^^^^^^ which include the trainable parameters of the projection neural network system. For example, consider an example projected weight matrix ^^^belonging to the training task neural network ^^^^^^^ defined as ^^^ ൌ ^^^^^^^ where ^^ and ^^ are the set of projectionmatrices for a frozen (i.e., not updated during training) weight matrix ^^^of the first neural network generated by the projection neural network system. During backpropagation, since P and Q are outputs of differentiable function(s) (e.g., outputs of neural network(s) belonging to the neural network projection system), backpropagation will continue through those function(s) to update those functions’ parameters. Thus, because the parameters of those functions belong to the projection neural network system, the parameters of the projection neural network will be updated. Further in some cases, the system also trains the first neural network as part of the training of the projection neural network system. For example, considering the above example objective of ^^^క,௧^∼^^∗ℒ^ೞ^௧^^^^^ and the associated above example projected weight matrix ^^^belonging to neural network ^^^^^^^ defined as ^^^ ൌ ^^^^^^^ . In this example, in addition to the projection neuralnetwork system being trained via backpropagation through P and Q via gradient descent, the weight matrix ^^^of the first neural network can be “unfrozen” (i.e., the parameters of ^^^are trainable and the system can update them during backpropagation via gradient descent). As another example, the system can train the projection neural network system on an objective such as: ^^^క,௧^∼^^∗ℒ^ೞ^௧^^^^^ ^ ^^src^^క∼^^ℒெభ^௧^^^^^,where performing training task ^^, ℒெభ^௧^^^^^ represents the performance of the first neural network ^^^^^^^ on a training ^^ from a training data set ^^ for the first neural network (that can be different from ^^∗described above) as a loss (i.e., loss function), ^^srcis an optional scaling factor to weight the term ^^క∼^^ℒெభ^௧^^^^^, and other terms and notation of this equation are defined as described above. The system can use the ℒெభ^௧^^^^^ term of the objective ^^^క,௧^∼^^∗ℒ^ೞ^௧^^^^^ ^ ^^src^^క∼^^ℒெభ^௧^^^^^ to update the trainable parameters of the first neural network) by, e.g., performing backpropagation using a gradient descent on the objective term ℒெభ^௧^^^^^. Additionally, for this example, the system can update the trainable parameters neural network ^^^^^^^ (i.e., train the Attorney Docket No.45288-0487WO1 first neural network) by, e.g., performing backpropagation using a gradient descent on the objective term ℒ^ೞ^௧^^^^^, e.g., as described above. For the previous example, the loss ℒ of ℒெభ^௧^^^^^ can be any of a variety of appropriate types of loss functions for the training task that can measure a performance of the first neural network ^^^^^^^ on a training example ^^ for the training task ^^. For example, if the training task were classification task or next token prediction task (e.g., next word prediction, next pixel then the loss can be the log loss (i.e., cross entropy loss, i.e., negative log likelihood loss) between the training output ^^ and generated output. As another example, the loss ℒ can be mean squared error between the training output y and the generated output for tasks like predictions of continuous values (e.g., time series forecasting of real world events), denoising noisy images, and so on. In some cases, the first neural network has been pre-trained prior to the system training it as part of the training of the projection neural network system. As described above, in some cases, when the system trains the projection neural network system, the system does not update the trainable parameters of the first neural network. But, in some other cases, when the system trains the projection neural network system, the system does update the trainable parameters of the first neural network. For example, the first neural network can be a decoder only neural network (e.g., Gemma model as described in arXiv:2403.08295) pre-trained on a next-token prediction task using web documents, mathematics, and code. As another example, the first neural network can be Gemini as described in ArXiv:2312.11805, a language model neural network designed to understand, operate across and combine different types of information including text, code, audio, image and video. Training the first neural network in parallel with the projection neural network system can improve the performance of the projection neural network system. For example,when using the objective ^^^క,௧^∼^^∗ℒ^ೞ^௧^^^^^ ^ ^^src^^క∼^^ℒெభ^௧^^^^^ as described above tosimultaneously update the trainable network (e.g., the weight matrices of the first neural network) and the projection neural network system, the updates to the weight matrices of the first neural network through the term ^^^క,௧^∼^^∗ℒ^ೞ^௧^^^^^ can improve how efficient applying sets of projection matrices to the weight matrices to generate projected weight matrices to generate the training task neural network is, and hence, the performance of projection neural network system. But also, updates to the weight Attorney Docket No.45288-0487WO1 matrices of the first neural network through the term ^^క∼^^ℒெభ^௧^^^^^ help ensure that the first neural network still performs well as a foundation model and therefore that its weight matrices are still useful to create a high performing network. Further details of an example of training the projection neural network system are described below with reference to FIG.3. The system provides data specifying the task neural network for performing the task (step 208). In some implementations, after the system provides data specifying the task neural network for performing the task, the system can receive a new input for the task to be performed by the task neural network and process the new input using the task neural network to generate an output for the task. That is, the system can generate outputs for new inputs using the task neural network for the task that has been generated from a first neural network. In some cases, when the system provides data specifying the task neural network for performing the task, the system provides the task neural network for deployment on a user device. That is, the system provides the data to a user device, e.g., to a user device using a communication network as described above, e.g., to a user smart phone over the internet, so that the task neural network can perform the task on the user device. When the first neural network is too large to run efficiently on the user device (e.g., due to excessive memory requirements or high latency during use) deploying the task neural network as described is advantageous. That is, the task neural network can offer faster inference and more efficient memory usage on the user device than the first neural network could, making it better suited for deployment on the user device. For example, a task neural network deployed on a user smart phone can perform an image processing task, e.g., process an image captured by the smart phone camera to upscale them to super resolution, once deployed on the smart phone. So, when the user of the smart phone captures an image using the smart phone, the task neural network can process the image to upscale the image. As another example, a task neural network deployed on a user laptop can perform a video generation task, e.g., generating a video conditioned on natural language text description of the video. So, when a user of the laptop provides a natural language text description of a video, the deployed task neural network on the laptop can generate the video. Attorney Docket No.45288-0487WO1 In some implementations, the first neural network is maintained on a user device and, when the system provides data specifying the task neural network for performing the task, the system deploys the task neural network on the same user device. For example, when the task has strict latency requirements (which the first neural network cannot meet on the device), the task neural network can be deployed on the same user device to perform the task while meeting the latency requirement for the task. For example, the task of speech language translation on a user smartphone requires real time results, so a task neural network that can perform this task in real time is much more useful than first neural network that cannot. FIG.3 is a flow diagram of an example process 300 for training a projection neural network system. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a neural network generation system, e.g., the neural network generation system 100 of FIG.1A, appropriately programmed in accordance with this specification, can perform the process 300. The system trains the projection neural network system to process a task input to generate an output that defines, for each of a plurality of weight matrices of a first neural network, a set of projection matrices. Therefore, the system trains the projection neural network system to generate task specific output. The system trains the projection neural network system by repeatedly updating the trainable parameters of the projection neural network system using a set of training data. That is, the system can repeatedly perform the following described example process using training examples to train the projection neural network system from scratch, i.e., train from randomly initialized trainable parameters, or further train the projection neural network system from pre-trained trainable parameters. In particular, the system obtains a set of training data that includes, for each of one or more training tasks, a respective set of training examples, where each training example includes a respective training input for the task (step 302). The system can receive the training dataset from any of a variety of appropriate sources, e.g., a user, another system, system data repository, and so on. As described above, the training input of a training example is a training input appropriate for the task, e.g., an image for image processing task, audio data for audio processing task, and so on. Furthermore, in some cases, the training example also includes a Attorney Docket No.45288-0487WO1 training output that is the expected output of processing the training input using a neural network. For each of the one or more tasks, the system performs the steps 304-308 below. The system processes a task input specifying the task using a projection neural network system to generate an output that defines, for each of a plurality of weight matrices of a first neural network, a set of projection matrices (step 304). In some cases, the system uses the projection neural network system to generate an output that defines a set of projection matrices for each of all of the weight matrices of the first neural network. In other cases, the system uses the projection neural network system to generate an output that defines a set of projection matrices for each of a proper subset of all of the weight matrices of the first neural network. In other cases, the system uses the projection neural network system to generate an output that defines a set of projection matrices for each weight matrix of the first neural network that belong to specific types of layers (e.g., weight matrices belong to attention-based layers). The system generates a task neural network for performing the task, where the task neural network has fewer parameters than the first neural network, and where generating includes, for each of the plurality of weight matrices of the first neural network, applying the projection matrices to the weight matrix to generate a projected weight matrix having a smaller size than the weight matrix (step 306). As described above, the task input can be any appropriate data characterizing a task (e.g., natural language description of a task, an encoded representation of a task such as a multi-dimensional vector or a sequence of multi-dimensional vectors, and so on). Also, as described above, the set of projection matrices are those that when applied to their corresponding weight matrices generate corresponding projected weight matrices that have smaller sizes than the corresponding weight matrices (e.g., smaller size corresponds to fewer matrix entries). The system, for each training example for the task, processes the respective training input in the training example using the task neural network to generate a task output for the training input (step 308). For example, if the task is to generate an image with bounding boxes around described objects of an image and the training input is a text description of the objects and the image, then the system can use the task neural network to generate an image that includes the objects with bounded boxes (i.e., the output for the training input) for each training example. Attorney Docket No.45288-0487WO1 The system trains the projection neural network system on an objective that measures, for each training task, a performance on the training examples for the training task of the task output for the task (step 310). For example, the system can train the projection neural network system on the objective ^^^క,௧^∼^^∗ℒ^ೞ^௧^^^^^ described above (with terms and notation defined above). In other words, the term represents a quantitative measurement of the performance of the task ^^ on a training example ^^ as the loss ℒ, and ^^^క,௧^∼^^∗represents a quantitative measurement of the performance over all training tasks ^^ and respective training examples. For the previous example, the loss ℒ can be any of a variety of types of loss functions that can measure a performance on the training example for the training task of the task output for the task. For example, the loss can be a measure of the difference between the expected task output for the training input and the generated task output for the training input. As a particular example, the loss can be the log loss (i.e., cross entropy loss, i.e., negative log likelihood loss) between the expect and generate task output for tasks like classification, next token prediction, and so on. As another example, the loss can be mean squared error between the expect and generate task output for tasks like predictions of continuous values (e.g., time series forecasting of real world events), denoising noisy images, and so on. In some implementations, the system, for each task and for one or more of the training examples for the task, processes the respective training input in the training example using the first neural network to generate another task output for the training input. Then the system trains the first neural network on an objective that measures, for each training task, a performance on the training examples for the training task of the other task outputs. For example, after processing the training input to generate another task output for the training input for each task and for one or more of the training examples for the task, the system can train the first neural network on the objective ^^క∼^^ℒெభ^௧^^^^^ described above (with terms and notation of defined above). In other words, the term ℒெభ^௧^^^^^ represents a quantitative measurement of the performance of the other task the task ^^ on a training example ^^ as the loss ℒ, and ^^^క,௧^∼^^∗represents a quantitative measurement of the performance over all training tasks ^^ and training examples. Attorney Docket No.45288-0487WO1 For the previous example, the loss ℒ can be any of a variety of types of loss functions described above for the example objective ^^^క,௧^∼^^∗ℒ^ೞ^௧^^^^^. In some implementations, the system trains multiple projection neural network systems in parallel (where each projection neural projection matrices of different sizes). For example, the system can train an objective that includes a term for each projection neural network system and a term for the first neural network. For examples, for a two projection neural network system, the system can train the projectionneural networks on the objective ^^^క,௧^∼^^∗ℒ^^ೞ^௧^^^^^ ^ ^^^క,௧^∼^^∗ℒ^ଶೞ^௧^^^^^ ^^^src^^క∼^^ℒெభ^௧^^^^^, where ^^1^indicates a task neural from a first projection neural network system, ^^2^is a task from a second projection neural network system, and all other terms and notation are as described above. The system updates the trainable parameters of the projection neural network system, the first neural network, or both to optimize the objectives for the projection neural network system, the first neural network, or both in any variety of ways, e.g., gradient based method, evolutionary algorithm-based method, Bayesian optimization, grid search, and so on. For example, the system can optimize the objective by minimizing the loss of one or more training examples described above using any of a variety of gradient descent techniques (e.g., batch gradient descent, stochastic gradient descent, or mini-batch gradient descent) that include the use of a backpropagation technique to estimate the gradient of the loss with respect to trainable parameters and to update the trainable parameters accordingly. Generally, the system repeats the above steps (304-310) until one or more criteria are satisfied (e.g., the system performs a pre-determined number of iterations, the updates to the trainable parameters (of the projection neural network system, the first neural network, or both) no longer exceed a pre-determined magnitude of change, a metric regarding a validation dataset exceeds a pre-determined value, and so on). FIG.4 is an example 400 of the performance of the described techniques. In particular, example 400 shows a table that summarizes the performance of example auto-regressive neural networks, that process image pixels to generate a predicted next image pixel to ultimately generate an image, generated using the described techniques ~ ~ (i.e., the columns labeled ^^^,^^) and other techniques (i.e., the columns labeled ^^^,^^^) in terms of average log loss (i.e., a metric of performance) over many inputs for many tasks. For this example, the tasks are image generation tasks, where the task input ^^ is a multi- Attorney Docket No.45288-0487WO1 dimensional vector, where in this example , that specifies one of three distinct types of the background textures and parameters such as scale, rotation, color and distortion to generate an image In particular, the texture parameters include the angle of rotation (^^^), scale (^^^), and the ^^ and ^^ coordinates of the rotation center (^^ଶand ^^ଷ). The actual rotation angle was2^^^^^, and the scale was defined as ^^^^^^^^^ ≔ 2 ^ 18^^^. The image was also warped usingthe transformation ^^ → ^^ ^ ^^௫^^^^^^^^^௫^^^ and ^^ → ^^ ^ ^^௬^^^^^^൫^^௬^^൯, where ^^௫,௬ ≔^^^^^^^^൫ã௫,௬ െ Each color by ^^^for red, of the image was randomly selected from one ofthree classes, depending on which of ^^^^, ^^^^, ^^^ଶ was largest. These classes were based onthe following function profiles: (a) ^^^^^^^^^^^, where ^^^ is a transformed x-coordinate; (b) ^^^^^^^^̃^^, where ^̃^ is the distance to the origin in transformed coordinates; and (c)^^^^^^^10^^^^^ଷ ^ 2^^^ସ^, where ^^ ≔ ^^^^^^^^^^ / 3^^^^^^^^^^^ / 3^ and ^^^ is a transformed y-coordinate.Finally, the generated texture was multiplied by a random 3-channel RGB color vector (φ + γ ^^^ହ,φ + γ ^^^^, φ + γ ^^^^), where φ = 0.5 and γ = 1.5. The final image was clipped to the output range [-1, 1]. The subscripts ^^ and column labels ^^ (i.e., ^^ ∈ ^1, 1 / 2, 1 / 4, 1 / 8^ indicate amultiplier of an original embedding size. So, the embedding size for s=1 is the fullembedding size ^^ because 1 ∗ ^^ ൌ ^^; the embedding size s=1 / 2 is half the full embedding~size because ½ ∗ ^^ is half of ^^, and so on. In other words, the models ^^^,^^ ^are neural ~networks trained using other techniques with embedding sizes equal to ^^ ∗ ^^, and ^^^,^^are the task neural networks generated using the described techniques with the first neural ~ network being ^^^ୀ^. Additionally, the neural network ^^^and ^^^differ from each other in ~ that ^^^was trained as a conditional autoregressive model with the task identifier ^^ ~ processed as the first embedding in a sequence, while ^^^was not. The neural networks of ^^ differ from those corresponding to ^^^in that each projection neural network system ~ responsible for each of the task neural networks corresponding to ^^(i.e., ^^ ∈^1 / 2, 1 / 4, 1 / 8^) were trained simultaneously, while each projection neural network systemresponsible for each of the task neural networks corresponding to ^^^were trained individually for each value of ^^. The entry for ^^^ୀ^is the first neural network performance, Attorney Docket No.45288-0487WO1 ~ ~ while the entries of ^^^ୀ^,^^^ୀ^are the average performances over all ^^^and ^^^. In this ~ example, the model ^^ simultaneously projects ^^^ୀ^to all sizes s ∈ {1 / 2, 1 / 4, 1 / 8}. Each ^^^has it’s own projection operator, while the source model ^^^ୀ^is compatible with all of the projected model sizes. Co-training projection models of various sizes results in a final tuned model ^^^ୀ^, with projection operators for all s ∈ {1 / 2, 1 / 4, 1 / 8}, which provides a full spectrum of models: (a) a large model describing the full image distribution, and (b) a capability to produce small task-specific models with a variety of different sizes. In this example, the neural network ^^^ୀ^is an autoregressive transformer based model, having 24 layers, 8 heads and using the embedding dimension d = 512. The neural network ^^^ୀ^is pre-trained in this example on images from the IMAGENET dataset and images from a synthetic image dataset, for which images are generated by overlaying an image from the MNIST dataset on a background specified by the task identifier t. The number of trainable parameters in model ^^^ୀ^is approximately equal to 76.7M in this example. The task neural network ^^^, with the multiplier ^^ ∈ ^1 / 2, 1 / 4, 1 / 8^, uses the samearchitecture as ^^^ୀ^, but has a smaller embedding size sd. The number of parameters used by the projection operators ranged from 2.4M for the s = 1 / 2 model to 1.9M for s = 1 / 8 model in this example. An MLP architecture is used in this example to produce the P and Q projection matrices used to generate the task neural network ^^^. The input task identifier t in this example is first linearly mapped to an r-dimensional vector (where r may be chosen between 4 and 32 for example), which after a SWISH nonlinearity is followed by a linear layer, producing diagonal elements of P and Q as described above. In this example, r=8. To train the projection neural network systems, images from a synthetic image dataset generated by overlaying an image from the MNIST dataset on a background specified by the task identifier t were used in this example. In particular, to synthesize an image in the training dataset, the task identifier t is sampled uniformly from [0, 1]18. The task identifier t is then used to produce an image texture and one of the MNIST images overlaid on top of this texture. In this example, the 32 × 32 RGB images from the synthetic image dataset were quantized by separately mapping each pixel into one of 512 discrete tokens corresponding to one of the color clusters. These tokens were then flattened into 1024-long sequences. In an example, training may be performed for 200k to 800k steps with an Adam optimizer and a learning rate of order of 10−3with cosine learning rate decay schedule, and 10k warmup Attorney Docket No.45288-0487WO1 steps. A weak weight decay may be used (for example, 10−10to 10−8). Dropout may be set to 10% for example. Considering the table entry ^^^ୀ^(i.e. the first neural network) and the rowscorresponding to ^^ ∈ ^1 / 2, 1 / 4, 1 / 8^, the table of example 400 shows that the task neuralnetworks ^^^ୀ^ / ଶand ^^^ୀ^ / ସcreated using the described techniques perform better than or as well as the first neural network ^^^ୀ^using neural network architectures that include half or a quarter of the number of parameters as the first neural network architecture. Therefore, the described techniques can adequately maintain the performance of the first neural network for a task using significantly less parameters (and therefore significantly less computational cost). ~ At the same time, the table shows that for ^^^ୀ^the task neural network ^^^ୀ^ / ଶ~ created using the described techniques match the performance of ^^^ୀ^using half the number of parameters. Therefore, the described techniques can also adequately match the performance of the first neural network when it takes into account the information of the task input while still using significantly less parameters (and therefore significantly less cost). Considering the rows corresponding to ^^ ∈ ^1 / 2, 1 / 4, 1 / 8^, the describedtechniques always outperform the techniques of retraining a neural network. That is, the described techniques can generate task neural networks that better maintain the emergent ~ performance of ^^^ୀ^and ^^^ୀ^than techniques that plainly train neural networks with the ~ same architecture as ^^^ୀ^and ^^^ୀ^but with smaller embedding sizes. FIG.5 is an example 500 of the performance of the described techniques. Example 500 also shows a table that summarizes the performance of auto-regressive neural networks, that process image pixels to generate a predicted next image pixel to ultimately generate an image, generated using the described techniques (i.e., the columns ~ labeled ^^^) and other techniques (i.e., the columns labeled ^^^) in terms of average log loss (i.e., a metric of performance) over many inputs for many tasks. The notation of FIG.5 follows that of FIG.4 above, but the task input ^^ differs. In particular, for example 500, the task input ^^ is an embedding produced by a pretrained SIMCLRV2 model on the ImageNet ~ data set and therefore the neural networks ^^^and ^^^generate images with corresponding embedding ^^. Attorney Docket No.45288-0487WO1 In this example, to train the projection neural network systems, 64x64 RGB images from the IMAGENET dataset were used. In this example, a separate VQ-VAE model was trained for mapping 64 × 64 RGB images to sequences of 1024 tokens taking values in a discrete set of size 512. A VQ-VAE model such as described in Van Den Oord et al., 2017 may be used for example. The model encoder in this example contained 3 CNN layers with 3 × 3 kernels and leaky ReLU nonlinearities. Layers had the following depths and strides in this example: (32, 2), (64, 1), (64, 1). The decoder in this example was composed of 4 transpose-convolutional leaky ReLU layers. Layers had the following depths, kernel sizes and strides: (64, 3, 2), (64, 3, 1), (32, 3, 1) and the final (3, 1, 1). Considering the rows corresponding to ^^ ∈ ^1 / 2, 1 / 4, 1 / 8^, the describedtechniques always outperform the techniques of retraining a neural network. That is, the described techniques can generate task neural networks that better maintain the emergent ~ performance of ^^^ୀ^than techniques that plainly train neural networks with the same ~ architecture as ^^^ୀ^but with smaller embedding sizes. FIG.6 is an example 600 of the performance of the described techniques, highlighting that the task neural networks of the described techniques perform better the larger the first neural network is. The notation of FIG.6 (the below mentioned task neural networks, projection neural networks) follows that described above for FIG.4. Example 600 shows a table where each entry is a task neural network (i.e., the rows ^^^ୀ^ / ସand ^^^ୀ^ / ଼) generated from a first neural network (i.e., the columns ^^^ୀ^, ^^^ୀ^ / ଶ, ^^^ୀ^ / ସ) using projection neural network systems that include different numbers of trainableparameters (i.e., the row labeled ^^ ൌ with values of 8, 16, or 32, where higher values of ^^correspond to projection neural network systems with more trainable parameters). The entries of table of example 600 show that the performance of the task neural network is generally better for larger first neural networks and that this performance can be maintained using a small number of trainable parameters for the projection neural network systems. FIG.7 is an example 700 of the performance of the described techniques, highlighting how the described techniques generate task neural networks that exhibit “cross- task knowledge transfer” (i.e., training a projection neural network system using training examples belonging to various tasks boosts the performance of the resulting task neural network for a given task over training the projection neural network system using training Attorney Docket No.45288-0487WO1 examples just for the given task). The tasks and task neural networks of FIG.7 are those described for FIG.5 above. Example 700 shows a table where each entry is a pair of performance measures (a left and a right value) of the task neural network ^^^ୀ^ / ଶgenerated using projection neural network systems trained with different training examples. The left performance value corresponds to a first projection neural network system trained using a first set of training tasks and training examples (i.e., first training data), and the right performance value corresponds to a second projection neural network system trained using a second set of training tasks and training examples (i.e., second training data). The row labeled “1 / 2 – 1 / 2” refers to the first training data including ImageNet images with labels below 500 and the second training data including ImageNet images labels above 500. The row labeled “mod 2”refers to the first training data including ImageNet images with labels ^^ ≡ 0^^^^^^^ 2^ and thesecond training data including ImageNet images with labels ^^ ≡ 1^^^^^^^ 2^. SinceImageNet labels are not entirely random, but are ordered in a semantically meaningful way, tasks in first training data and second training data in the “mod 2” scenario are more closely related to each other compared to the “1 / 2 – 1 / 2” scenario. The column labeled “1” refers to using all training examples and training tasks for the first training data and the second training data. The column labeled “1 / 8” refers to using only 1 / 8 of all training examples and training tasks for the first training data and second training data. The column labeled “1 and 1 / 8” refers to using 1 / 8 of the training examples and training tasks for the first training data but using all training examples and training tasks for the second training data. Comparing the entries of the columns “1 / 8” to “1 and 1 / 8” shows that that providing additional training examples and training tasks in the second training data actually boosts the performance of task neural networks generated for tasks in the first training data. This boost is especially noticeable in the “mod 2” scenario, where tasks in first training data and second training data are closely related. Similarly, the performance on second training data suffers from a smaller training set for first training data. In other words, there is a noticeable interaction and knowledge transfer between task neural networks generated for different, but related tasks. FIG.8 is an example 800 of the performance of the described techniques, highlighting how the described techniques generate task neural networks that can generalize better than neural networks generated using other techniques. Attorney Docket No.45288-0487WO1 Example 800 shows three plots, and the neural networks and the task within these plots are those described for FIG.4 above. Each plot displays the performance (average loss) of neural networks in terms of the task input ^^ changing (i.e., each plot displays average loss as a function of the ‘scale’ component of the task input ^^ for various neural networks). Each plot displays the results of GPT (i.e., ^^^), Projected (i.e., ^^^) and ~ Conditional GPT (i.e., ^^^) and, from left to right, the plots display neural networkscorresponding to ^^ ൌ 1 / 2, ^^ ൌ 1 / 4 and ^^ ൌ 1 / 8.~ Because the training examples of ^^^and ^^^only include the values of ‘scale’ from0 to 1, the plot values to the right of x-axis value ^^^^^^^^^^ ൌ 0.5 display how well theseneural networks generalize to unseen tasks. In particular, example 800 shows that while allneural networks start degrading around ^^^^^^^^^^ ൌ 1, the described techniques’ task neural~ network ^^^ୀ^ / ଶshows better generalization than the Conditional GPT-2 model ^^^ୀ^ / ଶ. However, ^^^ୀ^ / ସand especially ^^^ୀ^ / ଼have much greater difficulty generalizing to^^^^^^^^^^ ^ 2, which could be explained by the fact that the corresponding projection neuralnetwork systems use a large number of parameters compared to the task neural network sizes. In this specification, the term "configured" is used in relation to computing systems and environments, as well as computer program components. A computing system or environment is considered "configured" to perform specific operations or actions when it possesses the necessary software, firmware, hardware, or a combination thereof, enabling it to carry out those operations or actions during operation. For instance, configuring a system might involve installing a software library with specific algorithms, updating firmware with new instructions for handling data, or adding a hardware component for enhanced processing capabilities. Similarly, one or more computer programs are "configured" to perform particular operations or actions when they contain instructions that, upon execution by a computing device or hardware, cause the device to perform those intended operations or actions. The embodiments and functional operations described in this specification can be implemented in various forms, including digital electronic circuitry, software, firmware, computer hardware (encompassing the disclosed structures and their structural equivalents), or any combination thereof. The subject matter can be realized as one or more computer programs, essentially modules of computer program instructions encoded on a tangible non- transitory storage medium for execution by or to control the operation of a computing Attorney Docket No.45288-0487WO1 device or hardware. The storage medium can be a storage device such as a hard drive or solid-state drive (SSD), a storage medium, a random or serial access memory device, or a combination of these. Additionally or alternatively, the program instructions can be encoded on a transmitted signal, such as a machine-generated electrical, optical, or electromagnetic signal, designed to carry information for transmission to a receiving device or system for execution by a computing device or hardware. Furthermore, implementations may leverage emerging technologies like quantum computing or neuromorphic computing for specific applications, and may be deployed in distributed or cloud-based environments where components reside on different machines or within a cloud infrastructure. The term "computing device or hardware" refers to the physical components involved in data processing and encompasses all types of devices and machines used for this purpose. Examples include processors or processing units, computers, multiple processors or computers working together, graphics processing units (GPUs), tensor processing units (TPUs), and specialized processing hardware such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). In addition to hardware, a computing device or hardware may also include code that creates an execution environment for computer programs. This code can take the form of processor firmware, a protocol stack, a database management system, an operating system, or a combination of these elements. Embodiments may particularly benefit from utilizing the parallel processing capabilities of GPUs, in a General-Purpose computing on Graphics Processing Units (GPGPU) context, where code specifically designed for GPU execution, often called kernels or shaders, is employed. Similarly, TPUs excel at running optimized tensor operations crucial for many machine learning algorithms. By leveraging these accelerators and their specialized programming models, the system can achieve significant speedups and efficiency gains for tasks involving artificial intelligence and machine learning, particularly in areas such as computer vision, natural language processing, and robotics. A computer program, also referred to as software, an application, a module, a script, code, or simply a program, can be written in any programming language, including compiled or interpreted languages, and declarative or procedural languages. It can be deployed in various forms, such as a standalone program, a module, a component, a subroutine, or any other unit suitable for use within a computing environment. A program may or may not correspond to a single file in a file system and can be stored in various ways. This includes being embedded within a file containing other programs or data (e.g., scripts within a markup language document), residing in a dedicated file, or distributed Attorney Docket No.45288-0487WO1 across multiple coordinated files (e.g., files storing modules, subprograms, or code segments). A computer program can be executed on a single computer or across multiple computers, whether located at a single site or distributed across multiple sites and interconnected through a data communication network. The specific implementation of the computer programs may involve a combination of traditional programming languages and specialized languages or libraries designed for GPGPU programming or TPU utilization, depending on the chosen hardware platform and desired performance characteristics. In this specification, the term "engine" broadly refers to a software-based system, subsystem, or process designed to perform one or more specific functions. An engine is typically implemented as one or more software modules or components installed on one or more computers, which can be located at a single site or distributed across multiple locations. In some instances, one or more dedicated computers may be used for a particular engine, while in other cases, multiple engines may operate concurrently on the same one or more computers. Examples of engine functions within the context of AI and machine learning could include data pre-processing and cleaning, feature engineering and extraction, model training and optimization, inference and prediction generation, and post-processing of results. The specific design and implementation of engines will depend on the overall architecture and the distribution of computational tasks across various hardware components, including CPUs, GPUs, TPUs, and other specialized processors. The processes and logic flows described in this specification can be executed by one or more programmable computers running one or more computer programs to perform functions by operating on input data and generating output. Additionally, graphics processing units (GPUs) and tensor processing units (TPUs) can be utilized to enable concurrent execution of aspects of these processes and logic flows, significantly accelerating performance. This approach offers significant advantages for computationally intensive tasks often found in AI and machine learning applications, such as matrix multiplications, convolutions, and other operations that exhibit a high degree of parallelism. By leveraging the parallel processing capabilities of GPUs and TPUs, significant speedups and efficiency gains compared to relying solely on CPUs can be achieved. Alternatively or in combination with programmable computers and specialized processors, these processes and logic flows can also be implemented using specialized processing hardware, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs), for even greater performance or energy efficiency in specific use cases. Attorney Docket No.45288-0487WO1 Computers capable of executing a computer program can be based on general- purpose microprocessors, special-purpose microprocessors, or a combination of both. They can also utilize any other type of central processing unit (CPU). Additionally, graphics processing units (GPUs), tensor processing units (TPUs), and other machine learning accelerators can be employed to enhance performance, particularly for tasks involving artificial intelligence and machine learning. These accelerators often work in conjunction with CPUs, handling specialized computations while the CPU manages overall system operations and other tasks. Typically, a CPU receives instructions and data from read-only memory (ROM), random access memory (RAM), or both. The elements of a computer include a CPU for executing instructions and one or more memory devices for storing instructions and data. The specific configuration of processing units and memory will depend on factors like the complexity of the AI model, the volume of data being processed, and the desired performance and latency requirements. Embodiments can be implemented on a wide range of computing platforms, from small embedded devices with limited resources to large-scale data center systems with high-performance computing capabilities. The system may include storage devices like hard drives, SSDs, or flash memory for persistent data storage. Computer-readable media suitable for storing computer program instructions and data encompass all forms of non-volatile memory, media, and memory devices. Examples include semiconductor memory devices such as read-only memory (ROM), solid-state drives (SSDs), and flash memory devices; hard disk drives (HDDs); optical media; and optical discs such as CDs, DVDs, and Blu-ray discs. The specific type of computer-readable media used will depend on factors such as the size of the data, access speed requirements, cost considerations, and the desired level of portability or permanence. To facilitate user interaction, embodiments of the subject matter described in this specification can be implemented on a computing device equipped with a display device, such as a liquid crystal display (LCD) or an organic light-emitting diode (OLED) display, for presenting information to the user. Input can be provided by the user through various means, including a keyboard), touchscreens, voice commands, gesture recognition, or other input modalities depending on the specific device and application. Additional input methods can include acoustic, speech, or tactile input, while feedback to the user can take the form of visual, auditory, or tactile feedback. Furthermore, computers can interact with users by exchanging documents with a user's device or application. This can involve sending web content or data in response to requests or sending and receiving text messages or other Attorney Docket No.45288-0487WO1 forms of messages through mobile devices or messaging platforms. The selection of input and output modalities will depend on the specific application and the desired form of user interaction. Machine learning models can be implemented and deployed using machine learning frameworks, such as TensorFlow or JAX. These frameworks offer comprehensive tools and libraries that facilitate the development, training, and deployment of machine learning models. Embodiments of the subject matter described in this specification can be implemented within a computing system comprising one or more components, depending on the specific application and requirements. These may include a back-end component, such as a back-end server or cloud-based infrastructure; an optional middleware component, such as a middleware server or application programming interface (API), to facilitate communication and data exchange; and a front-end component, such as a client device with a user interface, a web browser, or an app, through which a user can interact with the implemented subject matter. For instance, the described functionality could be implemented solely on a client device (e.g., for on-device machine learning) or deployed as a combination of front-end and back-end components for more complex applications. These components, when present, can be interconnected using any form or medium of digital data communication, such as a communication network like a local area network (LAN) or a wide area network (WAN) including the Internet. The specific system architecture and choice of components will depend on factors such as the scale of the application, the need for real-time processing, data security requirements, and the desired user experience. The computing system can include clients and servers that may be geographically separated and interact through a communication network. The specific type of network, such as a local area network (LAN), a wide area network (WAN), or the Internet, will depend on the reach and scale of the application. The client-server relationship is established through computer programs running on the respective computers and designed to communicate with each other using appropriate protocols. These protocols may include HTTP, TCP / IP, or other specialized protocols depending on the nature of the data being exchanged and the security requirements of the system. In certain embodiments, a server transmits data or instructions to a user's device, such as a computer, smartphone, or tablet, acting as a client. The client device can then process the received information, display results to the user, and potentially send data or feedback back to the server for further Attorney Docket No.45288-0487WO1 processing or storage. This allows for dynamic interactions between the user and the system, enabling a wide range of applications and functionalities. While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination. Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products. Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. What is claimed is:

Claims

Attorney Docket No.45288-0487WO1 CLAIMS 1. A method performed by one or more computers, the method comprising: obtaining a task input specifying a task to be performed by a neural network; processing the task input using a projection neural network system to generate an output that defines, for each of a plurality of weight matrices of a first neural network, a set of projection matrices; generating a task neural network for performing the task, wherein the task neural network has fewer parameters than the first neural network, and wherein the generating comprises, for each of the plurality of weight matrices of the first neural network, applying the projection matrices to the weight matrix to generate a projected weight matrix having a smaller size than the weight matrix; and providing data specifying the task neural network for performing the task.

2. The method of claim 1, wherein providing data specifying the task neural network for performing the task comprises: providing the task neural network for deployment on a user device.

3. The method of claim 2, wherein obtaining a task input specifying a task to be performed by a neural network comprises: receiving the task input from the user device; or generating the task input based on data received from the user device.

4. The method of claim 1, wherein the first neural network is maintained on a user device and wherein providing data specifying the task neural network for performing the task comprises deploying the task neural network on the user device.

5. The method of any preceding claim, further comprising: receiving a new input for the task; and processing the new input using the task neural network to generate an output for the task.

6. The method of any preceding claim wherein, for a given weight matrix of a first neural network that has size dn x dn, the set of projection matrices includes a first projection matrix that has size sdn x dn and a second projection matrix that has size dn x sdn.Attorney Docket No.45288-0487WO1 7. The method of claim 6, wherein applying the projection matrices to the weight matrix to generate a projected weight matrix having a smaller size than the weight matrix comprises, for the given weight matrix, generating a projected weight matrix by computing a product of (i) the first projection matrix, (ii) the given weight matrix, and (iii) the second projection matrix.

8. The method of any preceding claim, wherein each projection matrix in the set is defined by a respective vector and wherein the output comprises the respective vector for each of the projection matrices.

9. The method of any preceding claim, wherein the projection neural network system comprises a respective projection neural network for each of the plurality of weight matrices that is configured to process the task input to generate an output that defines the set of projection matrices for the weight matrix.

10. The method of claim 9, wherein the respective projection neural networks are multi- layer perceptrons (MLPs).

11. The method of claim 9 or claim 10, wherein the respective projection neural networks share at least some parameters.

12. The method of any preceding claim, wherein the projection neural network system has been trained on a training data set that includes, for each of a plurality of training tasks, a respective set of training examples.

13. The method of claim 12, wherein the task is one of the training tasks.

14. The method of claim 12 or claim 13, wherein the projection neural network system has been trained on an objective that measures, for each training task, a performance on the training examples for the training task of a training task neural network for the training task that has been generated from the first neural network using sets of projection matrices generated by the projection neural network system by processing training task data specifying the training task.

15. The method of claim 14, wherein the first neural network is also trained as part of the training of the projection neural network.Attorney Docket No.45288-0487WO1 16. The method of claim 15, wherein the first neural network has been pre-trained prior to being trained as part of the training of the projection neural network.

17. A method performed by one or more computers, the method comprising: receiving a new input for a task; and processing the new input using a task neural network for the task to generate an output for the task, wherein the task neural network has been generated from a first neural network by performing the respective operations of the method of any preceding claim.

18. A method performed by one or more computers, the method comprising: obtaining a set of training data that includes, for each of one or more training tasks, a respective set of training examples, each training example comprising a respective training input for the task; for each of the one or more tasks: processing a task input specifying the task using a projection neural network system to generate an output that defines, for each of a plurality of weight matrices of a first neural network, a set of projection matrices; generating a task neural network for performing the task, wherein the task neural network has fewer parameters than the first neural network, and wherein the generating comprises, for each of the plurality of weight matrices of the first neural network, applying the projection matrices to the weight matrix to generate a projected weight matrix having a smaller size than the weight matrix; and for each training example for the task, processing the respective training input in the training example using the task neural network to generate a task output for the training input; and training the projection neural network system on an objective that measures, for each training task, a performance on the training examples for the training task of the task output for the task.

19. The method of claim 18, further comprising: for each task and for one or more of the training examples for the task: processing the respective training input in the training example using the first neural network to generate another task output for the training input; and training the first neural network on an objective that measures, for each training task, a performance on the training examples for the training task of the other task outputs.Attorney Docket No.45288-0487WO1 20. The method of any preceding claim, wherein the first neural network has a decoder- only architecture.

21. The method of any one of claims 1-19, wherein the first neural network has an encoder-decoder architecture.

22. The method of any one of claims 1-19, wherein the first neural network has an encoder-only architecture.

23. The method of any preceding claim, wherein the first neural network comprises one or more self-attention layers.

24. The method of any preceding claim, wherein the first neural network comprises one or more cross-attention layers.

25. The method of any preceding claim, wherein the first neural network is a denoising neural network.

26. The method of any preceding claim, wherein the task is an image processing task.

27. The method of any preceding claim, wherein the task is a video processing task.

28. The method of any preceding claim, wherein the task is an audio processing task.

29. The method of any preceding claim, wherein the task is a multi-modal task that requires processing two or more of an image, video, audio, or text.

30. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one more computers to perform the operations of the respective method of any one of claims 1-29.

31. One or more computer storage media storing instructions that when executed by one or more computers cause the one more computers to perform the operations of the respective method of any one of claims 1-29.

Images