Jointly decoding output sequences using multiple language model neural networks

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output sequence using a plurality of auto-regressive language model neural networks. An output sequence includes a token at each of a plurality of positions. For any given position in the output sequence, the token to occupy the given position is generated based on the respective outputs generated by two or more of the plurality of auto-regressive language model neural networks.

Description

[0001] JOINTLY DECODING OUTPUT SEQUENCES USING MULTIPLE LANGUAGE MODEL NEURAL NETWORKS

[0002] BACKGROUND

[0003] This specification relates to processing inputs using neural networks to generate output sequences.

[0004] Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., another hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters.

[0005] SUMMARY

[0006] This specification describes a system implemented as computer programs on one or more computers in one or more locations that generates an output sequence using a plurality of auto-regressive language model neural networks. An output sequence includes a token at each of a plurality of positions. For any given position in the output sequence, the system generates the token to occupy the given position based on the respective outputs generated by two or more of the plurality of auto-regressive language model neural networks.

[0007] The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages.

[0008] Performing various collaborative tasks require using multiple auto-regressive language model neural networks, e.g., that are deployed on or correspond respectively to multiple mobile computing devices, to generate an output sequence. Some existing systems do this by collecting private data from across multiple client computing devices and then processing the collected private data using one or more neural networks to generate an output sequence (or a portion of the output sequence) in a centralized manner. However, transmission of private data off-device can present a security risk and in addition, incurs a network bandwidth cost.

[0009] The techniques described in this specification can enhance data security and user privacy when jointly and collaboratively generating an output sequence by using a plurality of language model neural networks in a distributed decoding system. By using each language model neural network to process respective private data and then determining each token to be included in the output sequence based on the outputs generated by the plurality7of language model neural networks, sharing any private data beyond the client computing device that the private data is local to can be avoided while still generating a high-quality' output sequence. For example, centrally' collecting private data from across multiple client computing devices at a cloud server can be avoided. Instead, the private data stored on each client computing device can be kept on-device and private. Moreover, network bandwidth can be conserved since the private data no longer needs to be transmitted from the client computing devices to the cloud server.

[0010] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

[0011] BRIEF DESCRIPTION OF THE DRAWINGS

[0012] FIG. 1 shows an example distributed decoding environment.

[0013] FIG. 2 is a flow diagram of an example process for generating an output sequence using a plurality of language model neural networks.

[0014] FIG. 3 is an example illustration of generating a token to occupy a particular position in an output sequence.

[0015] Like reference numbers and designations in the various drawings indicate like elements.

[0016] DETAILED DESCRIPTION

[0017] FIG. 1 shows an example distributed decoding system 100 that includes a server 102 and a plurality of separate client computing devices 104A-N. The server 102 is typically hosted within a data center, which can be a distributed computing system having hundreds or thousands of computers in one or more locations. Each client computing device 104A-N can be, for example, a desktop computer, laptop computer, a tablet computer, a wearable computer, a cellular phone, a smart phone, a music player, an e-book reader, a navigation system, a gaming device, a digital assistant device, an augmented reality device, a virtual reality7device, or any other appropriate computing device.

[0018] The distributed decoding system 100 includes a network 122, such as a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof. The network 122 connects the server 102 and the plurality' of client computing devices 104A-N. The distributed decoding system 100 includes a plurality' of language model neural networks 106A-N. Each language model neural network 106A-N can have any of a variety of Transformer-based neural network architectures, e.g., encoder-only Transformer architectures, encoder-decoder Transformer architectures, decoder-only Transformer architectures, other attention-based architectures, and so on.

[0019] Examples of such architectures include those described in Cohn Raffel. Noam Shazeer, Adam Roberts. Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou. Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: 1910. 10683, 2019; Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall. Noah Fiedel. Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. Towards a human-like opendomain chatbot. CoRR, abs / 2001.09977, 2020; Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165. 2020; Aakanksha Chowdhery. et al. PaLM: Scaling Language Modeling with Pathways. arXiv preprint arXiv:2204.02311 ; and Rohan Anil, et al. Palm 2 technical report. arXiv preprint arXiv:2305. 10403, 2023.

[0020] In some implementations, all of the plurality' of language model neural networks 106A-N have the same architecture and the same parameter values. For example, they can be multiple instances of the same language model neural network.

[0021] In some other implementations, the plurality' of language model neural networks 106A-N have the same architecture, but different parameter values. For example, the values of the parameters of each of some of the language model neural networks can have been adjusted, e.g., through supervised fine-tuning, reinforcement learning from human feedback, or another suitable training technique, based at least in part on private data that is local to a client computing device that maps to the language model neural network.

[0022] In yet other implementations, the architectures (and therefore, the parameter values) of two or more of the plurality of language model neural networks 106A-N differ from each other. For example, some language model neural networks are large-scale neural networks that each include a greater number of parameters while some other language model neural networks are lightweight versions of the large-scale neural network that each include a smaller, sometimes an order of magnitude smaller, number of parameters. As another example, some language model neural networks are mixture-of-experts (MoE) neural networks that each include one or more conditional computation layers while others are not. As yet another example, some language model neural networks each have multiple neural network layers arranged one after another in a sequential order while others each have multiple neural network layers arranged in parallel.

[0023] As illustrated in FIG. 1, in some implementations, each client computing device 104A-N implements a corresponding language model neural network 106A-N. That is, each language model neural network 106A-N is stored and implemented locally at a respective one of the plurality of client computing devices 104A-N.

[0024] For example, each client computing device 104A-N can include one or more data processing apparatuses and one or more computer storage media that provide the processing resources and memory resources needed to store and run a corresponding language model neural network 106A-N.

[0025] Since each language model neural network 106A-N is stored and implemented locally at a respective one of the plurality of client computing devices 104A-N, it may have access to private data that is local to the client computing device. However, it may have limited access to any private data that is local to another one of the plurality of client computing devices 104A-N and that is not shared beyond the client computing device. For example, private data that is local to a first client computing device may not be accessible to a second client device without authorization by the first client computing device.

[0026] In some other implementations, the server 102 implements the plurality of language model neural networks 106A-N. For example, the plurality of language model neural networks 106A-N are stored and implemented within a data center, which can be a distributed computing system having hundreds or thousands of computers in one or more locations. The server 102 can create sandbox environments or other isolated environments to store and run the plurality of language model neural networks 106A-N.

[0027] For example, the server 102 can create within it a first sandbox environment for implementing a first language model neural network 106A, a second sandbox environment for implementing a second language model neural network 106B, and so on.

[0028] Each sandbox environment can have a respective set of processing resources and memory resources allocated to the sandbox environment, and the language model neural network that is running in the sandbox environment may not have direct access to processing resources and memory7resource allocated to another sandbox environment. A language model neural network running in a sandbox environment may have access to private data that is stored within the sandbox environment; however it may have limited, if any, access to data stored outside of the sandbox environment, e.g., private data stored at another sandbox environment. Thus, because of the isolated nature of sandbox environments and the provisioning of sets of computing resources, a plurality of sandbox environments implemented on a single server may also be considered to be a distributed system.

[0029] Because there can be a corresponding sandbox environment that maps to each one of the plurality of client computing device 104A-N, for ease of description, the private data that is stored within a given sandbox environment will also be referred to as private data that is local to the client computing device that the maps to the given sandbox environment, despite that such private data may be kept at the server 102 and rather than at the client computing device.

[0030] In yet other implementations, some of the language model neural networks 106A-N are each stored and implemented locally at a respective one of the client computing device 104A-N while others of the plurality of language model neural networks 106A-N are each stored and implemented within a respective sandbox environment in the server 102.

[0031] Performing various collaborative tasks require using multiple auto-regressive language model neural networks, e.g., that are deployed on. or correspond respectively to, the mobile computing devices 104A-N. to generate an output sequence.

[0032] The plurality of language model neural networks 106A-N in the distributed decoding system 100 can jointly and collaboratively generate an output sequence made up of tokens from a vocabulary. The vocabulary of tokens can include any of a variety of tokens that represent text symbols or other symbols. For example, the vocabulary of tokens can include one or more of characters, sub-words, words, punctuation marks, numbers, or other symbols that appear in a corpus of natural language text and / or computer code.

[0033] Additionally, or alternatively, the vocabulary of tokens can include tokens that can represent data other than text. For example, the vocabulary of tokens can include image tokens that represent a discrete set of image patch embeddings of an image that can be generated by an image encoder neural network based on processing the image patches of the image. As another example, the vocabulary of tokens can include audio tokens that represent code vectors in a codebook of a quantizer, e.g., a residual vector quantizer.

[0034] As an example, the distributed decoding system 100 can generate text sequences, i.e., each output sequence generated by the plurality of language model neural networks 106A-N working in tandem is a sequence of text tokens from a vocabulary' of text tokens that includes, e.g., one or more of characters, sub-words, words, punctuation marks, numbers, or other symbols that appear in natural language text. For example, the server 102 can generate text sequences in response to received requests and provide the text sequences for presentation to users.

[0035] As another example, the distributed decoding system 100 can generate images or videos that each have multiple frames (where each frame is an image) by generating images as sequences of pixels. For example, each output sequence generated by the plurality of language model neural networks 106A-N is a sequence of color values for pixels in an image arranged according to a specified order. As another example, each output sequence generated by the plurality of language model neural networks 106A-N is a sequence of tokens that represent image patch embeddings of an image which can then be processed by a decoder neural network to generate the image (pixel values).

[0036] In some cases, the distributed decoding system 100 can receive a context input as part of the request and generate an output sequence that is a response to the context input.

[0037] For example, the context input can be an input sequence of text and the output sequence is another sequence of text, e.g., a completion of the input sequence of text, a paraphrase of the input sequence of text, a response to a question posed in the input sequence, or a sequence of text that is about a topic specified by the input sequence of text. As another example, the context input can be an input other than text, e.g., an image, and the output sequence can be text that describes the input.

[0038] As another example, the context input represents data to be compressed, e.g., image data, text data, audio data, or any other type of data; and the output sequence is a compressed version of the data. The tokens included in the output sequence can include any representation of compressed data, e.g., symbols or embeddings to be decoded by a respective neural network.

[0039] As a particular example, the server 102 can implement a dialog system and the context input can include audio or text from the most recent conversational turn submitted by a user of the dialog system during the dialog while the output sequence is the next turn in the conversation, e.g., either text or audio that is a response to the most recent conversational turn. Optionally, the context input can also include one or more historical conversational turns that occurred earlier in the conversation.

[0040] As another particular example, the server 102 can implement a machine translation system and the context input can include text in a source language while the output sequence is text in a target language that is a translation of the source text into the target language.

[0041] As another particular example, the server 102 can implement a computer code generation system and the context input can be a text description of a desired piece of code or a snippet of computer code in a programming language and the output sequence can be computer code, e.g., a snippet of code that is described by the context input or a snippet of code that follows the context input in a computer program.

[0042] In particular, the output sequence that is generated by the plurality of language model neural networks 106A-N working in tandem includes a token at each of a plurality of positions. For any given position in the output sequence, the distributed decoding system 100 generates the token to occupy the given position based on the respective outputs generated by two or more of the plurality of language model neural networks 106A-N.

[0043] A user may utilize a software application (or “application” for short) installed on one of the plurality of client computing devices 104A-N to perform collaborative tasks with each of a group of other users who similarly has access to another one of the plurality of client computing devices 104 A-N.

[0044] Examples of such collaborative tasks include: arriving at a decision together while considering user-specific data (e.g., individual preferences) of each person, e g., where the application is a social media application, a collaborative chat application, and so on; composing a co-written document, e.g., where the application is a collaborative slide presentation application, a collaborative text processing application, a collaborative spreadsheet application, and so on; generating some content which is tailored to a number of people (e.g., content in a particular language, content suitable for a particular reading level, etc.), e.g., where the application is a book application, an education application, an entertainment application, and so on; and creating new training data together (e.g., for collaborative machine learning), e.g., where the application is a machine learning application; to name just a few.

[0045] To perform such collaborative tasks that involve processing private data local to each client computing device while delivering better security and privacy guarantees, for each of the plurality of client computing devices 104 A-N, the server 102 uses a corresponding language model neural network to generate one or more outputs based on processing private data that is local to the client computing device, and generates the output sequence based on a combination of the one or more outputs generated by the plurality of language model neural networks 106A-N.

[0046] In this way, the distributed decoding system 100 can jointly and collaboratively generate an output sequence based on respective private data that is local to each of the plurality of client computing devices 104A-N without having to share the private data beyond the client computing device that the private data is local to. Instead, the private data can be kept private. Because sharing private data across the server 102 and the plurality of client computing devices 104A-N is avoided, data security and user privacy can be enhanced when performing these collaborative tasks with the use of language model neural networks. Moreover, network bandwidth that is otherwise needed to transmit the private data can be preserved.

[0047] More specifically, the jointly and collaboratively generated output sequence can be created by generating each particular token in the output sequence conditioned on a current output sequence that includes any tokens that precede the particular token in the output sequence, i.e., the tokens that have already been generated for any previous positions in the output sequence that precede the particular position of the particular token, and a context input that provides context for the output sequence.

[0048] To generate a particular token at a particular position within an output sequence, the server 102 determines a subset of the plurality of language model neural networks 106A-N and then, for each language model neural netw ork in the subset, causes processing of the current output sequence using the language model neural network to generate a score distribution, e.g.. a probability distribution, that assigns a respective score, e.g.. a respective probability, to each token in the vocabulary of tokens.

[0049] That is, to generate the particular token at the particular position, the server 102 provides at least a portion of the current output sequence as input to each language model neural network in the subset and then receives, from each language model neural network in the subset, a score distribution that has been generated by the language model neural network from processing at least the current output sequence.

[0050] Suppose that the language model neural network 106A is in the subset, FIG. 1 thus illustrates that at least a portion of a current output sequence 112 that includes any tokens that precede the particular token in the output sequence is transmitted over the netw ork 122 from the server 102 to the client computing devices 104 A which implements the language model neural netw ork 106A.

[0051] Then, a score distribution 114 that has been generated by the language model neural network 106A from processing (i) the current output sequence 112 that is transmitted over the network 122 from the client computing devices 104A to the server 102 and, optionally, (ii) private data or data derived from the private data that is local to the client computing device 104A.

[0052] The server 102 can then select, as the particular token, a token from the vocabulary using a combined score distribution that is generated based on the respective score distributions generated by language model neural networks in the subset. For example, the server 102 can greedily select the highest-scoring token or can sample, e.g., using nucleus sampling or another sampling technique, a token from the combined score distribution.

[0053] FIG. 2 is a flow diagram of an example process 200 for generating an output sequence using a plurality of language model neural networks. For convenience, the process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a server, e.g.. the server 102 of FIG. 1. appropriately programmed, can perform the process 200.

[0054] Prior to the first iteration of the process 200, the server receives a request for an output sequence. For example, the request can be received from a user of a client computing device. In some implementations, the request also includes a context input. The output sequence includes a respective token at each of a plurality of positions.

[0055] The server can generate the output sequence by repeatedly performing an iteration of the process 200 at each of the plurality of positions of the output sequence (referred to below as a ‘"particular” position in the plurality of positions) to determine a token to occupy the position.

[0056] The server determines a subset of language model neural networks from a plurality of language model neural networks (step 202). Each language model neural network maps to a respective one of the plurality of client computing devices. The plurality of client computing devices are in data communication with the server over a network. For example, the server can select at least two and, in some cases, fewer than all, of the plurality of language model neural networks.

[0057] In some implementations, each client computing device locally stores and implements a corresponding language model neural network. In some other implementations, the server implements the plurality of language model neural networks. For example, the server has a sandbox environment or another isolated environment that corresponds to each of the plurality of client computing devices, and implements a language model neural network within each sandbox environment. In yet other implementations, some of the language model neural networks are each stored and implemented locally at a respective one of the client computing device while others of the plurality of language model neural networks are each stored and implemented within a respective sandbox environment in the server.

[0058] The language model neural networks to be included in the subset can be determined in any of a variety of ways. As an example, when the request identifies which client computing devices should be used to generate the output sequence, the server can select the language model neural networks that map to the identified client computing devices as the language model neural networks to be included in the subset. In this example, the same subset of language model neural networks may be selected for different positions in the plurality of positions of the output sequence.

[0059] As another example, the server can select the language model neural networks in a round-robin, a weighted round-robin, or a random manner. In this example, different subsets of language model neural networks may be selected for different positions in the plurality of positions of the output sequence (but the number of client computing devices to be selected for inclusion in the subset for different positions will generally be fixed).

[0060] The server provides data that specifies at least a portion of a current output sequence to each language model neural network in the subset (step 204). The current output sequence includes any tokens that precede the particular token in the output sequence, i.e., the tokens that have already been generated for any previous positions in the output sequence that precede the particular position of the particular token.

[0061] In some implementations, the server provides a copy of the entire current output sequence to each language model neural network in the subset. For example, when each client computing device locally stores and implements a corresponding language model neural netw ork, the server can transmit the current output sequence to each client computing device in the subset over the network.

[0062] In some other implementations, a language model neural network in the subset may- have stored a copy of the previously generated tokens in the output sequence which they can reuse later. Thus, in those other implementations, the server need only provide a portion of the current output sequence that is not included in the stored copy of previously generated tokens to a language model neural network in the subset. For example, the server can provide a preceding token that occupies a preceding position that immediately precedes the particular position in the plurality of positions to each language model neural network in the subset.

[0063] The serv er obtains data that specifies a combination of respective score distributions that have been generated by the language model neural networks in the subset (step 206). Each language model neural network in the subset can be configured to generate a score distribution, e g., a probability distribution, that assigns a respective score, e.g., a respectiv e probability, to each token in the vocabulary of tokens by processing an input sequence.

[0064] For example, the input sequence can be of the form:

[0065] <context input> <current output sequence> where “<context input>’' represents the context input that provides context for the output sequence, and ”<current output sequence^" represents the current output sequence that includes any tokens that precede the particular token in the output sequence, i.e., the tokens that have already been generated for any previous positions in the output sequence that precede the particular position. Potentially, these preceding tokens have been generated byusing different subsets of the plurality of language model neural networks.

[0066] The context input can include any data, including data derived from private data that is local to each client computing device. For example, the context input can be, or be derived (generated) from, a user input received in association with or as part of the request. The user input can include a keyboard input, a mouse input, a touch input, a voice input, etc. As another example, the context input can include a predetermined sequence of tokens that is stored locally at a client computing device. As yet another example, the context input can include data retrieved from a customized database, e.g., by using retrieval-augmented generation (RAG) techniques, and the customized database is local (or private) to the client computing device.

[0067] At least a portion of the current output sequence, e.g.. the preceding token that occupies a preceding position that immediately precedes the particular position in the plurality of positions, is provided to the language model neural network in the subset by the server.

[0068] Thus, the context inputs to be processed by different language model neural networks (e.g., that correspond to different client computing devices) may differ from each other. However, the current output sequences to be processed by different language model neural networks may be the same.

[0069] In implementations where each client computing device locally stores and implements a corresponding language model neural network, the data that specifies the combination of respective score distributions can be received by the server in any appropriate manner. As a general example, each client computing device can transmit the data that specifies a score distribution generated by a corresponding language model neural network, e.g., in either an unencrypted or encrypted form, to the server — and the server combines the received score distributions, optionally after first decrypting the received data, to generate the combination of respective score distributions.

[0070] For example, to enhance data security and user privacy, each client computing device can cooperate with each other by using a secure aggregation technique, a differential privacy technique, or both to generate the combination of the respective score distributions that have been generated by the language model neural networks, and then transmit the combination of the respective score distributions to the server.

[0071] Secure aggregation refers to aggregating, e.g., summing or averaging, data values from multiple sources (client computing devices) without disclosing these individual data values to the destination (server). Typically, secure aggregation involves each of the plurality' of client computing devices holding a cryptographic public / private key pair, and sharing its public key in the pair with at least another client computing device. Data that specifies a score distribution to be communicated from a first client computing device to a second client computing device will be encrypted by using the public key of the second client computing device, and signed by using the private key of the first client computing device.

[0072] Examples of suitable secure aggregation techniques that can be used to combine the respective score distributions include those described in Aaron Segal, et al. Practical secure aggregation for privacy-preserving machine learning. CCS. https: / / eprint.iacr.org / 2017 / 281.pdf (2017), and Mohamad Mansouri, et al. Sok: Secure aggregation based on cryptographic schemes for federated learning. Proceedings on Privacy Enhancing Technologies (2023). to name just a few.

[0073] Differential privacy involves adding calibrated noise and / or applying clipping on the data values from the client computing devices before aggregating the data values, so as to provide mathematically rigorous privacy guarantees of the data values.

[0074] Examples of suitable differential privacy techniques that can be used to combine the respective score distributions include those described in Naman Agarwal, et al. The skellam mechanism for differentially private federated learning. Advances in Neural Information Processing Systems 34 (2021): 5052-5064, Galen Andrew, et al. Differentially private learning with adaptive clipping. Advances in Neural Information Processing Systems 34 (2021): 17455-17466, and Peter Kairouz, et al. The distributed discrete gaussian mechanism for federated learning with secure aggregation. International Conference on Machine Learning. PMLR, 2021.

[0075] In practice one or more of the secure aggregation techniques and one or more of the differential privacy techniques can used together by the system to further enhance data security and user privacy.

[0076] FIG. 3 is an example illustration 300 of generating a token to occupy a particular position in an output sequence. FIG. 3 illustrates that there are a total of four client computing device devices A-D in the subset. Each client computing device locally implements a language model neural network that generates a distribution over a vocabulary of tokens. The distribution includes a score assigned to each token in the vocabulary. For ease of illustration, FIG. 3 shows that the vocabulary has four tokens; in practice, however, the vocabulary can be orders of magnitude larger, e.g., it can include 30,000 tokens, 60,000 tokens, 100,000 tokens, or more.

[0077] A language model neural network generates such a distribution based on processing an input sequence that includes a context input and a current output sequence in accordance with the parameters of the language model neural network. Therefore, because (i) the context inputs or (ii) the parameters or both (i) and (ii) of the client computing device devices A-D may differ, the distributions generated by the client computing device devices A-D may also differ, i.e., they may include different scores assigned to the same token in the vocabulary.

[0078] The server selects a token from the vocabulary of tokens based on the combination of the respective score distributions that have been generated by using the language model neural networks in the subset (step 208). The selected token can then be used as the token to occupy the particular position.

[0079] In FIG. 3, for example, the combination is a combined score distribution where the score assigned to each token in the vocabulary is an average of the scores that are assigned to the token and that are included in the respective score distributions generated by the client computing device devices A-D. The server uses the combined score distribution to select the token to occupy the particular position. For example, the server can greedily select the highest-scoring token or can sample, e.g., using nucleus sampling or another sampling technique, a token from the combined score distribution.

[0080] For each of the plurality of positions of the output sequence, the server can repeatedly (i.e., at each of the plurality of positions) perform the process 200 to select, from the vocabulary of tokens, a selected token to occupy the position. That is, by repeatedly performing the process 200 for all of the plurality of positions, the server can generate an output sequence in response to the received request by using a plurality of language model neural networks.

[0081] The output sequence generated by the server can be used in any of a variety of ways. For example, the server can provide the output sequences for presentation to the user on the client computing device. As another example, the server can provide the output sequence to another component in the distributed decoding environment for further processing. As yet another example, the server can store the output sequence in a data repository for some future purpose. This specification uses the term “configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

[0082] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of. data processing apparatus. The computer storage medium can be a machine- readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0083] The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g.. code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0084] A computer program, which may also be referred to or described as a program, software, a software application, an app. a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub-programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0085] In this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

[0086] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g.. an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0087] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory’ or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to. or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0088] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0089] Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads.

[0090] Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework or a JAX framework.

[0091] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g.. a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area netw ork (WAN), e.g.. the Internet.

[0092] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication netw ork. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g.. for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.

[0093] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, vanous features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0094] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0095] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

[0096] What is claimed is:

Claims

CLAIMS1. A method performed by a computing system, the method comprising: receiving, by a server of the computing system, a request for an output sequence, wherein the output sequence comprises a respective token at each of a plurality of positions; generating the output sequence, wherein generating the output sequence comprises, for a particular position in the plurality of positions: determining, by the server, a subset of language model neural networks from a plurality of language model neural networks; providing, by the server, data that specifies at least a preceding token that occupies a preceding position that precedes the particular position in the plurality of positions to each language model neural network in the subset; obtaining, by the server, data that specifies a combination of respective distributions generated by the language model neural networks in the subset, wherein the respective distributions are generated based upon processing, by each language model neural network in the subset, at least the preceding token using respective the language model neural network; and selecting, by the server, a selected token as the token to occupy the particular position from a vocabulary of tokens based on the combination of the respective distributions that have been generated by using the language model neural networks in the subset; and providing, by the server, the output sequence in response to the request.

2. The method of claim 1, wherein generating the output sequence comprises, for the particular position in the plurality of positions further comprises: processing, by each language model neural network in the subset, at least the preceding token to generate a respective distribution over the vocabulary of tokens.

3. The method of claim 1 or 2, wherein the combination of the respective distributions is based upon a secure aggregation technique.

4. The method of claim 3, wherein the secure aggregation technique uses a cryptographic key pair that is held by each of a plurality of client computing devices.

5. The method of any one of claims 1-4, wherein the language model neural networks have a same architecture and same parameter values.

6. The method of any one of claims 1-4, wherein the language model neural networks have different architectures or different parameter values.

7. The method of claim 6, wherein the parameters of the language model neural network have been adjusted based at least on private data that is local to a client computing device on which the language model neural network is implemented.

8. The method of any one of claims 1-7, wherein processing, by each language model neural network in the subset, at least the preceding token using the respective language model neural network comprises processing a context input and the preceding token, and wherein the context input is different for at least two of the language model neural networks in the subset .

9. The method of claim 8, wherein the context input comprises data derived from private data that is specific to the language model neural network.

10. A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the respective operations of any one of the methods of any of the preceding claims 1 -9.

11. One or more computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the respective operations of any one of the methods of any of the preceding claims 1-9.

Images