Generating paginated output sequences using language model neural networks

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating output sequence using a language model neural network. The output sequence includes a plurality of tokens that are each selected from a vocabulary of tokens. The vocabulary includes an end-of-page (EOP) token.

Description

[0001] Attorney Docket No.: 56113-0772WO1

[0002] GENERATING PAGINATED OUTPUT SEQUENCES USING LANGUAGE MODEL NEURAL NETWORKS

[0003] BACKGROUND

[0004]

[0001] This specification relates to generating output sequences using neural networks.

[0005]

[0002] Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to another layer in the network, e.g., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of weights.

[0006] SUMMARY

[0007]

[0003] This specification describes an output sequence generation system implemented as computer programs on one or more computers in one or more locations that generates an output sequence made up of tokens from a vocabulary by using an auto-regressive language model neural network. The vocabulary includes an end-of-page (EOP) token.

[0008]

[0004] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0009]

[0005] Existing systems that use an auto-regressive language model neural network to generate output sequences typically generate an output sequence by continuing to add tokens to the output sequence until the end of sequence (EOS) token is selected using a score distribution generated by the neural network or until a maximum number of tokens has been generated, whichever occurs first. In contrast, the output sequence generation system described in this specification can temporarily halt an ongoing output sequence generation process when an end-of-page (EOP) token is selected, and will continue the output sequence generation process only when a continuation signal is obtained.

[0010]

[0006] In particular, whenever the EOP token is selected from the vocabulary using a score distribution generated by the neural network, the described system makes a determination about whether to continue to generate the next tokens. The EOP token is thus different from the EOS token in that, while the selection of the EOS token will always cause the autoregressive token generation process to be terminated, the selection of the EOP token may not cause the auto-regressive token generation process to be terminated when the described system makes the determination to continue. Attorney Docket No.: 56113-0772WO1

[0011]

[0007] The described system can reduce compute and memory usage by early terminating the output sequence generation process, i.e., after the selection of the EOP token and before the selection of EOS token or reaching a maximum length of the output sequence, whilst maintaining the overall quality (e.g., accuracy or completeness) of the output sequences. The system effectively avoids spending extra computational resources to generate unnecessary tokens that may not be needed (e.g., read or looked at) by a user. The ability to early terminate the output sequence generation process after the selection of the EOP token also reduces the power consumption of a display of the client device and hence preserves battery life of the client device, because the display need not remain active for a prolonged period of time due to waiting for the additional tokens to be generated for presentation.

[0012]

[0008] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

[0013] BRIEF DESCRIPTION OF THE DRAWINGS

[0014]

[0009] FIG. 1 is a diagram of an example output sequence generation system.

[0015]

[0010] FIG. 2 is a diagram of an example training system.

[0016]

[0011] FIG. 3 is an illustration of an example user interface in which an incomplete output sequence can be presented.

[0017]

[0012] FIG. 4 is a flow diagram of an example process for generating an output sequence.

[0013] Like reference numbers and designations in the various drawings indicate like elements.

[0018] DETAILED DESCRIPTION

[0019]

[0014] FIG. 1 is a diagram of an example output sequence generation system 100. The output sequence generation system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented.

[0020]

[0015] The output sequence generation system 100 is a system that generates output sequences 132 in response to received requests 102.

[0021]

[0016] The output sequences 132 generated by the output sequence generation system 100 can be used in any of a variety of ways. For example, the system can provide the output sequences 132 for presentation to the user on a client device. As another example, the system Attorney Docket No.: 56113-0772WO1 can provide the output sequences 132 to another component in the system for further processing. As yet another example, the system can store the output sequences 132 in a data repository for some future purpose.

[0022]

[0017] As an example, the output sequence generation system 100 can generate text sequences, i.e., each output sequence 132 generated by the system is a sequence of tokens from a vocabulary of tokens that includes, e.g., one or more of characters, sub-words, words, punctuation marks, numbers, or other symbols that appear in natural language text. For example, the output sequence generation system 100 can generate text sequences in response to received requests 102 and provide the text sequences for presentation to users.

[0023]

[0018] As another example, the output sequence generation system 100 can generate images or videos that each have multiple frames (where each frame is an image) by generating images as sequences of pixels. For example, each output sequence 132 generated by the system is a sequence of color values for pixels in an image arranged according to a specified order. As a similar example, the output sequence generation system 100 can be a video generation system that generates video as a temporal sequence of video frames, where each frame is an output image. As another example, each output sequence 132 generated by the system is a sequence of tokens that represent image patch embeddings of an image which can then be processed by a decoder neural network to generate the image (pixel values).

[0024]

[0019] As another example, the output sequence generation system 100 can generate multimodal output sequences, i.e., each output sequence 132 generated by the system is a sequence of tokens that represents data in multiple data modalities, e.g., an output sequence of tokens that represents both audio and video data, an output sequence of tokens that represents both text and image data, or an output sequence of tokens that represents both text and audio data.

[0020] In some cases, the output sequence generation system 100 can receive a context input as part of or in association with the request 102 and generate an output sequence 132 that is a response to the context input. That is, the context input provides context for the output sequence 132.

[0025]

[0021] For example, the context input can be an input sequence of text and the output sequence is another sequence of text, e.g., a completion of the input sequence of text, a paraphrase of the input sequence of text, a response to a question posed in the input sequence, or a sequence of text that is about a topic specified by the input sequence of text. As another example, the context input can be an input other than text, e.g., an image, a video, an audio, or some combination of these, and the output sequence 132 can be text that describes the input. Attorney Docket No.: 56113-0772WO1

[0026]

[0022] As another example, the context input represents data to be compressed, e.g., image data, text data, audio data, or any other type of data; and the output sequence 132 is a compressed version of the data. The tokens included in the output sequence can include any representation of compressed data, e.g., symbols or embeddings to be decoded by a respective neural network.

[0027]

[0023] As a particular example, the output sequence generation system 100 can be part of a dialog system and the context input can include audio or text from the most recent conversational turn submitted by a user of the dialog system during the dialog while the output sequence 132 is the next turn in the conversation, e.g., either text or audio that is a response to the most recent conversational turn. Optionally, the context input can also include one or more historical conversational turns that occurred earlier in the conversation.

[0028]

[0024] As another particular example, the output sequence generation system 100 can be part of a machine translation system and the context input can include text in a source language while the output sequence 132 is text in a target language that is a translation of the source text into the target language.

[0029]

[0025] As another particular example, the output sequence generation system 100 can be part of a natural language processing system. For example, if the context input is a sequence of words in an original language, e.g., a sentence or phrase, the output sequence 132 can be a summary of the input sequence in the original language, i.e., a sequence that has fewer words than the input sequence but that retains the essential meaning of the input sequence. As another example, if the context input is a sequence of words that form a question, the output sequence 132 can be a sequence of words that form an answer to the question.

[0030]

[0026] As another particular example, the output sequence generation system 100 can be part of a computer-assisted medical diagnosis system. For example, the context input can be a sequence of data from an electronic medical record and the output sequences 132 can each be a sequence of predicted treatments.

[0031]

[0027] As another particular example, the output sequence generation system 100 can be part of a computer code generation system and the context input can be a text description of a desired piece of code or a snippet of computer code in a programming language and the output sequence 132 can be computer code, e.g., a snippet of code that is described by the context input or a snippet of code that follows the context input in a computer program.

[0032]

[0028] As another particular example, the output sequence generation system 100 can be part of a multi-modal system that processes, e.g., both text and image input sequences, or both text and audio input sequences, and generates the output sequences 132 that are either in a Attorney Docket No.: 56113-0772WO1 single data modality or in multiple data modalities, e.g., text and image output sequences, or text and audio output sequences. Examples of such multi-modal systems include an image captioning system, a text-based image search system, an image-based question answering system, and so on.

[0033]

[0029] As another particular example, the output sequence generation system 100 can be part of or associated with a search system that facilitates searching of resources on the Internet. A resource can be any data that can be provided over the Internet. A resource can be identified by a resource address that is associated with the resource. Resources include web pages, word processing documents, portable document format (PDF) documents, images, video, and news feed sources, to name a few.

[0034]

[0030] In this particular example, the search system can receive search queries submitted by client devices and, in response, identify resources that are relevant to the search query in the form of search results and return the search results to the user devices in search results pages. A search result page can include search result data generated by the search system that identifies a resource responsive to a search query, and includes a link to the resource. The search result page can additionally include a result in the form of an output sequence 132 that is generated by the output sequence generation system 100 based on a context input derived from the search query.

[0035]

[0031] To generate the output sequence 132, the output sequence generation system 100 uses a language model neural network 110. The language model neural network 110 can have any appropriate language model neural network architecture that allows the neural network to generate an output sequence 132 that is made up of tokens from a vocabulary 120.

[0036]

[0032] For example, the language model neural network 110 can have any of a variety of Transformer-based neural network architectures, e.g., encoder-only Transformer architectures, encoder-decoder Transformer architectures, decoder-only Transformer architectures, diffusion Transformer architectures, other attention-based architectures, and so on.

[0037]

[0033] Examples of such Transformer-based neural network architectures include those described in Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019; Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. Towards a human-like open-domain chatbot. CoRR, abs / 2001.09977, 2020; Tom B Brown, Attorney Docket No.: 56113-0772WO1

[0038] Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are fewshot learners. arXiv preprint arXiv:2005.14165, 2020; Aakanksha Chowdhery, et al. PaLM: Scaling Language Modeling with Pathways, arXiv preprint arXiv:2204.02311 ; Rohan Anil, et al. Palm 2 technical report. arXiv preprint arXiv:2305. 10403, 2023; and Gemini Team, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023).

[0039]

[0034] The language model neural network 110 operates auto-regressively; it executes an auto-regressive token generation process to auto-regressively generate the output sequence 132 of tokens by generating each particular token in the output sequence 132 conditioned on a current input sequence that includes any tokens that precede the particular token in the output sequence, i.e., the tokens that have already been generated for any previous positions in the output sequence that precede the particular position of the particular token, and in some cases, a context input that provides context for the output sequence.

[0040]

[0035] To generate a particular token at a particular position within an output sequence, the language model neural network 110 can process the current input sequence to generate a score distribution, e.g., a probability distribution, that assigns a respective score, e.g., a respective probability, to each token in the vocabulary 120 of tokens. The output sequence generation system 100 can then select, as the particular token, a token from the vocabulary 120 using the score distribution. For example, the output sequence generation system 100 can greedily select the highest-scoring token or can sample, e.g., using top-k sampling, nucleus sampling or another sampling technique, a token from the distribution.

[0041]

[0036] As a particular example, the language model neural network 110 can include a sequence of attention blocks followed by an output subnetwork, and, during the processing of the current input sequence, each attention block in the sequence receives a respective input hidden state for each token in the current input sequence. The attention block then updates at least the hidden state for the last token in the current input sequence at least in part by applying self-attention to generate an output hidden state for the last token.

[0042]

[0037] In some implementations, the attention block can make use of key-value (KV) caching or other caching techniques to avoid recomputing, at each time step, the respective output hidden states for tokens other than the last one in the current input sequence by storing the key and value vectors that have been generated by the attention block in a cache, such that they are later be reused. Attorney Docket No.: 56113-0772WO1

[0043]

[0038] The input hidden states for the first attention block are embeddings of the input tokens in the input sequence and the input hidden states for each subsequent attention block are the output hidden states generated by the preceding attention block. The output subnetwork processes the output hidden state generated by the last attention block in the sequence for the last token in the current input sequence to generate the score distribution.

[0044]

[0039] The vocabulary 120 of tokens can include any of a variety of tokens that represent text symbols or other symbols. For example, the vocabulary of tokens can include one or more of: characters, sub-words, words, punctuation marks, numbers, or other symbols that appear in a corpus of natural language text and / or computer code.

[0045]

[0040] Additionally or alternatively, the vocabulary 120 of tokens can include tokens that can represent data other than text. For example, the vocabulary of tokens can include image tokens that represent a discrete set of image patch embeddings of an image that can be generated by an image encoder neural network based on processing the image patches of the image. As another example, the vocabulary of tokens can include audio tokens that represent code vectors in a codebook of a quantizer, e.g., a residual vector quantizer.

[0046]

[0041] The vocabulary 120 also includes one or more additional tokens in addition to those described above. In particular, the vocabulary 120 includes an end-of-page (EOP) token 122.

[0042] The end-of-page (EOP) token 122 is a special token that, whenever selected by the output sequence generation system 100, causes the output sequence generation system 100 to temporarily halt the auto-regressive token generation process, such that no further tokens (beyond what have already been generated in the auto-regressive token generation process - or, stated differently - at subsequent positions within the output sequence after the particular position) will be generated by using the language model neural network 110 without a continuation signal being obtained.

[0047]

[0043] In practice, when the end-of-page (EOP) token is selected, the output sequence generation system 100 temporarily halts the auto-regressive token generation process, and determines whether to continue or terminate the auto-regressive token generation process while the auto-regressive token generation process is temporarily halted.

[0048]

[0044] If the system determines to continue the auto-regressive token generation process, then the system continues to add tokens to the output sequence as a continuation of what have already been generated in the auto-regressive token generation process so far, i.e., continues to generate tokens to occupy subsequent positions within the output sequence after the particular position, such that the output sequence 132 includes tokens generated by using the language model neural network 110 prior to when the EOP token has been selected, as well Attorney Docket No.: 56113-0772WO1 as one or more additional tokens generated by using the language model neural network 110 after when the EOP token has been selected.

[0049]

[0045] Alternatively, if the system determines to terminate the auto-regressive token generation process, then the tokens that have already been generated in the auto-regressive token generation process will be used as the output sequence 132, i.e., the system refrains from generating tokens to occupy subsequent positions within the output sequence after the particular position, such that the output sequence 132 includes only tokens generated by using the language model neural network 110 prior to when the EOP token has been selected.

[0050]

[0046] Note that the end-of-page (EOP) token is different from other special tokens which are commonly included in vocabulary 120 in addition to the EOP token. As used in this specification, a “special” token refers to a token that is in the vocabulary 120, but is not presented as part of the output sequence 132 even if selected.

[0051]

[0047] As a particular example, the EOP token is different from an end-of-sequence (EOS) token. In some implementations, the output sequence generation system 100 continues adding tokens to the output sequence until the EOS token has been selected. Thus, unlike the EOP token, the selection of the EOS token will always cause the auto-regressive token generation process to be terminated.

[0052]

[0048] As another particular example, the EOP token is also different from a separator (SEP) token. In some implementations, SEP tokens are used to separate one group of tokens from another group of tokens - for example to separate a first sentence from a second sentence within the output sequence. Thus, unlike the EOP token, even if the SEP token is selected, the auto-regressive token generation process automatically continues, i.e., the selection of the SEP token will not cause the auto-regressive token generation process to be temporarily halted.

[0053]

[0049] FIG. 2 is a diagram of an example training system 200. The training system 200 is an example of a system implemented as computer programs on one or more computers in one or more locations that trains the language model neural network 110 of FIG. 1 to determine the trained values of the parameters of the language model neural network 110.

[0054]

[0050] The training system 200 obtains a training dataset 210. The training dataset 210 includes a plurality of training sequences 214. Each training sequence includes a plurality of tokens. Each training sequence can be a part of an electronic document. An electronic document is data that presents a set of content at a client device. Examples of electronic documents include webpages, word processing documents, portable document format (PDF) documents, spreadsheets, images, videos, audios, and so on. Attorney Docket No.: 56113-0772WO1

[0055]

[0051] For example, the training sequences can include sequences that are obtained from a large dataset of text in one or more natural languages, e.g., text that is publicly available from the Internet or another text corpus. As another example, the training sequences can include sequences that are obtained from a large dataset of computer code in one or more programming languages, e.g., Python, C++, C#, Java, Ruby, PHP, and so on, e.g., computer code that is publicly available from the Internet or another code repository.

[0056]

[0052] The training system 200 includes a pre-processing engine 220. The pre-processing engine 220 pre-processes the training sequences 214 from the training dataset 210 by applying one or more pre-processing operations on each training sequence before the training sequences are used to train the language model neural network 110.

[0057]

[0053] That is, the training system 200 obtains a training sequence 214 from the training dataset 210, uses the pre-processing engine 220 to apply one or more pre-processing operations on the training sequence 214 to generate an augmented training sequence 224, and then uses the augmented training sequence 224 to train the language model neural network 110.

[0058]

[0054] The one or more pre-processing operations applied by the pre-processing engine 220 include inserting additional tokens into each training sequence. The additional tokens to be inserted into each training sequence include end-of-page (EOP) tokens.

[0059]

[0055] There are many different ways in which the EOP tokens can be inserted into each training sequence 214.

[0060]

[0056] Some electronic documents may have known structures or formats that are recognizable on a client device, e.g., the content of an electronic document (e.g., webpages, word processing documents, portable document format (PDF) documents, or spreadsheets) may have different paragraphs, pages, sections, frames, sheets, or other structural components, and the pre-processing engine 220 can utilize information about the known structures or formats to insert the EOP tokens into each of the training sequences generated from such electronic documents. For example, for a training sequence that spans from one page to another in such an electronic document, the pre-processing engine 220 can insert an EOP token to the training sequence after the last token on the first page.

[0061]

[0057] Some other electronic documents (e.g., images, videos, audios) may not have known structures or formats, and the pre-processing engine 220 can partition a training sequence generated from such electronic documents into N (being two or more) segments and insert an EOP token to the end of each segment of the training sequence. The segments within a training sequence may be, but need not be, of equal length. Attorney Docket No.: 56113-0772WO1

[0062]

[0058] After having generated the augmented training sequences 224, the training system 200 trains the language model neural network 110 on a language modeling objective on the augmented training sequences 224 to determine the trained values of the parameters of the language model neural network 110.

[0063]

[0059] More specifically, the training system 200 can train the language model neural network 110 on a next token prediction task based on minimizing a loss for a next token prediction task, where the next token prediction task requires the language model neural network 110 to predict the next token in an augmented training sequence 224 given the preceding tokens in the augmented training sequence 224. In effect, the training system 200 trains the language model neural network 110 to predict when the EOF token should be selected given the preceding tokens in the augmented training sequence 224.

[0064]

[0060] FIG. 3 is an illustration of an example user interface 300 in which an incomplete output sequence 310 can be presented. The incomplete output sequence 310 is generated by using the language model neural network 110 of FIG. 1. The user interface 300 can be presented on a display of a client device that is in data communication with the output sequence generation system 100 of FIG. 1.

[0065]

[0061] The user interface 300 presents an entiy window 320 for a user to enter a context input. In the example of FIG. 3, the user has entered “Write a description about the city of Mountain View, California.” and in response, the user interface 300 presents a display window 330 within which the incomplete output sequence 310 can be displayed.

[0066]

[0062] In the example of FIG. 3, the incomplete output sequence 310 displayed within the display window 330 includes two paragraphs: a first paragraph that begins with “Mountain View is located in. . .” and a second paragraph that begins with “Located 10 miles north of San Jose. . .” Each token included in the incomplete output sequence 310 is selected from the vocabulary 120 based on a score distribution generated by using the language model neural network 110.

[0067]

[0063] In some implementations, the tokens included in the incomplete output sequence 310 are displayed sequentially one after another, in the order in which they are generated by using the language model neural network 110 over a plurality of time steps. For example, “Mountain” is displayed within the display window 330 at a first time step, “Mountain View” is displayed within the display window 330 at a second time step, “Mountain View is” is displayed within the display window 330 at a second time step, and so on, in that order.

[0068]

[0064] After the last token in the incomplete output sequence 310, the language model neural network 110 generates a score distribution that results in the selection of an EOP token from Attorney Docket No.: 56113-0772WO1 the vocabulary 120. Thus, the user interface 300 displays a user interface element. An example of the user interface element is a system tray notification or pop-up window, which is shown at 340 in the user interface 300 of FIG. 3. As another example, the user interface element can be in the form of a button or other clickable user interface element.

[0069]

[0065] In some cases, the output sequence generation system 100 receives, through the user interface 300, an input corresponding to a selection by a user of the user interface element 340. The receipt of this input can enable the output sequence generation system 100 to resume the auto-regressive token generation process to continue to use the language model neural network 110 to generate one or more additional tokens to add to the incomplete output sequence 310.

[0070]

[0066] The one or more additional tokens will then be displayed within the display window 330, e g., as a third paragraph that is a continuation of the first two paragraphs included in the incomplete output sequence 310 that has already been displayed within the display window 330.

[0071]

[0067] In other cases, the output sequence generation system 100 does not receive any input corresponding to the selection by the user of the user interface element 340. Absent the receipt of the input, the output sequence generation system 100 refrains from continuing to use the language model neural network 110 to generate one or more additional tokens to add to the incomplete output sequence 310. Thus, no further tokens beyond what are already included in the first two paragraphs will be presented within the display window 330. That is, the incomplete output sequence 310 will be used as the final output sequence for presentation in response to the context input.

[0072]

[0068] FIG. 4 is a flow- diagram of an example process 400 for generating an output sequence. For convenience, the process 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, an output sequence generation system, e.g., the output sequence generation system 100 depicted in FIG. 1, appropriately programmed in accordance with this specification, can perform the process 400.

[0073]

[0069] The system receives a request for an output sequence from a user and, in response, performs the process 400 to generate the output sequence by using a language model neural network. The process 400 may also be viewed as an auto-regressive token generation process. The output sequence includes a respective token at each of a plurality of positions within the output sequence. In some cases, each position corresponds to a time step; that is, the output Attorney Docket No.: 56113-0772WO1 sequence is generated by the system across a plurality of time steps, with each time step corresponding to a respective one of the plurality of positions.

[0074]

[0070] Each respective token included in the output sequence is selected from a vocabulary of tokens. As described above, the vocabulary of tokens can include one or more of: characters, sub-words, words, punctuation marks, numbers, or other symbols that appear in a corpus of natural language text and / or computer code: image tokens that represent a discrete set of image patch embeddings of an image; audio tokens that represent code vectors in a codebook of a quantizer; and so on. The vocabulary of tokens can also include an end-of- page (EOP) token. The vocabulary of tokens can further include an end-of-sequence (EOS) token.

[0075]

[0071] The system uses the language model neural network to generate a score distribution for a particular position (step 402). To generate the score distribution for the particular position, the language model neural network processes at least the most recently selected token, i.e., the token that is selected for the immediately preceding position in the output sequence that precedes the particular position of the particular token. In some implementations, as part of the processing, the language model neural network reuses the respective input or output hidden states that have previously been generated for (i) the tokens that have already been generated for any further previous positions in the output sequence that further precede the immediately preceding position and, optionally, (ii) the context input that provides context for the output sequence.

[0076]

[0072] The score distribution generated by the language model neural network assigns a respective score to each token in the vocabulary' of tokens. For example, the score distribution can be a probability' distribution, and the respective score assigned to each token can be a respective probability score.

[0077]

[0073] The system uses the score distribution to select a token for the particular position (step 404). For example, the system can greedily select the highest-scoring token or can sample, e.g., using nucleus sampling or another sampling technique, a token from the distribution.

[0074] The system determines whether the selected token for the particular position is the end-of-sequence (EOS) token (step 406). That is, the system determines whether the EOS token is selected for the particular position based on the score distribution generated by the language model neural network for the particular position.

[0078]

[0075] If the selected token is the EOS token, the process 400 branches from step 406 to step 414. The system provides the tokens that have already been generated for some or all of the Attorney Docket No.: 56113-0772WO1 previous positions in the output sequence that precede the particular position for presentation on a client device to the user (step 414) and terminates the process 400 (step 416).

[0079]

[0076] That is, the selection of the EOS token causes the process 400 to be terminated, such that no additional tokens, let alone any additional output sequences, will be generated by the system without another request for another output sequence being received.

[0080]

[0077] For example, step 414 can involve transmitting from the system to the client device instructions that cause the client device to display the tokens. In some implementations, the system can perform the step 414 by presenting the tokens sequentially one after another, in the order in which they are generated by using the language model neural network. In some other implementations, the system can perform the step 414 by presenting the tokens in one shot, after the EOS token (indicating the end of the output sequence) is selected.

[0081]

[0078] If the selected token is not the EOS token, the process 400 proceeds from step 406 to step 408.

[0082]

[0079] The system determines whether the selected token is the end-of-page (EOP) token (step 408). That is. the system determines whether the EOP token is selected for the particular position based on the score distribution generated by the language model neural network for the particular position.

[0083]

[0080] If the selected token is not the EOP token, the process 400 returns from step 408 to step 402. In another step 402, the system generates, using the language model neural network, a score distribution for a next position after the particular position within the output sequence. In steps following the other step 402 in process 400, the next position after the particular position will become “the particular position” as discussed with reference to FIG. 4.

[0084]

[0081] To generate the score distribution for the next position, the language model neural network processes at least the selected token for the particular position (which is neither an EOS token nor an EOP token). In some implementations, as part of the processing, the language model neural network reuses the respective input or output hidden states that have previously been generated for (i) the tokens that have already been generated for any previous positions in the output sequence that precede the particular position and, optionally, (ii) the context input that provides context for the output sequence.

[0085]

[0082] If the selected token is the EOP token, the process 400 proceeds from step 408 to step 410.

[0086]

[0083] The system provides the tokens that have already been generated for some or all of the previous positions in the output sequence that precede the particular position for presentation on the client device to the user (step 410). In some implementations, the system Attorney Docket No.: 56113-0772WO1 can perform the step 410 by presenting the tokens sequentially one after another, in the order in which they are generated by using the language model neural network. In some other implementations, the system can perform the step 410 by presenting the tokens in one shot, after the EOP token is selected.

[0087]

[0084] The system determines whether to continue or terminate the process 400 (step 412). Such a determination can be made in many different ways. While the system is making this determination, the process 400 is temporarily halted, such that such that no additional tokens, let alone any additional output sequences, will be generated by the system before the determination on whether to continue has been made.

[0088]

[0085] In some implementations, the system can process the tokens that have already been generated for some or all of the previous positions in the output sequence that precede the particular position using a classification neural network to generate a classification output that represents a likelihood, e.g., a confidence score, that these tokens represent a complete response to the request, and determine whether to continue or terminate the process 400 based on the classification output - for example can determine to terminate the process 400 if the likelihood satisfies (e.g., is greater than) a threshold value.

[0089]

[0086] For example, the classification neural network can be a feed-forward neural network (e.g., a Transformer-based neural network) or a recurrent neural network which has been trained on a training dataset that includes a plurality of training inputs. Each training input can be in the form of a tuple < context input, output sequence > that is associated with a ground truth output indicating the completeness of the output sequence given the context input. For example, the ground truth output can be a score between 0 and 1, with 0 indicating incomplete and 1 indicating complete. The classification neural network can be trained based on optimizing a classification loss function that measures, for each training input, a difference between the ground truth output and a predicted output generated by the classification neural network based on processing the context input and the output sequence included in the training input.

[0090]

[0087] In some implementations, the system can transmit instructions to the client device that cause the client device to display a user interface element in a user interface within which the output sequence is presented. An example of the user interface element is a system tray notification or pop-up window; which is shown in an example user interface 300 of FIG. 3. Additionally or alternatively, the system can transmit instructions to the client device that cause the client device to play a verbal signal in the user interface, asking whether the user wants to continue or terminate the process 400. Attorney Docket No.: 56113-0772WO1

[0091]

[0088] For example, the system can then receive, from the client device, a user input interacting with the user interface element, e.g., a user input corresponding to a selection by the user of the user interface element. The receipt of the selection can be used as a user input requesting to continue the process 400. Alternatively, the lack of the receipt of the selection (e.g., within a predetermined amount of time since the display of the user interface element) can be used as a user input requesting to terminate the process 400.

[0092]

[0089] As another example, the system can receive a natural language speech input from the user, e.g., in response to the verbal signal played in the user interface. The natural language speech input may be received in the form of a sound (speech) signal, captured by a microphone of the client device, indicating whether the user wants to continue (e.g., a speech signal of “Yes, I want to see more / ’) or terminate (e.g., a speech signal of “No. this is good enough.”).

[0093]

[0090] As yet another example, the system can receive a text input from the user. The text input can be entered by ty ping using a data input device of the client device, indicating whether the user wants to continue (e.g., a text input of “Yes, I want to see more.”) or terminate (e.g., a text input of “No, this is good enough.”).

[0094]

[0091] Thus, in some implementations, the selection of the EOP token causes the system to present a user interface element (rather than an actual token from the vocabulary) for presentation to the user, and the system can determine whether to continue or terminate the process 400 based on the user input received in response to presenting the user interface element for presentation to the user.

[0095]

[0092] If the system determines to continue, the system generates a continuation signal, and the process 400 returns from step 412 to step 402. In another step 402, the system generates, using the language model neural network, a score distribution for a next position after the particular position within the output sequence. In steps following the other step 402 in process 400, the next position after the particular position will become “the particular position” as discussed with reference to FIG. 4.

[0096]

[0093] To generate the score distribution for the next position, the language model neural network processes at least the selected token for the particular position (which is the EOP token). In some implementations, as part of the processing, the language model neural network reuses the respective input or output hidden states that have previously been generated (and rather than generating new input or output hidden states) for (i) the tokens that have already been generated for any previous positions in the output sequence that precede Attorney Docket No.: 56113-0772WO1 the particular position and. optionally, (ii) the context input that provides context for the output sequence.

[0097]

[0094] In doing so, the process 400 resumes starting from where it is temporarily halted. The system continues to generate one or more additional tokens to add to the output sequence that is being generated in response to the request which is received prior to the beginning of process 400. The one or more additional tokens represent a continuation of what has already been generated in the process 400 so far.

[0098]

[0095] If the system determines to terminate, the process 400 proceeds from step 412 to step 416.

[0099]

[0096] The system terminates the process 400 (step 416). In doing so, the process 400 proceeds from being temporarily halted to being terminated. Thus, no additional tokens, let alone any additional output sequences, will be generated by the system without another request for another output sequence being received.

[0100]

[0097] Vary ing numbers of EOP tokens may be selected over different iterations of the process 400. Suppose that, during the process 400, two or more EOP tokens are selected before any EOS is selected and in response to the last EOP token, the system determines to terminate the process 400, then the tokens selected during the plurality of time steps will include two or more EOP tokens and no EOS token.

[0101]

[0098] Suppose that, during the process 400, an EOS token is selected after one or more EOP tokens are selected and in response to each of which the system determines to continue the process 400, then the tokens selected during the plurality of time steps will include one or more EOP tokens and only one EOS token.

[0102]

[0099] This specification uses the term ‘‘configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

[0103]

[0100] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Attorney Docket No.: 56113-0772WO1

[0104] Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine- readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0105]

[0101] The term “data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0106]

[0102] A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0107]

[0103] In this specification, the term “database"’ is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or structured at all, and it can be stored on storage devices in one or more locations. Thus, for example, the index Attorney Docket No.: 56113-0772WO1 database can include multiple collections of data, each of which may be organized and accessed differently.

[0108]

[0104] Similarly, in this specification the term “engine’’ is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

[0109]

[0105] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0110]

[0106] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0111]

[0107] Computer readable media suitable for storing computer program instructions and data include all forms of non volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.

[0112]

[0108] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the Attorney Docket No.: 56113-0772WO1 user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory' feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0113]

[0109] Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads.

[0114]

[0110] Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework or a Jax framework.

[0115]

[0111] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0116]

[0112] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g.. for purposes of displaying data to and receiving user input from a user interacting with the Attorney Docket No.: 56113-0772WO1 device, which acts as a client. Data generated at the user device, e.g.. a result of the user interaction, can be received at the server from the device.

[0117]

[0113] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0118]

[0114] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0119]

[0115] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

[0120]

[0116] What is claimed is:

Claims

1. Attorney Docket No.: 56113-0772WO1CLAIMS1. A method performed by one or more computers, the method comprising: receiving a request for an output sequence; generating the output sequence by performing an auto-regressive token generation process using a language model neural network, wherein the output sequence comprises a plurality of tokens that are each selected from a vocabulary of tokens that comprises an end- of-page (EOP) token, and wherein performing the auto-regressive token generation process comprises, at each of a plurality of time steps: generating, using the language model neural network, a score distribution that assigns a respective score to each token in the vocabulary of tokens; using the score distribution to select a token for the time step; and when the selected token is the EOP token: providing tokens preceding the EOP token for presentation to a user; and determining whether to continue or terminate the auto-regressive token generation process.

2. The method of claim 1, wherein the vocabulary comprises an end-of-sequence (EOS) token, and wherein performing the auto-regressive token generation process comprises, at each of the plurality of time steps: when the selected token is the EOS token: providing the tokens preceding the EOS token for presentation to the user; and terminating the auto-regressive token generation process.

3. The method of any one of claims 1-2, wherein the tokens selected during the plurality of time steps comprise two or more EOP tokens and no EOS token.

4. The method of any one of claims 1-2, wherein the tokens selected during the plurality of time steps comprise one or more EOP tokens and only one EOS token.

5. The method of any one of claims 1-4, wherein determining whether to continue or terminate the auto-regressive token generation process comprises: processing the tokens preceding the EOP token using a classification neural netw ork to generate a classification output that represents a likelihood that the tokens preceding the EOP token represent a complete response to the request; andAttorney Docket No.: 56113-0772WO1 determining whether to continue or terminate the auto-regressive token generation process based on the classification output.

6. The method of any one of claims 1 -4, wherein determining whether to continue or terminate the auto-regressive token generation process comprises: receiving a user input requesting to continue the auto-regressive token generation process.

7. The method of claim 6, wherein determining whether to continue or terminate the auto-regressive token generation process comprises: presenting a user interface element for presentation to the user, wherein the user input is an input interacting wi th the user interface element.

8. The method of any one of claims 1-7, further comprising training the auto-regressive language model neural network by: obtaining a training dataset, the training dataset comprising a plurality of training sequences, each training sequence comprising a plurality of tokens, the plurality of tokens comprising one or more EOP tokens; and training the auto-regressive language model neural network on the training dataset.

9. The method of claim 8, wherein obtaining the training dataset comprises: inserting one or more EOP tokens into each of the plurality of training sequences.

10. A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the respective operations of any one of the methods of any of the preceding claims 1-9.

11. One or more computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the respective operations of any one of the methods of any of the preceding claims 1-9.

Images