Refining prompts for generative neural networks

Abstract

System, comprising: one or more computers;and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising: Receiving a prompt to a generative neural network, the prompt comprising a text sequence of text tokens; Generating a language model input for a language model neural network from the prompt; Processing the language model input using the neural language model network to generate a language model output that (i) identifies one or more initial text segments from the text sequence and (ii) includes, for each of the identified initial text segments, one or more initial candidate refinements for the text segment, wherein each of the one or more initial text segments comprises a respective correct subset of the text tokens in the text sequence; Identifying, using the language model output, (i) one or more final text segments from the text sequence and (ii) for each of the final text segments, one or more final candidate refinements for the final text segment; and Providing data identifying the one or more final candidate refinements for the final text segments for display in a user interface of a user device, the user interface enabling a user to generate a modified prompt by replacing one or more of the final text segments with one of the final candidate refinements for the final text segment.

Description

BACKGROUND

[0001] This description refers to processing inputs using neural networks.

[0002] Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output from a received input. Some neural networks contain one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network—that is, another hidden layer, or the output layer. Each layer of the network generates an output from a received input according to the current values ​​of a respective set of parameters. SUMMARY

[0003] This specification describes a system, implemented as computer programs on one or more computers at one or more locations, that enables a user to refine a prompt to a generative neural network—that is, to modify one or more text segments in an initial prompt submitted by a user. After the prompt has been refined, the refined prompt can be provided as input to the generative neural network, which uses the refined prompt to generate a data item.

[0004] Certain embodiments of the subject matter described in this specification may be implemented to achieve one or more of the following advantages.

[0005] Many existing systems allow users to send requests to an interface with one or more generative models. For example, some systems allow users to submit requests through a user interface, which are then provided as input to a generative model. As another example, some systems allow users to access a generative model through an application programming interface (API).

[0006] However, the quality of the data item generated by a given generative model can vary greatly between different inputs ("prompts") and even between semantically similar prompts. Furthermore, it is often unclear to users how a prompt should be formatted for the generative model. Since generating a data item using a generative model is often computationally expensive and can introduce significant latency, generating a high-quality data item may require generating many different candidate data items in response to many different prompts, consuming a large number of computational resources and degrading the user experience.

[0007] Various existing approaches attempt to assist users in generating prompts that can be effectively processed by a generative model, that is, in generating prompts that, when processed by the generative model, cause the generative model to produce a high-quality output data item.

[0008] For example, prompt rewriting is a technique that automatically transforms a user's input into a generative model to improve the quality of the model's output or to account for properties such as diversity. This modifies the entire prompt rather than allowing for detailed exploration and discovery, and is often invisible to the user. That is, a process running in the "background" rewrites or augments a user prompt and provides the rewritten prompt as input to the model, without requiring any further input from the user. The user therefore receives little to no feedback on how to optimize their interaction with the generative model.

[0009] As another example, some techniques allow users to select one of several preset options, each corresponding to a different prompt for the generative model. This technique guides users toward inputs that may be technically feasible and creatively interesting. However, this does not work with the user's own freehand input and is inherently limited to the predetermined elaboration options, thus limiting the user's ability to interact with the generative model (because although the generative model can respond to any appropriate free-text prompt, the user is restricted to choosing from a relatively small set of preset options).

[0010] This specification describes techniques that address these deficiencies of these and other techniques and solve the problem for the user by providing the user with an option to refine individual segments of the prompt with prompt-specific alternatives. This can guide a user toward deeper, broader, or more model-appropriate inputs for arbitrary concepts. In particular, the described techniques utilize a neural language model network to suggest refinements for each of the one or more segments of the prompt and allow users to refine the prompt using the suggested refinements.

[0011] For example, a user input "Photorealistic woman with elaborate earrings, front-lit, full-body portrait, hyperrealistic, Rembrandt lighting" might offer the described techniques "Surrealized / Abstract / Impressionistic" as alternatives to "Photorealistic," terms that may be creatively unfamiliar to a user but could be suitable as inputs for a given model. In the same input, the described techniques might offer "Split / Broad / Butterfly" as alternative options to "Rembrandt" for the lighting types.

[0012] Thus, the described techniques provide a transparent and flexible way to improve the interaction between user and generative model by allowing users to flexibly refine sections of prompts in a transparent manner to effectively explore the space of possible prompts that can lead to a high-quality data item.

[0013] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below.

[0014] Further features, aspects and advantages are apparent from the description, drawings and claims. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 is a diagram of an example prompt refinement system. Fig. Figure 2 is a flowchart of an example prompt refinement process. Fig. Figure 3A shows an example of a user interface. Fig. Figure 3B shows the example of the user interface after a user submits input to select an identified text segment.

[0015] Like reference numerals in the different drawings indicate like elements. DETAILED DESCRIPTION

[0016] Fig. 1 shows an example of a prompt refinement system 100. The prompt refinement system 100 is an example of a system implemented as computer programs on one or more computers at one or more locations in which the systems, components, and techniques described below may be implemented.

[0017] The prompt refinement system 100 is a system that interfaces between a user 102 of a user device 104 and a generative neural network 110.

[0018] In general, the prompt refinement system 100 receives a prompt 120 from the user device 104 to the generative neural network 130.

[0019] Prompt 120 is a text prompt that includes a sequence of text tokens. Each text token is one of a vocabulary of text tokens, each representing a respective text unit, e.g., a set of tokens that includes words, characters, word parts, or other text symbols.

[0020] That is, the user 102 sends a request for a data item to be generated by the generative neural network 130 via the user device 104. The request includes a prompt, ie, the prompt 120, which describes the desired content of the requested data item.

[0021] The generative neural network 130 may be any suitable generative neural network that generates a data item by processing an input that includes a request. A "data item" is a content item of a corresponding type. For example, a data item may be an image, an audio signal representing, for example, speech, music, or both, a video, etc.

[0022] For example, the generative neural network 130 may be an image generation neural network that generates images in response to user input. Examples of such neural networks include diffusion models and autoregressive image generation neural networks. As specific examples, the generative neural network 130 may be the Parti model described in "Scaling Autoregressive Models for Content-Rich Text-to-Image Generation," arXiv: 2206.10789, the MobileDiffusion model described in "MobileDiffusion: Instant Text-to-Image Generation on Mobile Devices," arXiv: 2311.16567, or the Imagen model described in "Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding," arXiv: 2205.11487.

[0023] As another example, the generative neural network 130 may be an audio generation neural network that generates audio signals, such as audio signals representing speech, music, or other audio signals, in response to user input. Examples of such neural networks include diffusion models and autoregressive audio generation neural networks. As specific examples, the generative neural network 130 may be the AudioLM model described in "AudioLM: a Language Modeling Approach to Audio Generation," arXiv:2209.03143, or the MusicLM model described in "MusicLM: Generating Music From Text," arXiv:2301.1132.

[0024] As another example, the generative neural network 130 may be a video generation neural network that generates videos in response to user input. Examples of such neural networks include diffusion models and autoregressive neural networks for video generation. As specific examples, the neural network 130 may be the Phenaki model described in "Phenaki: Variable Length Video Generation From Open Domain Textual Description," arXiv:2210.02399, or the WALT model described in "Photorealistic Video Generation with Diffusion Models," arXiv:2312.06662.

[0025] In some cases, instead of providing input to only one generative neural network 130, system 100 may be connected to multiple different generative neural networks 130. For example, system 100 may interface between users and two or more of the following: a generative neural network 130 that generates images, a generative neural network 130 that generates videos, a generative neural network 130 that generates audio, and so on.

[0026] In some cases, the request may also include additional data.

[0027] For example, the request may include one or more context data items that the generative neural network 130 uses as context when generating the data item.

[0028] In some implementations, rather than simply providing the prompt 120 directly as input to the generative neural network 130, the system 100 instead allows the user 102 to refine the prompt before submitting the prompt to the generative neural network 130.

[0029] In some other implementations, the system 100 may provide the prompt 120 to the generative neural network 130 and obtain a data item generated by the generative neural network 130 by processing the prompt 120. The system 100 may then allow the user 102 to refine the prompt 120 while viewing the data item generated by the generative neural network 130 in response to the prompt 120.

[0030] In particular, the system 100 uses a neural language model network 140 to identify one or more text segments from the text sequence and, for each of the identified text segments, one or more candidate refinements 142 for the identified text segment.

[0031] The system 100 then provides data identifying the one or more candidate refinements 142 for the text segments for presentation in a user interface of the user device 104.

[0032] In general, the user interface allows the user 102 to generate a modified prompt 150 by replacing one or more of the identified text segments with one of the candidate refinements 142 for the identified text segment.

[0033] An example of a user interface is given below with reference to the Fig. 3A and Fig. 3B described.

[0034] Once the user 102 has generated the modified prompt 150, the system 100 receives the modified prompt 150 from the user device 104 and provides the generative neural network 130 with an input including the modified prompt 150 and optionally other data.

[0035] The system 100 obtains a generated data item 160 as output from the generative neural network 130 and provides the generated data item 160 for presentation to the user 102 on the user device 104.

[0036] The system 100 may proceed to allow the user to further refine the modified prompt 150 to generate additional data items 160. That is, the system 100 may continue to utilize the neural language model network 140 to allow the user to explore the space of prompts that may result in the generation of a data item having the properties desired by the user.

[0037] Fig. Figure 2 is a flowchart of an example process 200 for refining a prompt. For simplicity, the process 200 is described as being performed by a system of one or more computers located at one or more locations. For example, a prompt refinement system, such as the one described in Fig. 1, which is appropriately programmed according to this specification, perform the process 200.

[0038] The system receives a prompt for a generative neural network (step 202). As described above, the prompt generally includes a text sequence of text tokens.

[0039] The system generates a language model input for a neural language model network from the prompt (step 204).

[0040] For example, the system may combine the prompt with a predetermined prompt for the language model to generate the language model input.

[0041] As another example, the system may apply one or more rules or criteria to the prompt to determine whether certain terms in the prompt need to be removed or modified before the prompt is included in the language model input. For example, the system may check whether terms in the prompt violate rules or restrictions regarding appropriateness or safety.

[0042] The system processes the language model input using the language model neural network to generate a language model output (step 206).

[0043] The language model output (i) identifies one or more initial text segments from the text sequence and (ii) includes, for each of the identified initial text segments, one or more initial candidate refinements for the text segment.

[0044] Each of the one or more initial text segments contains a respective correct subset of the text tokens in the text sequence. That is, each initial text segment contains fewer than all of the tokens in the text sequence. For example, the initial text segments may contain words or phrases within the prompt, but a given text segment is generally not the entire prompt.

[0045] Each candidate refinement is a text segment that can replace the corresponding text segment in the prompt.

[0046] In general, the language model output identifies one or more initial text segments and includes structured information for each of the identified text segments.

[0047] The structured information contains the candidate refinements for the text segment, but may also contain additional information.

[0048] For example, the structured information may include information about semantically related segments. As one example, the structured information may identify that multiple semantically related segments should be updated together when a user decides to refine. That is, for each of the semantically related segments and for each candidate refinement for the semantically related segment, the structured information may identify corresponding candidate refinements for the other semantically related segments. In response to the user's selection of the candidate refinement, the system may either automatically refine the other semantically related segments to the corresponding candidate refinements or provide an indication of the corresponding candidate refinements in the user interface.

[0049] The structured information can also include information about the types of refinement, allowing for further user control, such as a refinement that improves the variety of the prompt or a refinement that changes the aesthetic style of the output. That is, when presented in the user interface, each candidate refinement can be presented along with data identifying the type of refinement.

[0050] More generally, by providing structured data in the response (which is then sent to the user device), a user interface can allow a user to further examine the refinements provided by user interface elements, such as toggles and controls, in the user interface without additional calls to a server. That is, by providing this structured information, users can obtain additional information about candidate refinements and switch between candidate refinements locally on the user device without having to invoke the neural language model network, which is generally remote from the user device.

[0051] The language model neural network is a neural network configured to process an input to produce an output comprising a probability distribution over a set of text tokens in the vocabulary of tokens, where the probability for each token represents the probability that the token immediately follows the input.

[0052] The vocabulary of tokens generally includes text tokens and may optionally include tokens representing one or more other modalities, e.g., audio, image, video, etc. The text tokens may include any suitable tokens that appear in natural language text, e.g., ASCII characters, words, word parts, or differently distributed n-grams. For example, the vocabulary of text tokens may be fixed or generated by applying a suitable tokenizer, e.g., a byte-pair encoding tokenizer or the SentencePiece tokenizer, to a text corpus.

[0053] For example, the neural language model network can be an autoregressive neural language model network.

[0054] The language model neural network is called an autoregressive neural network because the neural network autoregressively generates an output sequence of tokens by generating each individual token in the output sequence under the condition of a current input sequence that includes all tokens preceding the respective text token in the output sequence, i.e., the tokens already generated for all previous positions in the output sequence preceding the particular position of the particular token, and a context input that provides the context for the output sequence (a “context sequence”).

[0055] For example, when generating a token at a given position in the output sequence, the current input sequence may include the context sequence and the tokens at all previous positions preceding the given position in the output sequence. As a particular example, the current input sequence may include the context sequence followed by the tokens at all previous positions preceding the given position in the output sequence. Optionally, the context and the current output sequence may be separated by one or more predetermined tokens within the current input sequence.

[0056] More specifically, to generate a specific token at a specific position within a candidate output sequence, the neural network may process the current input sequence to generate a score distribution, e.g., a probability distribution, that assigns a corresponding score, e.g., a corresponding probability, to each text token in the text token vocabulary. The neural network may then use the score distribution to select a text token from the vocabulary as the specific token. For example, the neural network may greedily select the token with the highest score or, for example, sample a token from the distribution using kernel sampling or another sampling technique.

[0057] As a specific example, the language model neural network may be an autoregressive transformer-based neural network that includes (i) a plurality of attention blocks, each applying a self-supervision operation, and (ii) an output subnetwork that processes an output of the last attention block to generate the score distribution.

[0058] The neural network can represent a reliable Transformer-based neural network architecture. Examples of such architectures are described in J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, LA Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language models, arXiv preprint arXiv:2203.15556, 2022; JW Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, HF Song, J. Aslanides, S. Henderson, R. Ring, S. Young, E. Rutherford, T. Hennigan, J. Menick, A. Cassirer, R. Powell, G. van den Driessche, LA Hendricks, M. Rauh, P. Glag, J. P. Huang, J. Huang, R. Ring, S. Young, E. Welbl, S. Dathathri, S. Huang, J. Uesato, J. Mellor, I. Higgins, A. Creswell, N. McAleese, A. Wu, E. Elsen, SM Jayakumar, E. Buchatskaya, D. Budden, E. Sutherland, K. Simonyan, M. Paganini, L. Sifre, L. Marten, A. Li, A. Kunco, A. Kunco. Nematzadeh, E. Gribovskaya, D. Donato, A. Lazaridou, A. Mensch, J. Lespiau, M. Tsimpoukelli, N. Grigorev, D. Fritz, T. Sottiaux,M.Pajarskas, T. Pohlen, Z. Gong, D. Toyama, C. de Masson d'Autume, Y. Li, T. Terzi, V. Mikulik, I. Babuschkin, A. Clark, D. de Las Casas, A. Guy, C. Jones, J. Bradbury, M. Johnson, B. A. Hechtman, L. Weidinger, I. Gabriel, W. S. Isaac, E. Lockhart, S. Osindero, L. Rimell, C. Dyer, O. Vinyals, K. Ayoub, J. Stanway, L. Bennett, D. Hassabis, K. Kavukcuoglu und G. Irving. Scaling language models: Methods, analysis & insights from training gopher. CoRR, abs / 2112.11446, 2021; Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li und Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019; Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu und Quoc V. Le. Towards a human-like open-domain chatbot. CoRR, abs / 2001.09977, 2020; Gemini (beschrieben in arXiv:2403.05530), Gemma (described in arXiv:2403.08295) and PaliGemma (described in arXiv:2412.03555).

[0059] In general, however, the transformer-based neural network includes a sequence of attention blocks, and while processing a given input sequence, each attention block in the sequence receives a respective hidden input state for each input token in the given input sequence. The attention block then updates at least the hidden state for the last token in the given input sequence, at least in part by applying self-attention, to generate a corresponding hidden output state for the last token. The hidden input states for the first attention block are embeddings of the input tokens in the input sequence, and the hidden input states for each subsequent attention block are the hidden output states generated by the preceding attention block.

[0060] In this example, the output subnetwork processes the hidden output state generated by the last attention block in the sequence for the last input token in the input sequence to generate the score distribution.

[0061] In general, the neural language model network may have already been trained through one or more previous training phases before the system uses the neural language model network.

[0062] For example, one or more previous training phases may include a pre-training phase. During the pre-training phase, the language model neural network may have been trained by the system or a separate system on a next token prediction task, e.g., a task that requires, given a current sequence of tokens, to predict the next token following the current sequence in the training data.

[0063] As a particular example, the neural language model network may have been trained using maximum likelihood on a large dataset of texts in one or more natural languages, e.g., texts publicly available on the Internet or another text corpus, a large dataset of computer code in one or more programming languages, e.g., Python, C++, C#, Java, Ruby, PHP, etc., e.g., computer code publicly available on the Internet or another code repository, a large dataset of audio samples, e.g., audio recordings or waveforms representing the audio recordings, a large dataset of images, where each image includes an array of pixels, a large dataset of videos, where each video includes a temporal sequence of individual images, or a large multimodal dataset that includes a combination of two or more of these datasets.

[0064] As another example, the one or more prior training phases may include one or more additional training phases that occur, for example, after the pre-training phase. For example, the one or more prior training phases may include one or more of the following phases: a supervised fine-tuning phase, a reinforcement learning phase, e.g., reinforcement learning through human or other feedback, a preference learning phase, an instruction tuning phase, etc.

[0065] The system can cause the neural language model network to generate the language model output described above in various ways.

[0066] For example, the system may include in the language model input a k-shot prompt that includes k examples, where each example includes an example prompt and an example language model output generated for the example prompt. Generally, k can be a fixed integer greater than or equal to one.

[0067] As another example, the system may include in the language model input a natural language instruction that explains how the language model neural network should generate the language model output.

[0068] As another example, the system may have fine-tuned the language model neural network to improve the neural network's performance in accurately generating language model outputs of the required type. For example, the system may have fine-tuned the language model neural network on a fine-tuning dataset that includes multiple training examples, where each training example includes a prompt and a target language model output that identifies one or more segments in the prompt and includes corresponding structured data for each identified segment.

[0069] Using the language model output, the system identifies (i) one or more final text segments from the text sequence and (ii) for each of the final text segments, one or more final candidate refinements for the final text segment (step 208).

[0070] In some implementations, the system uses the initial text segments and the corresponding initial refinements as final text segments and final refinements.

[0071] In some other implementations, the system may modify one or more of the initial segments and refinements to generate final segments and corresponding final refinements.

[0072] For example, the system can apply one or more rules or criteria to the initial refinements to determine whether certain terms in the refinements need to be removed or modified before being included in the final refinements. For example, the system can check whether terms in the initial refinements violate rules or restrictions regarding appropriateness or security.

[0073] As another example, the system may retain historical data. The historical data may be specific to the current user or may be data generated based on interactions from multiple different users. As an example, the retained historical data for a set of candidate refinements may indicate how frequently the candidate refinement is adopted, i.e., inserted into a prompt after being suggested by the system. As a further example, the retained historical data for a set of candidate text segments may indicate how frequently the text segment is refined by users.

[0074] The system can then use this historical data to modify the initial refinements or initial text segments. For example, the system can remove all refinements from the initial refinements that are applied by the current user or by multiple different users in less than a threshold proportion of the time they are suggested by the system. As another example, the system can remove the initial refinements for the text segments for which the data indicates that they were refined by the current user or by multiple different users in less than a threshold proportion of the time they are suggested by the system.

[0075] In some examples, the determination of whether to modify the initial output of the neural language model network may be context-dependent, based on a specific prompt. For example, if the system determines that the prompt relates to photorealistic people, the system may determine that refinements related to lighting and contrast are more beneficial than refinements related to pose or setting. This may be determined after processing the prompt to determine the context, e.g., using a machine learning model. For example, the system may process the prompt using the neural language model network or another machine learning model to generate an output that identifies the context for the prompt.The system can then process an input that identifies the context and the refinements using the neural network language model or a different machine learning model to generate an output that (i) identifies the refinements that are relevant to the context or (ii) identifies the refinements that are not relevant to the context. The system can then determine to include in the final refinements (i) only the refinements identified as relevant to the context or (ii) only the refinements that were not identified as irrelevant to the context.

[0076] The system provides data identifying the one or more final candidate refinements for the final text segments for presentation to a user device in a user interface. The user interface enables a user to generate a modified prompt by replacing one or more of the final text segments with one of the final candidate refinements for the final text segment (step 210).

[0077] If the system receives an indication that the user has submitted, via the user interface, one or more inputs that refine the initial prompt by replacing one or more of the final text segments with one of the final candidate refinements for the final text segment, the system may provide the resulting modified prompt as input to the generative neural network and, in response, obtain an output data item generated in response to the modified prompt.

[0078] The system may then provide the output data item for presentation to the user on the user device. In some implementations, after providing the output data item for presentation, the system may further present the (not yet selected) final candidate refinements to allow the user to further modify the prompt. In some other implementations, after providing the output data item for presentation, the system may perform process 200 from the modified prompt to generate a new set of final candidate refinements for a new set of final text segments of the modified prompt.

[0079] Fig. 3A shows an example of a user interface 300 that displays an initial prompt 310 and allows a user to refine the initial prompt 310.

[0080] In particular, the user interface 300 displays the initial prompt 310 “Steampunk flying bike in the air, powered by a cute squirrel with aviator goggles, lively, picturesque.”

[0081] The user interface 300 also identifies three text segments 312, 314 and 316 that have been identified by the system 100 as candidates for refinement.

[0082] In particular, user interface 300 displays each identified text segment 312, 314, and 316 in association with a respective user interface element that, when selected by a user, represents the candidate refinements. For example, text segments 312, 314, and 316 may be displayed in a visually distinct manner from the other text segments of initial prompt 310 that were not identified as candidates for refinement by system 100.

[0083] Fig. 3B shows an example of the user interface 300 after a user has selected the user interface element associated with an identified text segment, ie, the identified text segment 312.

[0084] As from Fig. 3B, in response to the user input selecting the user interface element, the user interface is updated to display the candidate refinements for the identified text segment 312.

[0085] If the user selects one of the candidate refinements, the system may update the prompt to include the candidate refinement instead of the identified text segment 312.

[0086] As can be seen from the examples in Fig. 3A and Fig.3B, the system 100 does not rewrite the entire prompt, but instead identifies various text segments within the prompt as candidates for refinement and allows the user to select from various candidate refinements for each identified text segment.

[0087] Once the user has modified the prompt as needed using the candidate refinements, the user may select the "Create" user interface element 320 to cause the system 100 to provide the resulting refined prompt as input to the generative neural network. The system 100 may then display the generated data item to the user, e.g., in the user interface 300 or in another user interface. In some other cases, instead of requiring the user to select the user interface element 320, the system 100 may automatically provide the current prompt as input to the generative neural network each time the user selects a refinement.

[0088] While the above description describes that the system generates the data item after receiving the modified prompt, in some cases the system also generates a data item from the initial prompt. In these cases, the system can display the candidate refinements along with the data item generated from the initial prompt in the user interface, allowing the user to reference the initially generated data item to identify how the initial prompt should be modified.

[0089] In this specification, the term "configured" is used in the context of systems and computer program components. A system comprising one or more computers may be configured to perform particular operations or actions by having software, firmware, hardware, or a combination thereof installed in the system that, when used, causes the system to perform the operations or actions. That one or more computer programs may be configured to perform particular operations or actions means that the one or more programs include instructions which, when executed by a data processing device, cause the device to perform the operations or actions.

[0090] Embodiments of the subject matter and the functional operations described in this specification may be implemented in digital electronic circuits or in physically embodied computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more thereof. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a physical non-transitory storage medium for execution by, or for controlling the operation of, a computing device.The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access storage device, or a combination of one or more thereof. Alternatively or additionally, the program instructions may be encoded on a synthetically generated propagated signal, for example, a machine-generated electrical, optical, or electromagnetic signal generated to encode information for transmission to a suitable receiving device for execution by a data processing device.

[0091] The term "data processing device" refers to data processing hardware and includes all types of devices, apparatus, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. The device may further consist of or include special-purpose logic circuits, for example, an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the device may optionally include code that generates an execution environment for the respective computer program, for example, code representing processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0092] A computer program, which may also be called or described as a program, software, software application, app, module, software module, script, or code, may be written in any programming language, including compiled or interpreted languages, or declarative or procedural languages, and it may be delivered in any form, including as a stand-alone program or module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but does not have to, correspond to a file in a file system. A program may be stored in a part of a file that contains other programs or data, such as one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in several coordinated files, such asFiles that store one or more modules, subprograms, or sections of code. A computer program can be deployed to run on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0093] In this specification, the term "database" is used in its broadest sense to refer to any collection of data: The data need not be structured in any particular way or at all and can be stored on storage devices at one or more locations. Thus, for example, the index database can contain multiple data collections, each organized and accessed differently.

[0094] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Generally, an engine is implemented as one or more software modules or components and installed on one or more computers at one or more locations. In some cases, one or more computers are dedicated to a specific engine; in other cases, multiple engines may be installed and running on the same computer or computers.

[0095] The processes and logic sequences described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by processing input data and generating output. The processes and logic sequences can also be performed by special-purpose logic circuitry, such as an FPGA or ASIC, or by a combination of special-purpose logic circuitry and one or more programmed computers.

[0096] Computers suitable for executing a computer program may be based on microprocessors of a general or special type, or both, or on any other type of central processing unit. Generally, a central processing unit receives instructions and data from a read-only memory or a random access memory, or both. The essential elements of a computer are a central processing unit for performing or executing instructions, and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or integrated with special logic circuitry. Generally, a computer also includes one or more mass storage devices for storing data, e.g.magnetic, magneto-optical, or optical storage devices, or is operatively coupled to one or more of these devices to receive data from them, transmit data to them, or both. However, a computer need not include such devices. In addition, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a GPS (Global Positioning System) receiver, or a portable storage device such as a USB (Universal Serial Bus) flash drive, to name a few.

[0097] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0098] To provide interaction with a user, the subject matter embodiments described in this specification may be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor for displaying information to the user, and a keyboard and pointing device (such as a mouse or trackball) for the user to provide input to the computer. Other types of devices may also be used to interact with a user; for example, feedback provided to the user may be any form of sensory feedback (such as visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including auditory, voice, or tactile input.Additionally, a computer can interact with a user by sending and receiving documents to and from a device used by the user. For example, it can send web pages to a web browser on a user's device in response to requests received from the web browser.

[0099] A computer may also interact with a user by sending text messages or other messages to a personal device, such as a smartphone running a messaging application, and receiving reply messages from the user.

[0100] For example, computing devices for implementing machine learning models may also include dedicated hardware accelerator units for processing general and computationally intensive parts of machine learning training or production, e.g., inference workloads.

[0101] Machine learning models can be implemented and deployed using a machine learning framework, such as a TensorFlow framework or a Jax framework.

[0102] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component, e.g., as a data server; or that includes a middleware component, e.g., an application server; or that includes a front-end component, e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein; or any combination of such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, e.g., a communications network. Examples of communications networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0103] The computing system may include clients and servers. A client and a server are generally remote from each other and typically interact over a communications network. The client-server relationship arises because the respective computers run computer programs that are in a client-server relationship with each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., to display data to and receive user input from a user interacting with the device acting as a client. Data generated on the user device, e.g., a result of the user interaction, may be received from the device at the server.

[0104] Although this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features described in this specification in connection with separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in connection with a single embodiment may also be implemented in multiple embodiments separately or in any suitable combination.Furthermore, although features may be described above as operating in certain combinations and may initially be claimed as such, in some cases one or more features of a combination may be extracted from that combination, and the claimed combination may relate to a subordinate combination or a modification of a subordinate combination.

[0105] Although the operations are depicted in a particular order in the drawings and claims, this should not be understood to mean that it is required to achieve the desired results by performing these operations in the particular order shown or in the order stated, or that all of the illustrated operations be performed. In certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various system modules and components in the embodiments described above should not be understood to mean that such separation is required in all embodiments. It should also be understood that the described program components and systems may generally be integrated together into a single software product or bundled into multiple software products.

[0106] Certain embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims may be performed in a different order and still achieve the desired results. By way of example, the processes depicted in the accompanying figures do not necessarily require the particular order or sequential order shown to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Zitierte Nicht-Patentliteratur

[0000] Scaling Autoregressive Models for Content-Rich Text-to-Image Generation“, arXiv: 2206.10789

[0022] MobileDiffusion: Instant Text-to-Image Generation on Mobile Devices“, arXiv: 2311.16567

[0022] Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding“, arXiv: 2205.11487

[0022] AudioLM: a Language Modeling Approach to Audio Generation“, arXiv:2209.03143

[0023] MusicLM: Generating Music From Text“, arXiv:2301.1132

[0023] Phenaki: Variable Length Video Generation From Open Domain Textual Description“, arXiv:2210.02399

[0024] Photorealistic Video Generation with Diffusion Models“, arXiv:2312.06662

[0024] J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language models, arXiv preprint arXiv:2203.15556, 2022

[0058] JW Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, HF Song, J. Aslanides, S. Henderson, R. Ring, S. Young, E. Rutherford, T. Hennigan, J. Menick, A. Cassirer, R. Powell, G. van den Driessche, LA Hendricks, M. Rauh, P. Glag, J. P. Huang, J. Huang, R. Ring, S. Young, E. Welbl, S. Dathathri, S. Huang, J. Uesato, J. Mellor, I. Higgins, A. Creswell, N. McAleese, A. Wu, E. Elsen, SM Jayakumar, E. Buchatskaya, D. Budden, E. Sutherland, K. Simonyan, M. Paganini, L. Sifre, L. Marten, A. Li, A. Kunco, A. Kunco. Nematzadeh, E. Gribovskaya, D. Donato, A. Lazaridou, A. Mensch, J. Lespiau, M. Tsimpoukelli, N. Grigorev, D. Fritz, T. Sottiaux, M. Pajarskas, T. Pohlen, Z. Gong, D. Toyama, C. de Masson d'Autu, Y. Milik, T. V. Mili, V. Y. Babuschkin, A. Clark, D. de Las Casas, A. Guy, C. Jones, J. Bradbury, M. Johnson, BA Hechtman, L. Weidinger, I. Gabriel, WS Isaac, E. Lockhart, S. Osindero, L. Rimell, J. Dyer, O. Vinyals, K. Ayoub, J. L. Stanway, K. Bennett, D. K. Bennett, Hassac. Kavukcuoglu and G. Irving.Scaling language models: Methods, analysis & insights from training gopher. CoRR, abs / 2112.11446, 2021

[0058] . Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li und Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019

[0058] Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu und Quoc V. Le. Towards a human-like open-domain chatbot. CoRR, abs / 2001.09977, 2020

[0058] Gemini (beschrieben in arXiv:2403.05530), Gemma (beschrieben in arXiv:2403.08295) und PaliGemma (beschrieben in arXiv:2412.03555

[0058]

Claims

[1] System comprising: one or more computers;and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising: Receiving a prompt to a generative neural network, the prompt comprising a text sequence of text tokens; Generating a language model input for a language model neural network from the prompt; Processing the language model input using the neural language model network to generate a language model output that (i) identifies one or more initial text segments from the text sequence and (ii) includes, for each of the identified initial text segments, one or more initial candidate refinements for the text segment, wherein each of the one or more initial text segments comprises a respective correct subset of the text tokens in the text sequence; Identifying, using the language model output, (i) one or more final text segments from the text sequence and (ii) for each of the final text segments, one or more final candidate refinements for the final text segment; and Providing data identifying the one or more final candidate refinements for the final text segments for display in a user interface of a user device, the user interface enabling a user to generate a modified prompt by replacing one or more of the final text segments with one of the final candidate refinements for the final text segment. [2] The system of claim 1, wherein the operations further comprise: Receiving the modified request from the user device. [3] The system of claim 2, wherein the operations further comprise: Providing an input comprising the modified request to the generative neural network; Obtaining a generated data item as output from the generative neural network; and Providing the generated data element for display on the user device. [4] The system of claim 3, wherein the input to the generative neural network further comprises an initial data element. [5] A system according to claim 3 or claim 4, wherein the generated data item is an image. [6] A system according to claim 3 or claim 4, wherein the generated data item is a video. [7] A system according to claim 3 or claim 4, wherein the generated data item is an audio signal. [8] A system according to any preceding claim, wherein generating a language model input for a neural language model network from the input prompt comprises: Modify the prompt before including the prompt in the language model input. [9] A system according to any one of the preceding claims, wherein identifying using the language model output (i) one or more final text segments from the text sequence and (ii) for each of the final text segments, one or more final candidate refinements for the final text segment comprises one or more of the following: Removing one of the initial text segments; or Removing one of the initial candidate refinements for one of the initial text segments. [10] A system according to any preceding claim, wherein the language model output for each of the identified initial text segments includes respective structured data including the one or more initial candidate refinements for the text segment. [11] The system of claim 10, wherein the respective structured data includes information about semantically related segments to the identified initial text segment. [12] The system of claim 10 or claim 11, wherein the respective structured data includes information identifying, for each candidate refinement, a respective type of refinement. [13] The system of any of claims 10-12, wherein the user interface includes one or more user interface elements corresponding to the respective structured data. [14] One or more computer-readable storage media storing instructions which, when executed by one or more computers, cause the one or more computers to perform the respective operations, comprising: Receiving a prompt to a generative neural network, the prompt comprising a text sequence of text tokens; Generating a language model input for a language model neural network from the prompt; Processing the language model input using the neural language model network to generate a language model output that (i) identifies one or more initial text segments from the text sequence and (ii) includes, for each of the identified initial text segments, one or more initial candidate refinements for the text segment, wherein each of the one or more initial text segments comprises a respective correct subset of the text tokens in the text sequence; Identifying, using the language model output, (i) one or more final text segments from the text sequence and (ii) for each of the final text segments, one or more final candidate refinements for the final text segment; and Providing data identifying the one or more final candidate refinements for the final text segments for display in a user interface of a user device, the user interface enabling a user to generate a modified prompt by replacing one or more of the final text segments with one of the final candidate refinements for the final text segment. [15] The one or more computer-readable storage media of claim 14, wherein the operations further comprise: Receiving the modified request from the user device. [16] The one or more computer-readable storage media of claim 15, wherein the operations further comprise: Providing an input comprising the modified request to the generative neural network; Obtaining a generated data item as output from the generative neural network; and Providing the generated data element for display on the user device. [17] The one or more computer-readable storage media of claim 16, wherein the input to the generative neural network further comprises an initial data element. [18] The one or more computer-readable storage media of claim 16 or claim 17, wherein the generated data item is an image. [19] The one or more computer-readable storage media of claim 16 or claim 17, wherein the generated data item is a video. [20] The one or more computer-readable storage media of claim 16 or claim 17, wherein the generated data item is an audio signal. [21] The one or more computer-readable storage media of any one of claims 14 to 20, wherein generating a language model input for a neural language model network from the prompt comprises: Modify the prompt before including the prompt in the language model input. [22] The one or more computer-readable storage media of any one of claims 14 to 21, wherein identifying using the language model output (i) one or more final text segments from the text sequence and (ii) for each of the final text segments, one or more final candidate refinements for the final text segment comprises one or more of the following: Removing one of the initial text segments; or Removing one of the initial candidate refinements for one of the initial text segments. [23] The one or more computer-readable storage media of any one of claims 14 to 22, wherein the language model output for each of the identified initial text segments includes respective structured data including the one or more initial candidate refinements for the text segment. [24] The one or more computer-readable storage media of claim 23, wherein the respective structured data includes information about semantically related segments to the identified initial text segment. [25] The one or more computer-readable storage media of claim 23 or claim 24, wherein the respective structured data includes information identifying, for each candidate refinement, a respective type of refinement. [26] The one or more computer-readable storage media according to any one of claims 23-25, wherein the user interface includes one or more user interface elements corresponding to the respective structured data.

Images