Generating content items based on source document metadata using a generative neural network.

Abstract

The present invention provides a method, system, and apparatus for generating content items based on metadata of a source document using a generative neural network. [Solution] The method includes receiving a request from a user to generate a content item using a contextually conditioned generative neural network. The contextual input includes content derived from a source electronic document. The method also includes retrieving metadata associated with the source electronic document, generating a prompt about the generative neural network based on the contextual input and the metadata associated with the source electronic document, processing the prompt using the generative neural network to generate a content item, and providing the content item for presentation to the user.

Description

Technical Field

[0001] This specification relates to using a neural network to process an input and generate content items. For example, content items can include text data, image data, video data, audio data, and the like.

[0002] A neural network is a machine learning model that uses one or more layers of non-linear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., another hidden layer or the output layer. Each layer of the network generates an output from the received input according to the current values of its respective set of parameters.

Summary of the Invention

Means for Solving the Problems

[0003] This specification describes a content item generation system implemented as a computer program on one or more computers in one or more locations that uses a generative neural network to generate content items based on metadata of a source document.

[0004] According to one embodiment, a method is provided that is performed by one or more computers. The method includes receiving a request from a user for generating a content item using a contextually conditioned generative neural network, wherein the contextually conditioned input includes content derived from a source electronic document; receiving metadata associated with the source electronic document; generating a prompt for the generative neural network based on the contextually conditioned input and the metadata associated with the source electronic document; processing the prompt using the generative neural network to generate a content item; and providing the content item for presentation to the user.

[0005] Generating prompts for a generative neural network may include generating prompts that include contextual input and metadata.

[0006] Generating prompts for a generative neural network may include generating prompts that include contextual input, metadata, and additional information generated based on metadata associated with the source electronic document and metadata associated with the generative neural network.

[0007] The metadata associated with a generative neural network may include the cutoff date of the generative neural network, which represents the most recent publication date of the data contained in the training data used to train the generative neural network.

[0008] Retrieving metadata associated with a source electronic document may involve receiving metadata from an operating system running on one or more computers.

[0009] Retrieving metadata associated with a source electronic document may involve performing a search within a document corpus to identify electronic documents related to the content, and using the metadata associated with the identified electronic documents as metadata associated with the source electronic document.

[0010] Content derived from the source electronic document may include content copied from the source electronic document.

[0011] In another embodiment, one or more computer-readable storage media are provided on which instructions are encoded that cause one or more computers to perform the actions of the method described above when executed by one or more computers.

[0012] In a further embodiment, a system is provided which includes one or more computers and, when executed by one or more computers, one or more storage devices that store instructions causing one or more computers to perform each of the operations of the above-described embodiment.

[0013] The subject matter described herein may be implemented in particular embodiments to achieve one or more of the following advantages:

[0014] Generally, content item generation systems can use generative neural networks to generate content items conditioned on contextual inputs that provide context for the content items. Often, the contextual inputs provided to the system include content derived from a source electronic document, such as transferred content. Traditionally, metadata related to the source electronic document is ignored and does not proceed to the content item generation process, although it may include additional information that can be used to improve the performance of the generative neural network and generate content items that are of higher quality or more relevant to the contextual input.

[0015] Using the techniques described herein, a content item generation system can incorporate metadata related to a source electronic document into prompts for processing by a generative neural network, enabling the generative neural network to generate higher-quality content items, such as those that are more informative and contextually relevant. This ability to generate higher-quality content items also improves the user experience of the content item generation system.

[0016] The metadata that can be obtained by the content item generation system using the techniques described herein may include information such as the source, creation date, and publication date of the source electronic document. Therefore, although the metadata encodes additional information relevant to the generation task, such as information that is richer than the derived content alone, or information that can identify whether the derived content originates from a particular source, this additional information is missing from the contextual input originally received by the system. Consequently, content items generated by the described system may be more informative and more relevant to the contextual input than content items generated based solely on the contextual input.

[0017] Details of one or more embodiments of the subject matter of this specification are described in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from this specification, the drawings, and the claims. [Brief explanation of the drawing]

[0018] [Figure 1] This is a diagram illustrating an example of a content item generation system. [Figure 2] This is a diagram illustrating an exemplary environment including a content item generation system. [Figure 3] This is an illustrative diagram of a content item generation system that generates content items based on prompts that include derived content and metadata. [Figure 4] This is an illustrative flowchart of a process for generating content items using a generative neural network based on metadata from a source document. [Modes for carrying out the invention]

[0019] Similar reference numbers and names in various drawings refer to the same items.

[0020] Figure 1 shows an exemplary content item generation system 100. The content item generation system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations, and can implement the systems, components, and technologies described below.

[0021] The content item generation system 100 is a system that generates content items 152 using a generative neural network 110 that is at least conditioned on a contextual input 104.

[0022] For example, the content item generation system 100 may receive a context input 104 as part of a request 102 for a content item 152, or as an associated context input, and in response, may use the generative neural network 110 to process a prompt 108 containing the context input 104 in order to generate the content item 152.

[0023] The content item generation system 100 can generate any type of content item 152, such as text content items, image content items, video content items, audio content items, and so on.

[0024] The content item 152 generated by the content item generation system 100 can be used in any of a variety of ways. For example, the system can provide the content item 152 for presentation to a user on a display device. As another example, the system can provide the content item 152 for further processing to other components within the system or to a different system. As yet another example, the system can store the content item 152 in a data repository for some future purpose.

[0025] In some cases, the content item generation system 100 may be a text generation system that generates a text sequence, i.e., each content item 152 generated by the system is an output sequence of text that includes a sequence of text tokens from the vocabulary of text tokens that includes, for example, one or more of characters, subwords, words, punctuation, numbers, or other symbols that appear in a natural language or a computer language. For example, the system can generate a text sequence in response to the context input 104 provided by a user of the system and provide the text sequence for presentation to the user who provided the context input 104.

[0026] For example, the context input 104 can be an input sequence of text, and the output sequence can be a sequence of text of other text, such as a translation of the input sequence of text, a completion of the input sequence of text, a rephrasing of the input sequence of text, a response to a question raised in the input sequence, or a sequence of text related to a topic specified by the input sequence of text. As another example, the context input 104 can be an input other than text, such as an image, video, or audio, and the output sequence can be text that describes the input.

[0027] As a specific example, the content item generation system 100 may be part of a dialogue system, where the context input 104 may contain audio or text from the most recent conversational turn submitted by the user of the dialogue system during the dialogue, while the text output sequence may be either text or audio that is a response to the next turn of the conversation, for example, the most recent conversational turn. Optionally, the context input 104 may also contain one or more previous conversational turns that occurred earlier in the conversation.

[0028] As another specific example, the content item generation system 100 may be part of a computer code generation system, the context input 104 may be a text description of a desired code portion or a fragment of computer code in a programming language, and the text output sequence may be computer code, for example, a fragment of code described by the context input or a fragment of code of a computer program that follows the context input.

[0029] In some cases, the content item generation system 100 may be an image or video generation system that generates images or videos (each frame being an image) each having multiple frames, either as a sequence of pixels or through an iterative denoising process. For example, the content item generation system 100 may generate images or videos conditioned on a contextual input 104 provided by the system user, which includes a text description of the content of the image or video.

[0030] In some cases, the content item generation system 100 may be an audio generation system that generates audio signals, for example, each content item 152 being an example of output audio containing samples of an audio wave in each of a sequence of output time steps over a predetermined time window. For example, the content item generation system 100 may generate audio conditioned on a context input 104 provided by the system user, which includes a text description of the audio content.

[0031] In these cases, output time steps can be placed at regular intervals within a specified time window. An audio sample in a given output time step may be the amplitude value of a sound wave, or an amplitude value that has been compressed, decompressed, or both. For example, an audio sample may be the raw amplitude value, or a mu-law decompressed representation of the amplitude value.

[0032] In many scenarios, context input 104 contains, for example, transferred content derived from a source electronic document 106. An electronic document is data that represents a set of content. Examples of electronic documents include web pages, word processing documents, portable document format (PDF) documents, images, videos, audio, source code files, and feed sources. Native applications (e.g., “apps” and / or applications) installed on mobile, tablet, or desktop computing devices are also examples of electronic documents.

[0033] Source electronic document 106 has associated metadata. Examples of such metadata include the date and time source electronic document 106 was created or last modified, the publication date of source electronic document 106, the title or file name of source electronic document 106, the owner or author of source electronic document 106 (whether an individual or an organization), the release number or version number of source electronic document 106, the source of source electronic document 106, such as its network location (e.g., Universal Resource Locator (URL)), source host name, source domain name, or source Internet Protocol (IP) address.

[0034] Content derived from source electronic document 106 has associated metadata. Examples of such metadata include the data and time the content was copied from source electronic document 106, the location of the content within source electronic document 106, such as the line number or byte offset within source electronic document 106, the context surrounding the content, and the size of the content.

[0035] In these scenarios, if the generative neural network 110 is provided with not only content derived from the source electronic document 106, but also metadata, content, or both associated with the source electronic document 106, and potentially other metadata, such as metadata associated with the generative neural network 110 itself, it is possible to improve the performance of the generative neural network 110 in many content generation applications, such as conditional text, image, video, or audio generation tasks.

[0036] When provided as part of a prompt 108 to the generative neural network 110 for processing, the metadata can improve the performance of the generative neural network 110, facilitating the generation of higher-quality content items, such as content items 152 that are more informational and, for several reasons, more relevant to the contextual input 104.

[0037] Firstly, metadata contains more comprehensive information than derived content alone, and therefore provides additional context to content items generated by the generative neural network 110.

[0038] Secondly, since the generative neural network 110 can access information about where the derived content originates and when it is created, the metadata can link the derived content to a specific source and thus facilitate the generation of content items that are more relevant responses to the contextual input 104 containing the derived content.

[0039] For example, a user transfers a text fragment (e.g., a news article about a news event) from a web page to context input 104, where context input 104 is a sequence of text inputs representing a question about the transferred text fragment (e.g., a question about the news event) or another request made referring to the transferred text fragment (e.g., a request to summarize the news article). In various cases, when providing context input 104, the user may transfer the text fragment by using an input device to copy and paste, cut and paste, or input verbatim or paraphrased text.

[0040] In this example, metadata regarding when the webpage containing the text fragment was created (and therefore implies when the news event occurred) may enable the generative neural network 110 to generate more relevant content items, such as responses that are more relevant to the text fragment (e.g., responses that are more relevant to the news event), or responses that represent a more accurate summary of the news article.

[0041] Assuming that the text fragments are created after the knowledge cutoff date of the generative neural network 110, in some embodiments, the generative neural network 110 may generate the text output sequence by using a search engine (or other external tool that can retrieve external data), more specifically, it may generate the text output sequence based on the latest information contained in the search engine results (or external data retrieved by other external tools).

[0042] In other words, the generative neural network 110 can avoid generating text output sequences by relying only on old information that was available to the generative neural network during training, and thus can avoid generating text output sequences that are inaccurate or at least outdated and therefore unreliable.

[0043] In this embodiment, metadata about where a webpage with derived content originated (e.g., where a news article was found) may also enable the generative neural network 110 to generate output sequences that represent more informative content items, such as a more accurate response to a question raised regarding a news event.

[0044] Assuming that metadata indicates the text fragment originates from a source associated with a high level of factual accuracy, the generative neural network 110 can process the contextual input 104 containing the text fragment to generate a text output sequence. Alternatively, assuming the metadata indicates the text fragment originates from a source associated with a low level of factual accuracy, the generative neural network 110 can generate a text output sequence indicating that the text fragment is effectively inaccurate (e.g., "I don't think this is accurate. Rather, it is known that...").

[0045] In another example, the user copies an image from an image source (e.g., a camera application, a photo album application, or an image / video processing application), and then pastes the copied image into context input 104, which is a multimodal input sequence of both text and images representing a question raised or a request made regarding the copied image. In this example, metadata about when and / or where the image was taken may enable the generative neural network 110 to generate more relevant content items.

[0046] For example, metadata can enable the generative neural network 110 to generate a response or other reaction, such as a text output sequence representing a caption, which is more relevant to the copied image than a text output sequence generated without using metadata.

[0047] As another example, metadata may enable the generative neural network 110 to generate other images of higher quality (e.g., higher fidelity) than images generated without the metadata. For example, these other images could be modified versions of the copied image, such as a super-resolution image (with higher resolution than the copied image), an inpainted image (reconstructing missing parts of the copied image), or an image that is the next predicted frame of the copied image.

[0048] As yet another example, the user transfers a fragment of a text report from a specific software tool to context input 104, where context input 104 represents a question raised about the text fragment transferred from the specific software tool, or a request made in relation to it.

[0049] For example, a software tool could be an integrated development environment (IDE) tool, and the text fragments could include fragments of source code, such as incomplete source code fragments and source code fragments containing bugs. Another example is a software tool that could be a compiler, and the text fragments could include error messages included as part of an error report generated by the compiler after the source code compilation process has stopped due to an error that occurred in the source code.

[0050] In this embodiment, metadata about a specific software tool (e.g., compiler version) and metadata about the source code (e.g., source code owner or source) may enable the generative neural network 110 to generate more relevant content items, such as more relevant output sequences for text fragments (e.g., more relevant answers to questions about why a source code fragment has a bug / why compilation failed, or a more accurate completion of an incomplete source code fragment).

[0051] To this end, after the user provides a context input 104 containing, for example, transferred content derived from the source electronic document 106, the content item generation system 100 collects metadata associated with the source electronic document 106, metadata associated with the derived content, or both, based on the context input 104 and possibly other metadata, such as metadata associated with the generative neural network 110, and then incorporates the metadata into the prompt 108 before processing the context input 104 using the generative neural network 110.

[0052] For example, the prompt 108 may include (i) contextual input 104 (provided by the user), and one or more of the following: (ii) metadata related to the source electronic document 106 (obtained by the system), (iii) metadata related to derived content (obtained by the system), or (iv) metadata related to the generative neural network 110.

[0053] In this way, the prompt 108 improves the performance of the generative neural network 110 and facilitates the generation of higher-quality content items 152 by incorporating more comprehensive information, including metadata not directly provided by the user as part of or in connection with the request 102 for the content item 152.

[0054] The generative neural network 110 may be any suitable generative neural network that has a set of generative neural network parameters and can be used to generate content items containing data from a single modality or multiple modalities by processing prompt 108 according to the set of generative neural network parameters.

[0055] In some embodiments, the generative neural network 110 may have an architecture that enables the generative neural network 110 to more effectively map a prompt 108 containing (i) contextual input 104 and (ii) metadata to a content item. For example, the generative neural network 110 may include a prompt encoder subnetwork that processes the contextual input 104 to generate an embedding of the contextual input 104, a metadata encoder subnetwork that processes the metadata to generate an embedding of the metadata, and a core subnetwork that processes the embedding of the contextual input 104 and the embedding of the metadata to generate a content item.

[0056] In some embodiments, the generative neural network 110 may include a language model neural network, for example, as a core subnetwork or as another component of the generative neural network 110, which can perform an autoregressive token generation process by generating one token at each time step conditioned on any token already generated in a previous time step, for example, to regressively generate content items 152, such as a sequence of text tokens, a sequence of pixel tokens, a sequence of audio tokens, a sequence of multimodal tokens, such as text and pixel tokens over multiple time steps.

[0057] Language model neural networks can have any of the following transformer-based neural network architectures, such as a transformer architecture dedicated to encoders, an encoder-decoder type transformer architecture, a transformer architecture dedicated to decoders, or other attention-based architectures.

[0058] Examples of language model neural networks include those described below. Colin Raffel et al.,Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019, Daniel Adiwardana et al.,Towards a human-like open-domain chatbot. CoRR, abs / 2001.09977, 2020;Tom B Brown et al.,Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020;Aakanksha Chowdhery et al.,PaLM: Scaling Language Modeling with Pathways, arXiv preprint arXiv:2204.02311;Rohan Anil et al.,Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023;Borsos, Zalan et al., Audiolm: a language modeling approach to audio generation. IEEE / ACM Transactions on Audio, Speech, and Language Processing (2023), and Agostinelli, Andrea et al., “Musiclm: Generating music from text.” arXiv preprint arXiv:2301.11325 (2023).

[0059] In some embodiments, the generative neural network 110 may include a spread model neural network as a core subnetwork or as another component of the generative neural network 110, which starts with random noise and goes through a despread process through several despread steps to iteratively generate content items 152, such as images, videos, or audio.

[0060] For example, a diffusion model neural network can generate an image by performing a dediffusion process to produce a diffusion output that includes, or otherwise specifies, multiple color values ​​of pixels in the image arranged according to a specified order.

[0061] As another example, a diffusion model neural network performs a despreading process to generate an image and produces a diffusion output that contains or otherwise specifies multiple tokens representing image patch embeddings in the image, which can then be processed by a decoder neural network for image generation.

[0062] Examples of diffusion model neural networks include those described below: Chitwan Saharia et al., Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-36494, 2022; Aditya Ramesh et al., Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125; and Robin Rombach et al., High-resolution image synthesis with latent diffusion model, Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2022.

[0063] As another example, a diffusion model neural network can generate an image or video with multiple frames (each frame being an image) by iteratively predicting masked tokens in a decoding process in a discrete token space, as described, for example, in Huiwen Chang, et al., Muse: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301.00704, 2023 and Huiwen Chang et al., Maskgit: Masked generative image transformer. arXiv preprint arXiv:2202.04200, 2022.

[0064] Figure 2 is a diagram of an exemplary environment 200, which includes the content item generation system 100 and computing device 160 of Figure 1. In addition, the environment 200 includes at least one of the following: a search engine 120, an operating system 130, or a training system 140.

[0065] In some embodiments, the environment 200 may include a network such as a local area network (LAN), a wide area network (WAN), the internet, or a combination thereof. If included, the network connects the content item generation system 100, computing devices 160, and one or more of the search engine 120, operating system 130, or training system 140.

[0066] The content item generation system 100 can operate in conjunction with an artificial intelligence software application 162 (or "AI application 162" for short) installed on a computing device 160. Examples of computing devices 160 include personal computers, game devices, mobile communication devices, digital assistant devices, augmented reality (AR) devices, virtual reality (VR) devices, wearable devices, and other electronic devices.

[0067] The user can interact with the content item generation system 100 using the AI ​​application 162. For example, the user can use the AI ​​application 162 to provide contextual input 104 as part of a request 102 for content item 152, or associated with it, by using the input device of the computing device 160, and the generative neural network 110 can provide the content item 152 for presentation to the user within the AI ​​application on the display device of the computing device 160.

[0068] Examples of input devices include keyboards, mice, microphones, AR / VR input devices, and touchscreens. Examples of output devices include monitors, screens, and speakers. Output devices can be used to display images, text, videos, and / or play audio for the user.

[0069] Referring briefly to Figure 3, Figure 300 is an exemplary content item generation system 100 that generates content items based on prompts including derived content 105 and metadata 107. The user interacts with the content item generation system 100 using the AI ​​application 162 to generate a prompt 108, and the generative neural network 110 processes the prompt 108 to generate content items 152 that can then be presented within the AI ​​application 162.

[0070] Prompt 108 contains derived content from the source electronic document 106. For example, as shown in Figure 3, the derived content 105 may include text or image data that is transferred (e.g., copied or moved) by the user from the source electronic document 106.

[0071] Furthermore, in many cases, prompt 108 also includes requests made by the user referring to derived content 105. For example, if derived content 105 contains text data, the request could be to translate, explain, summarize, expand on, analyze, or otherwise process derived content 105. Additionally or alternatively, the request could be to generate data in some other modality conditioned on derived content 105, such as image data, video data, or audio data. Another example is if derived content 105 contains image data, the request could be to generate a text caption or some other description of derived content 105.

[0072] In response to a user providing derived content 105, the content item generation system 100 retrieves metadata 107 based on the derived content 105. To retrieve the metadata 107, the content item generation system 100 can interact with one or more of the search engine 120, operating system 130, or training system 140 included in the environment 200.

[0073] The search engine 120 is accessible by the content item generation system 100 and can be any suitable search engine that searches any suitable corpus of documents, e.g., web pages, books, or other documents. For example, the search engine 120 could be an internet search engine that searches for electronic documents available on the internet and returns results. As another example, the search engine 120 could be a different search engine that searches a private corpus of documents, e.g., electronic documents available on an internal network or stored in a collection of one or more databases.

[0074] In embodiments where the search engine 120 is included in the environment 200, in response to the computing device 160 providing the content item generation system 100 with contextual input 104 containing derived content, the content item generation system 100 can use the search engine 120 to perform a search within the document corpus based on the derived content and identify electronic documents related to the derived content, for example, those that meet the relevance threshold for the derived content. The content item generation system 100 can then use the identified electronic documents as source electronic documents 106 and retrieve metadata associated with the source electronic documents 106.

[0075] The operating system 130, if included, can run on the computing device 160. The operating system provides an interface between the computing device's hardware (e.g., input / output devices and a processor that executes instructions obtained from computer-readable media) and software. The operating system provides a platform for running various software applications on the computing device.

[0076] As described above, the software application may include AI applications 162, and one or more software applications may provide transfer functions. Examples of transfer functions include copy commands, cut commands, and paste commands. Examples of such software applications include document processing applications, spreadsheet applications, presentation applications, web browser applications, email applications, camera applications, photo album applications, and image / video processing applications.

[0077] Examples of transfers may include content transfers between a source electronic document and a software application (for example, between a source electronic document 106 and an AI application that can be used by a user to interact with a content item generation system 100), between two different electronic documents, or between two different software applications.

[0078] A user of computing device 160 can select any content within source electronic document 106 presented in AI application 162 (or other software application) on the display device of computing device 160, provide a transfer request to transfer the selected content to AI application 162 (optionally via the system clipboard), and use AI application 162 to provide context input 104 containing the transferred content to content item generation system 100. For example, a transfer request may include a copy request to copy the selected content from source electronic document 106 to AI application 162, or a cut request to move the selected content from source electronic document 106 to AI application 162.

[0079] In embodiments where the operating system 130 is included in the environment 200, in response to the computing device 160 providing a context input 104 containing content transferred to the content item generation system 100, the content item generation system 100 may, for example, make a data request to the operating system 130 to obtain from the operating system 130 metadata associated with the source electronic document 106, metadata associated with the transferred content, or both.

[0080] In some cases, a paste command may include an extended paste command that automatically transfers, for example, a predetermined set of content provided by a content provider, in addition to the selected content. In such cases, the content item generation system 100 may further obtain metadata, for example, by making a data request to the operating system 130, which generally contains additional information or information different from what would be included in the predetermined set of content. In fact, in these cases, the content item generation system 100 may delete the automatically transferred content, resulting in a context input that includes the selected content but excludes the automatically transferred content.

[0081] The training system 140 is a system implemented as a computer program on one or more computers in one or more locations that trains the generative neural network 110 and determines the trained values ​​of the generative neural network's parameter set. That is, the generative neural network 110 is trained by the training system 140 so that it can generate content items 152 containing data from a single modality or multiple modalities by processing prompts 108 according to the trained values ​​of the generative neural network's parameter set.

[0082] For example, the training system 140 may have trained the generative neural network 110 in two stages: a pre-training stage and a fine-tuning stage. In the pre-training stage, the generative neural network 110 is pre-trained by the training system 140 on a large pre-training dataset based on optimizing one or more unsupervised or self-supervised objective functions, such as a maximum likelihood objective function.

[0083] Examples of large pre-training datasets include large datasets of text in one or more natural languages, e.g., text publicly available from the internet or other text corpora; large datasets of computer code publicly available from the internet or other code repositories, e.g., in one or more programming languages, e.g., Python, C++, C#, Java, Ruby, PHP, etc.; large datasets of audio samples, e.g., audio recordings or waveforms representing audio recordings; large datasets of images, each containing an array of pixels; large datasets of videos, each containing a temporal sequence of frames; or large multimodal datasets containing two or more combinations of these datasets.

[0084] In the fine-tuning phase, the pre-trained generative neural network 110 is then fine-tuned for the generative task through fine-tuning adaptive techniques or other adaptation techniques, such as prompt tuning or instruction tuning. The generative task may include any combination of one or more of the above generative tasks and possibly other tasks. Examples of fine-tuning adaptive techniques include supervised fine-tuning (SFT), human feedback-based reinforcement learning (RLHF), and AI feedback-based reinforcement learning (RLHF), which use different training objectives, different fine-tuning datasets, or both.

[0085] In some embodiments of the training system 140, low-rank adaptive techniques or other techniques may be further used to achieve computationally efficient fine-tuning of the pre-trained generative neural network 110 by reducing the total number of parameter values ​​that need to be trained during the fine-tuning phase.

[0086] Training by the training system 140 gives the generative neural network 110 a knowledge cutoff date. The knowledge cutoff date represents the most recent publication date of the data included in the training data used to train the generative neural network 110. Therefore, the training data used by the training system 140 to train, such as the pre-training dataset or the fine-tuning dataset, contains only information up to the knowledge cutoff date and does not contain any latest information that has become available only after the knowledge cutoff date.

[0087] In embodiments where the training system 140 is included in the environment 200, the content item generation system 100 can obtain metadata related to the generative neural network 110 from the training system 140, which may include its knowledge cutoff date and potentially other information such as model version and model size.

[0088] As part of the fine-tuning phase, in some embodiments, the training system 140 may have trained the generative neural network 110 on the metadata-enhanced training dataset using one of the fine-tuning adaptive techniques described above.

[0089] For example, if the generative neural network 110 is a neural network for a language model, the metadata-attached training dataset may include multiple training prompts. Each training prompt includes a training context input containing derived content from a source electronic document, and training metadata containing metadata related to the source electronic document.

[0090] For example, if the generative neural network 110 is a diffusion model neural network, the metadata-enhanced training dataset may include multiple training content items, such as image content items, video content items, or audio content items. Each training content item is associated with training metadata that contains metadata associated with the source electronic document from which the training content item is obtained.

[0091] Figure 4 is a flowchart of an exemplary process 400 that generates content items using a generative neural network based on metadata from a source document. For convenience, process 400 is described as being performed by a system of one or more computers located in one or more locations. For example, a content item generation system appropriately programmed according to this specification, e.g., the content item generation system 100 shown in Figure 1, can perform process 400.

[0092] The system receives a request from a computing device to generate content items using a context-conditioned generative neural network (step 402). The system can use the generative neural network to generate any type of content items, such as text data items, image data items, video content items, audio data items, etc.

[0093] Contextual input provides context for content items generated by a generative neural network. Contextual input can include data provided by the user through the use of input devices on the computing device. In particular, contextual input includes derived content from a source electronic document. For example, contextual input can include text fragments (in natural or computer language), images (or patches of images), videos (or video frames), or audio (or audio frames) transferred from a source electronic document.

[0094] The system retrieves metadata associated with the source electronic document (step 404). For example, the system can retrieve metadata by using a search engine to perform a search within a corpus of documents based on derived content, and identify electronic documents related to the derived content, such as electronic documents that meet a relevance threshold with respect to the derived content. The system can then use the identified electronic document as the source electronic document and retrieve the metadata associated with the source electronic document. As another example, the system can make a data request to an operating system running on a computing device and retrieve metadata from the operating system.

[0095] Optionally, the system may also retrieve metadata related to the derived content. For example, the system may similarly retrieve metadata from the operating system running on the computing device. Furthermore, and still optional, the system may also retrieve metadata associated with the generative neural network. For example, the system may retrieve metadata from the training system that trained the generative neural network, or from a database that holds metadata associated with the generative neural network.

[0096] The system generates prompts for the generative neural network based on contextual input, metadata associated with the source electronic document, and optionally, metadata associated with derived content or metadata associated with the generative neural network (step 406). Thus, the prompts include not only contextual input directly provided by the user, but also additional information not directly provided by the user (metadata obtained by the system).

[0097] In some implementations, the system can generate prompts by concatenating contextual input with retrieved metadata. For example, a prompt might take the following form: <Contextual input><Metadata>, Here, "<context input>" represents context input received from the user, including derived content from the source electronic document, and "<metadata>" represents metadata associated with the source electronic document, metadata associated with the derived content, metadata associated with the generative neural network, or any combination thereof. In this example, <metadata> can be placed before <context input>.

[0098] In other embodiments, the prompt may take the following form. <System prompt><Contextual input><Metadata>, Here, "<system prompt>" can represent additional information generated by the system based on metadata. In this example, "<system prompt>", "<metadata>", and "<context input>" can be arranged in different orders.

[0099] For example, the system can compare the knowledge cutoff date of a generative neural network with the publication date of the source electronic document and generate information, as part of a system prompt, indicating whether the knowledge cutoff date is earlier than the publication date.

[0100] As a concrete example of this, if we assume that the publication date of the source electronic document is "2024-May-01" and the knowledge cutoff date of the generative neural network is "2024-March-01", then the information to be included in the system prompt could be formatted as a text sequence representing the time difference, such as "the context is two months after the knowledge cutoff."

[0101] As another example, the system may determine that the source electronic document is not included as part of a large pre-training dataset used to pre-train a generative neural network. Furthermore, the source electronic document may also not be included in the fine-tuning dataset used to subsequently tune the generative neural network. In some cases, such a determination may be made by the system after determining that a particular content provider offering the source electronic document is not within a list of known content providers offering data for the pre-training and fine-tuning datasets.

[0102] In this embodiment, the system can generate information, as part of a system prompt, indicating that the training data used to train the generative neural network does not contain the source electronic document. In some cases, including such information may encourage the generative neural network to utilize a search engine (or other external tool) to generate content items more efficiently.

[0103] As a concrete example of this, if the system determines that the source electronic document is not included in the training data, the information to be included in the system prompt may then be formatted as a text sequence indicating to the generative neural network that the contextual input contains information not included in the training data used to train the generative neural network, i.e., "This information is from a set of data that was not used during training."

[0104] In any embodiment, the prompt may also include additional information, such as task-specific information or other system-generated information, such as a predetermined set of system instructions.

[0105] Furthermore, or alternatively, “<system prompt>” may represent a predetermined system prompt, which may include, for example, a predetermined set of instructions on how content items should be generated, a list of examples of content items to be generated, or a list of search engine results obtained by using a search engine (or external data obtained by other external tools) based on derived content contained in the contextual input.

[0106] In some embodiments, the system can provide prompts to present information to the user. When provided for presentation, metadata can be presented to the user, for example, along with contextual input. Alternatively, metadata may be presented separately from contextual input, such as in a footnote. Optionally, metadata may be displayed as a short summary that expands in response to selection actions (e.g., double-click, hover, etc.) to provide more detailed information.

[0107] The system uses a generative neural network to process prompts and generate content items (step 408). The generative neural network can generate any type of content item, such as text data items, image data items, video content items, audio data items, etc.

[0108] In some embodiments, a generative neural network can generate content items by having a set of parameters for the generative neural network and processing prompts according to the set of parameters for the generative neural network.

[0109] In general, as described above, by including metadata in the prompt, a generative neural network can generate content items that are more informative and relevant to the contextual input than content items generated based solely on the contextual input (even though the same set of parameters for the generative neural network are used).

[0110] In some embodiments, the generative neural network has a set of generative neural network parameters and a set of adaptive parameters, each corresponding to (or mapping to) a set of predetermined use cases. Each set of adaptive parameters is an additional set of parameters that can be used in conjunction with the set of generative neural network parameters to adapt the generative neural network to generate content items.

[0111] For example, a generative neural network may have a first set of adaptive parameters corresponding to a first use case where the derived content included in the contextual input is after the generative neural network's knowledge cutoff date, a second set of adaptive parameters corresponding to a second use case where the derived content included in the contextual input is before the generative neural network's knowledge cutoff date, and so on.

[0112] As another example, a generative neural network may have a first set of adaptive parameters corresponding to a first use case in which the content included in the contextual input is derived from a source electronic document provided by a first content provider, a second set of adaptive parameters corresponding to a second use case in which the content included in the contextual input is derived from a source electronic document provided by a second content provider, and so on.

[0113] In these embodiments, the system may further utilize a classification neural network configured to classify prompts into one of several predetermined use cases. For example, to generate content items, the system may first use a classification neural network to process metadata and optionally contextual input to generate a classification output specifying a particular user case, and then use both (i) a set of generation neural network parameters and (ii) a set of specific adaptive parameters corresponding to a particular use case to process prompts and generate content items.

[0114] The system provides content items to the user on a display device (step 410).

[0115] This specification uses the term “configured” in relation to systems and computer program components. When one or more computer systems are configured to perform a particular operation or action, it means that, while in operation, software, firmware, hardware, or a combination thereof is installed on the system that causes the system to perform that operation or action. When one or more computer programs are configured to perform a particular operation or action, it means that one or more programs, when executed by a data processing device, contain instructions that cause the device to perform that operation or action.

[0116] The subject matter and functional embodiments described herein can be implemented in digital electronic circuits, tangibly embodied computer software or firmware, or computer hardware, including the structures disclosed herein and their structural equivalents, or one or more combinations thereof. Embodiments of the subject matter described herein may be implemented as one or more computer programs, for example, one or more modules of computer program instructions, encoded on tangible, non-temporary storage media, which are executed by or control the operation of a data processing device. The computer storage media may be a machine-readable storage device, a machine-readable storage board, a random-access memory device or a serial-access memory device, or one or more combinations thereof. Alternatively or additionally, the program instructions may be encoded into artificially generated propagated signals, for example, mechanically generated electrical signals, optical signals or electromagnetic signals, which are generated to encode information for transmission to a receiving device suitable for execution by a data processing device.

[0117] The term "data processing device" refers to data processing hardware and encompasses all types of devices, machines, and equipment for processing data, including, for example, programmable processors, computers, or multiple processors or computers. A device may also be, or further include, specialized logic circuits such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). Optionally, in addition to hardware, a device may include code that constitutes the execution environment for computer programs, such as processor firmware, protocol stacks, database management systems, operating systems, or one or more combinations thereof.

[0118] Computer programs, which may be called or described as programs, software, software applications, apps, modules, software modules, scripts, or code, can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. A program may, but is not required to, correspond to a file in a file system. A program may be stored in a single file dedicated to a program of interest, in part with other programs or data, for example, in a file holding one or more scripts stored in a markup language document, or in multiple collaborative files, for example, in a file storing one or more modules, subprograms, or parts of code. A computer program can be deployed to run on one computer, or it can be located in one place or distributed across multiple locations and interconnected by a data communication network to run on multiple computers.

[0119] In this specification, the term “database” is used broadly to refer to any collection of data. The data does not need to be structured in any particular way, or not structured at all, and can be stored on one or more storage devices. Therefore, for example, an index database may contain multiple collections of data, each of which may be organized and accessed in a different way.

[0120] Similarly, in this specification, the term “engine” is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine is implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers are dedicated to a particular engine, while in other cases, multiple engines may be installed and run on the same one or more computers.

[0121] The processes and logic flows described herein can be performed by one or more programmable computers executing one or more computer programs to act on input data and produce outputs, thereby performing their functions. Alternatively, the processes and logic flows can be performed by special-purpose logic circuits, such as FPGAs or ASICs, or by a combination of special-purpose logic circuits and one or more programmed computers.

[0122] A computer suitable for running computer programs may be based on a general-purpose or dedicated microprocessor, or both, or other types of central processing unit. Generally, the central processing unit receives instructions and data from read-only memory, random-access memory, or both. The basic components of a computer are the central processing unit for executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be complemented by or incorporated into special-purpose logic circuits. Generally, a computer also includes one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or is operablely connected to them to receive data from them, transmit data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer may be incorporated into other devices, such as mobile phones, personal digital assistants (PDAs), mobile audio or video players, game consoles, Global Positioning System (GPS) receivers, or portable storage devices (such as Universal Serial Bus (USB) flash drives) (these are just a few examples).

[0123] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0124] To provide user interaction, embodiments of the subject matter described herein may be implemented in a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse or trackball, to which the user can provide input to the computer. Other types of devices can also be used to interact with the user. For example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or haptic feedback, and input from the user may be received in any form, including acoustic, voice, or haptic input. Furthermore, the computer may interact with the user by sending documents to and receiving documents from the device used by the user, for example, by sending a web page to a web browser on the user's device in response to a request received from a web browser. The computer may also interact with the user by sending text messages or other forms of messages to a personal device (for example, a smartphone running a messaging application) and then receiving a response message from the user.

[0125] Data processing devices for implementing machine learning models may also include, for example, dedicated hardware accelerator units for handling the computationally intensive parts of machine learning training or production, such as inference and workloads.

[0126] Machine learning models can be implemented and deployed using machine learning frameworks, such as the TensorFlow framework or the JAX framework.

[0127] Embodiments of the subject matter described herein may be implemented in a computing system that includes, for example, a data server as a backend component, or in a computing system that includes a middleware component, for example, an application server, or in a computing system that includes a client computer having a frontend component, for example, a graphical user interface, a web browser, or an application that enables a user to interact with embodiments of the subject matter described herein, or in a computing system that includes one or more such backend, middleware, or frontend components in any combination. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and, for example, the Internet.

[0128] A computing system can include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The client-server relationship arises from computer programs that run on each computer and have a client-server relationship with each other. In some embodiments, the server sends data, such as an HTML page, to a user device for the purpose of displaying data to a user interacting with a device acting as a client and receiving user input from that user. Data generated on the user device, such as the results of user interactions, can be received from the device by the server.

[0129] While this specification includes details of many specific embodiments, these should not be construed as limiting the scope of any invention or claimable content, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described herein in the context of other embodiments may also be realized in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be realized individually or in any suitable subcombination in multiple embodiments. Furthermore, features may be described above as functioning in a particular combination, and even if initially claimed as such, one or more features from the claimed combination may be removed from the combination, and the claimed combination may cover a subcombination or a variation of a subcombination.

[0130] Similarly, while operations are shown in the drawings and described in a specific order in the claims, this should not be understood as requiring that such operations be performed in a specific or sequential order shown, or that all shown operations be performed, in order to obtain the desired results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described can generally be integrated into a single software product or packaged into multiple software products.

[0131] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions described in the claims may be performed in a different order, and this may still yield the desired results. As an example, the process shown in the accompanying diagram does not necessarily require to be performed in the specific order or sequence shown to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method performed by one or more computers, wherein the method is Receiving a request from a user to generate content items using a contextually conditioned generative neural network, wherein the contextual input includes content derived from a source electronic document. Acquisition of metadata associated with the source electronic document, wherein acquisition of metadata includes receiving the metadata from an operating system running on one or more computers. To generate prompts for the generative neural network based on the contextual input and the metadata associated with the source electronic document, To generate the aforementioned content items, the generative neural network is used to process the prompt, To provide the content items for presentation to the user, Methods that include...

2. Generating the prompts for the generative neural network is, The method according to claim 1, comprising generating the prompt including the context input and the metadata.

3. A method performed by one or more computers, wherein the method is Receiving a request from a user to generate content items using a contextually conditioned generative neural network, wherein the contextual input includes content derived from a source electronic document. To obtain metadata associated with the aforementioned source electronic document, To generate a prompt for the generative neural network based on the contextual input and the metadata associated with the source electronic document, wherein the prompt includes additional information generated based on the contextual input, the metadata, and the metadata associated with the source electronic document and the metadata associated with the generative neural network. To generate the aforementioned content items, the generative neural network is used to process the prompt, To provide the content items for presentation to the user, Methods that include...

4. The method according to claim 3, wherein the metadata associated with the generative neural network includes the cutoff date of the generative neural network, the cutoff date representing the most recent publication date of the data contained in the training data used to train the generative neural network.

5. Obtaining the metadata associated with the source electronic document is: To identify electronic documents related to the aforementioned content, a search is performed within the document corpus, The method according to claim 1, further comprising using the metadata associated with the identified electronic document as the metadata associated with the source electronic document.

6. The method according to claim 1, wherein the content derived from the source electronic document includes content copied from the source electronic document.

7. A system comprising one or more computers and one or more storage devices that, when executed by the one or more computers, store instructions causing the one or more computers to perform an operation, wherein the operation is Receiving a request from a user to generate content items using a contextually conditioned generative neural network, wherein the contextual input includes content derived from a source electronic document. Acquisition of metadata associated with the source electronic document, wherein acquisition of metadata includes receiving the metadata from an operating system running on one or more computers. To generate prompts for the generative neural network based on the contextual input and the metadata associated with the source electronic document, To generate the aforementioned content items, the generative neural network is used to process the prompt, To provide the content items for presentation to the user, A system that includes this.

8. Generating the prompts for the generative neural network is, The system according to claim 7, comprising generating the prompt including the context input and the metadata.

9. A system comprising one or more computers and one or more storage devices that, when executed by the one or more computers, store instructions causing the one or more computers to perform an operation, wherein the operation is Receiving a request from a user to generate content items using a contextually conditioned generative neural network, wherein the contextual input includes content derived from a source electronic document. To obtain metadata associated with the aforementioned source electronic document, To generate a prompt for the generative neural network based on the contextual input and the metadata associated with the source electronic document, wherein the prompt includes additional information generated based on the contextual input, the metadata, and the metadata associated with the source electronic document and the metadata associated with the generative neural network. To generate the aforementioned content items, the generative neural network is used to process the prompt, To provide the content items for presentation to the user, A system that includes this.

10. The system according to claim 9, wherein the metadata associated with the generative neural network includes the cutoff date of the generative neural network, the cutoff date representing the most recent publication date of the data contained in the training data used to train the generative neural network.

11. Obtaining the metadata associated with the source electronic document is: To identify electronic documents related to the aforementioned content, a search is performed within the document corpus, The system according to claim 7, comprising using the metadata associated with the identified electronic document as the metadata associated with the source electronic document.

12. The system according to claim 7, wherein the content derived from the source electronic document includes content copied from the source electronic document.

13. One or more non-temporary computer storage media that, when executed by one or more computers, store instructions that cause the one or more computers to perform an action, wherein the action is Receiving a request from a user to generate content items using a contextually conditioned generative neural network, wherein the contextual input includes content derived from a source electronic document. Acquisition of metadata associated with the source electronic document, wherein acquisition of metadata includes receiving the metadata from an operating system running on one or more computers. Based on the contextual input and the metadata related to the source electronic document, a prompt for the generative neural network is generated. To generate the content items, the generative neural network is used to process the prompts, To provide the content items for presentation to the user, Computer storage media, including...

14. Generating the prompts for the generative neural network is, The computer storage medium according to claim 13, comprising generating the prompt including the context input and the metadata.

15. One or more non-temporary computer storage media that, when executed by one or more computers, store instructions that cause the one or more computers to perform an action, wherein the action is Receiving a request from a user to generate content items using a contextually conditioned generative neural network, wherein the contextual input includes content derived from a source electronic document. To obtain metadata associated with the aforementioned source electronic document, Generating a prompt for the generative neural network based on the contextual input and the metadata associated with the source electronic document, wherein the prompt includes additional information generated based on the contextual input, the metadata, and the metadata associated with the source electronic document and the metadata associated with the generative neural network. To generate the aforementioned content items, the generative neural network is used to process the prompt, To provide the content items for presentation to the user, Computer storage media, including...

16. The computer storage medium according to claim 15, wherein the metadata associated with the generative neural network includes the cutoff date of the generative neural network, the cutoff date representing the most recent publication date of the data contained in the training data used to train the generative neural network.

17. Obtaining the metadata associated with the source electronic document is: To identify electronic documents related to the aforementioned content, a search is performed within the document corpus, The computer storage medium according to claim 13, comprising using the metadata associated with the identified electronic document as the metadata associated with the source electronic document.

18. The method according to claim 1, wherein the generative neural network includes a multimodal neural network.

19. The system according to claim 7, wherein the generative neural network includes a multimodal neural network.

20. The computer storage medium according to claim 13, wherein the generative neural network includes a multimodal neural network.

21. The method according to claim 3, wherein the generative neural network includes a multimodal neural network.

22. The system according to claim 9, wherein the generative neural network includes a multimodal neural network.

23. The computer storage medium according to claim 15, wherein the generative neural network includes a multimodal neural network.

24. The method according to claim 1, wherein the metadata associated with the generative neural network includes the cutoff date of the generative neural network, the cutoff date representing the most recent publication date of the data contained in the training data used to train the generative neural network.

25. The system according to claim 7, wherein the metadata associated with the generative neural network includes the cutoff date of the generative neural network, the cutoff date representing the most recent publication date of the data contained in the training data used to train the generative neural network.

26. The computer storage medium according to claim 13, wherein the metadata associated with the generative neural network includes the cutoff date of the generative neural network, the cutoff date representing the most recent publication date of the data contained in the training data used to train the generative neural network.

27. The method according to claim 3, wherein the content derived from the source electronic document includes content copied from the source electronic document.

28. The system according to claim 9, wherein the content derived from the source electronic document includes content copied from the source electronic document.

29. The computer storage medium according to claim 13, wherein the content derived from the source electronic document includes content copied from the source electronic document.

30. The computer storage medium according to claim 15, wherein the content derived from the source electronic document includes content copied from the source electronic document.

Images