Generated artificial intelligence

Abstract

A method, system, and apparatus for generating digital components includes a computer program encoded on a computer storage medium. A prompt is generated that includes a query and a set of constraints that limit the clauses generated by a language model. The set of constraints includes summaries of specified sources of online content. A plurality of candidate digital components are generated using the clauses generated by the language model using the summaries of the identified sources of online content. One or more post-processing operations are performed that evaluate one or more characteristics of the plurality of candidate digital components. Each of the plurality of candidate digital components is ranked based on the post-processing operations. At least one output digital component from the set of highest-ranked candidate digital components is provided.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 63 / 503,685, filed May 22, 2023. The disclosure of the prior application is considered part of the disclosure of this application and is incorporated by reference into the disclosure of this application. [Background technology]

[0002] This specification relates to data processing and generative artificial intelligence.

[0003] Advances in machine learning are enabling artificial intelligence to be implemented in more applications. For example, large-scale language models are being implemented to enable conversational interactions with computers using natural language rather than a limited set of prompts. This allows for more natural interactions with computers. Summary of the Invention

[0004] In general, one innovative aspect of the subject matter described in this specification may be embodied in a method that includes the following actions: generating, by an artificial intelligence system, a prompt that includes a query and a set of constraints that constrain the clauses generated by the language model, the set of constraints including a summary of a specified source of online content; generating, by the artificial intelligence system, a plurality of candidate digital components using the clauses generated by the language model using the summary of the specified source of online content; performing, by the artificial intelligence system, one or more post-processing operations that evaluate one or more characteristics of the plurality of candidate digital components; ranking, by the artificial intelligence system, each of the plurality of candidate digital components based on the post-processing operations; and providing, by the artificial intelligence system, at least one output digital component that is within the set of highest-ranked candidate digital components.

[0005] These and other embodiments may each optionally include one or more of the following features: A method may include collecting passages from a set of online resources using a site-constrained query that requires collecting passages from one or more network locations specified in a site constraint, and summarizing the passages collected from the one or more network locations into a passage summary.

[0006] Generating the prompt can include inserting at least a portion of the passage summary into the prompt as a context constraint that limits content produced by the language model to a subject matter specified in the context constraint.

[0007] Generating the prompt may include inserting into the prompt entity names of entities referenced by one or more network locations as entity constraints specifying that content identifying the entities must be included in content created by the language model.

[0008] Generating the prompt can include inserting a grounding constraint into the prompt that requires the content produced by the language model to be present in a specified set of online resources.

[0009] Inserting a grounding constraint into the prompt may include inserting a second-level domain into the prompt that requires that content produced by the language model reside in a resource within the second-level domain.

[0010] Generating the plurality of candidate digital components may include combining the output of the language model with links to second level domains.

[0011] Performing one or more post-processing operations to evaluate one or more characteristics of the plurality of candidate digital components may include: generating, for each clause of the output of the language model, a grounding score that specifies the likelihood that the clause is factual; filtering the output of the language model by removing one or more clauses having a grounding score that does not satisfy a grounding threshold that distinguishes between clauses classified as factual and clauses classified as not factual; and replacing the one or more removed clauses with other clauses of the output having a grounding score that satisfies the grounding threshold.

[0012] Performing one or more post-processing operations to evaluate one or more characteristics of the plurality of candidate digital components may include, for each given candidate digital component among the plurality of candidate digital components, evaluating the relevance of a clause of the given candidate digital component to the prompt query; evaluating a level of completeness of the clause that specifies how comprehensively the clause of the given candidate digital component describes the topic of the second-level domain linked by the given candidate digital component; and evaluating the tone of the clause of the candidate digital component to determine whether the clause characterizes the item as positive or negative.

[0013] The details of one or more embodiments of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a block diagram of an exemplary environment in which generative artificial intelligence can be implemented. [Figure 2] FIG. 1 is a block diagram illustrating interactions between an artificial intelligence system, a language model, and a client device. [Figure 3]1 is a flowchart of an exemplary process for artificial intelligence to generate creative and fact-based digital components. [Figure 4] FIG. 1 is a block diagram of an exemplary computer. DETAILED DESCRIPTION OF THE INVENTION

[0015] Like reference symbols and designations in the various drawings refer to like elements.

[0016] This specification describes techniques for enabling artificial intelligence to generate new digital components that are creative and fact-based. Artificial intelligence (AI) is a field of computer science focused on creating intelligent agents that can learn and act autonomously (e.g., without human intervention). Artificial intelligence systems can utilize one or more of: (i) machine learning, which focuses on developing algorithms that can learn from data; (ii) natural language processing, which focuses on understanding and generating human language; and / or (iii) computer vision, which is a field focused on understanding and interpreting images and videos. Artificial intelligence systems can include generative models that generate new content (e.g., images / video, text, audio, or other content) in response to input prompts.

[0017] The techniques described throughout this specification enable artificial intelligence to generate numerous new digital components using various combinations of creative as well as factual text and / or images. For example, an artificial intelligence system can gather information from various sources, such as various web pages or other trusted online resources, and combine this information in different ways to create different candidate digital components. Generally speaking, the system utilizes input prompts into a language model, such as a large-scale language model (LLM), that outputs multiple clauses. The system uses the clauses to create multiple different digital components and then performs post-processing to select an output digital component from among the different candidate digital components.

[0018] As described in more detail below, the prompts are specialized (e.g., created or enhanced) to improve the overall quality of the generated candidate digital components. Post-processing operations are then used to evaluate the generated candidate digital components against each other to determine which candidate digital components are of higher quality than the other candidate digital components (e.g., given the current context), and the higher quality digital component or components are output to a computing device (e.g., the user's computer, mobile device, tablet device, audio device, gaming device, etc.).

[0019] The use of specialized prompts reduces wasted computing resources that would otherwise be used to generate lower-quality digital components if more general prompts were used. Similarly, as described in more detail below, using specialized prompts to constrain the parameters used by a language model to generate candidate digital components can reduce the number of candidate digital components generated, thereby conserving computing resources and generating output more quickly. For example, by constructing prompts to limit the types of content that can be included in a generated candidate digital component, the language model will not generate candidate digital components that violate the prompt constraints, thereby avoiding the creation of unnecessary candidate digital components. This reduces the time required to generate candidate digital components, the memory required to store them, and the computing resources required to generate and evaluate them. All of this contributes to a system that can create new digital components more quickly, allowing new digital components to be created and provided in a real-time interactive environment, for example, in response to a user's search query.

[0020] Post-processing operations can include, for example, evaluating the candidate digital components based on various criteria and scoring each of the candidate digital components based on the evaluation. For example, one post-processing operation can perform a prediction regarding the likelihood that a particular candidate digital component is unsubstantiated (e.g., contains information that cannot be verified in a specified corpus). Using this type of post-processing operation allows for more relaxed constraints in the construction of specialized prompts, thereby enabling the language model to generate more creative candidate digital components while still ensuring that the output digital component has at least a baseline level of veracity. The post-processing operation can also use various heuristics to evaluate different characteristics of each of the candidate digital components, and scores can be assigned based on the various heuristics. In some implementations, the scores are weighted and aggregated to create a final score, which is used to rank the candidate digital components. Additionally, or alternatively, a machine learning model can be trained to score the quality of digital components, and those scores can be used to rank the candidate digital components. One or more of the top candidate digital components are then selected to serve as the output digital component.

[0021] As used throughout this specification, the phrase "digital component" refers to a discrete unit of digital content or information (such as a video clip, audio clip, multimedia clip, game content, image, text, bullet points, artificial intelligence output, language model output, or other unit of content). A digital component can be stored electronically on a physical memory device as a single file or as a collection of files, and a digital component can take the form of a video file, audio file, multimedia file, image file, or text file and include advertising information; therefore, an advertisement is a type of digital component.

[0022] 1 is a block diagram of an exemplary environment 100 in which generative artificial intelligence can be implemented. The exemplary environment 100 includes a network 102, such as a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof. The network 102 connects electronic document servers 104, user devices 106, digital component servers 108, and service appliances 110. The exemplary environment 100 may include many different electronic document servers 104, user devices 106, and digital component servers 108.

[0023] A client device 106 is an electronic device that can request and receive online resources over the network 102. Exemplary client devices 106 include personal computers, gaming devices, mobile communication devices, digital assistant devices, augmented reality devices, virtual reality devices, and other devices that can send and receive data over the network 102. A client device 106 typically includes a user application, such as a web browser, to facilitate sending and receiving data over the network 102, although native applications (other than a browser) executed by the client device 106 can also facilitate sending and receiving data over the network 102.

[0024] A gaming device is a device that allows a user to participate in a gaming application. For example, a user controls one or more characters, avatars, or other rendered content displayed in the gaming application. A gaming device typically includes a computer processor, a memory device, and a controller interface (physical or visually rendered) that allows user control over the content rendered by the gaming application. A gaming device can store and execute gaming applications locally, or execute gaming applications that are at least partially stored and / or provided by a cloud server (e.g., online gaming applications). Similarly, a gaming device can execute gaming applications and interface with a gaming server that "streams" the gaming application to the gaming device. A gaming device may be a tablet device, a mobile communication device, a computer, or other device that performs functions other than running gaming applications.

[0025] Digital assistant devices include devices that include a microphone and a speaker. Digital assistant devices are generally capable of receiving input via voice and can respond with content using audible feedback and present other audible information. In some situations, the digital assistant device also includes a visual display or is in communication with a visual display (e.g., via a wireless or wired connection). If a visual display is present, feedback or other information can also be provided visually. In some situations, the digital assistant device can control other devices, such as lights, locks, cameras, room temperature control devices, alarm systems, and other devices registered with the digital assistant device.

[0026] As shown, client device 106 is presenting an electronic document 150. An electronic document is data that presents a set of content at client device 106. Examples of electronic documents include web pages, word processing documents, portable document format (PDF) documents, images, videos, search result pages, and feed sources. Native applications (e.g., "apps" and / or game applications), such as applications installed on mobile, tablet, or desktop computing devices, are also examples of electronic documents. Electronic documents may be provided to client device 106 by electronic document server 104 ("Electronic Document Server").

[0027] For example, the electronic document server 104 may include a server hosting a publisher's website. In this example, the client device 106 may initiate a request for a given publisher web page, and the electronic server 104 hosting the given publisher web page may respond to the request by sending machine-executable instructions that initiate presentation of the given web page at the client device 106.

[0028] In another example, the electronic document server 104 may include an app server from which the client device 106 can download apps. In this example, the client device 106 downloads the files necessary to install the app on the client device 106 and then executes the downloaded app locally (i.e., on the client device). Additionally or alternatively, the client device 106 may initiate a request to execute the app, which is sent to a cloud server. In response to receiving the request, the cloud server may execute the application and stream the application's user interface to the client device 106, so that the client device 106 does not need to execute the app itself. Rather, the client device 106 may present a user interface generated by the cloud server's execution of the app and communicate any user interaction with the user interface back to the cloud server for processing.

[0029] Electronic documents can include a variety of content. For example, electronic document 150 can include native content 152 that is internal to electronic document 150 itself and / or does not change over time. Electronic documents can also include dynamic content that can change over time or on demand. For example, the publisher of a given electronic document (e.g., electronic document 150) can maintain data sources used to populate portions of the electronic document. In this example, the given electronic document can include a script, such as script 154, that causes client device 106 (or a cloud server) to request content (e.g., digital components) from the data sources when the given electronic document is processed (e.g., rendered or executed) by client device 106 (or a cloud server). Client device 106 (or a cloud server) integrates the content (e.g., digital components) retrieved from the data sources into the given electronic document to create a composite electronic document that includes the content retrieved from the data sources.

[0030] In some situations, a given electronic document (e.g., electronic document 150) may include a digital component script (e.g., script 154) that references the service appliance 110 or a particular service provided by the service appliance 110. In these situations, when the given electronic document is processed by the client device 106, the digital component script is executed by the client device 106. Execution of the digital component script configures the client device 106 to generate a request for the digital component 112 (referred to as a “component request”), which is transmitted over the network 102 to the service appliance 110. For example, the digital component script may cause the client device 106 to generate a packetized data request that includes header and payload data. The component request 112 may include event data that specifies characteristics such as the name (or network location) of the server from which the digital component is being requested, the name (or network location) of the requesting device (e.g., client device 106), and / or information that the service appliance 110 can use to select one or more digital components or other content to provide in response to the request. The component request 112 is sent by the client device 106 over the network 102 (eg, a telecommunications network) to a server of the service apparatus 110 .

[0031] The component request 112 may include event data specifying the requested electronic document and other event features, such as characteristics of the electronic document's location where the digital component can be presented. For example, event data specifying a reference (e.g., a URL) to the electronic document (e.g., a web page) where the digital component will be presented, available locations of the electronic document available for presenting the digital component, the size of the available locations, and / or media types eligible for presentation within the locations may be provided to the service device 110. Similarly, event data specifying keywords associated with the electronic document (“document keywords”) or entities referenced by the electronic document (e.g., people, places, or things) may also be included in the component request 112 (e.g., as payload data) and provided to the service device 110 to facilitate identification of eligible digital components for presentation with the electronic document. The event data may also include a search query submitted by the client device 106 to obtain a search results page.

[0032] The component request 112 may also include event data related to other information, such as information provided by a user of a client device, geographic information indicating the state or territory in which the component request was submitted, or other information that provides context about the environment in which the digital component is displayed (e.g., the time of the component request, the day of the week for the component request, the type of device on which the digital component is displayed, such as a mobile device or tablet device). The component request 112 may be transmitted, for example, over a packetized network, and the component request 112 itself may be formatted as packetized data having a header and payload data. The header may specify the destination of the packet, and the payload data may include any of the information described above.

[0033] The service device 110 selects digital components (e.g., third-party content such as video files, audio files, images, text, game content, augmented reality content, and combinations thereof, all of which may take the form of advertising or non-advertising content) to be presented with the given electronic document (e.g., at a location specified by the script 154) in response to receiving the component request 112 and / or using information contained in the component request 112.

[0034] In some implementations, the digital component is selected in less than one second to avoid errors that can occur due to delays in selecting the digital component. For example, delays in providing the digital component in response to a component request 112 can cause page load errors on the client device 106 or can cause portions of the electronic document to remain undisplayed even after other portions of the electronic document have been presented on the client device 106.

[0035] Additionally, as the delay in providing the digital component to the client device 106 increases, the likelihood that the electronic document will not be presented at the client device 106 when the digital component is delivered to the client device 106 increases, thereby adversely affecting the user's experience with the electronic document. Additionally, delays in providing the digital component may result in failure to deliver the digital component, for example, if the electronic document is not presented at the client device 106 when the digital component is provided.

[0036] In some embodiments, the service appliance 110 is implemented within a distributed computing system that includes, for example, a server and a set of computing devices 114 interconnected to identify and distribute digital components in response to requests 112. The set of computing devices 114 may store millions of available digital components (DCs). 1-x ), work together to identify a set of digital components suitable for presentation in an electronic document. Millions of available digital components may be indexed, for example, in a digital component database 116. Each digital component index entry may reference a corresponding digital component and / or may include distribution parameters (DP1 through DP2) that contribute (e.g., triggers, conditions, or limits) to the distribution / transmission of the corresponding digital component. x For example, the distribution parameters can contribute to (e.g., trigger) the transmission of a digital component by requiring a component request that includes at least one criterion that matches one of the distribution parameters of the digital component (e.g., an exact match or some pre-specified level of similarity).

[0037] In some implementations, the distribution parameters for a particular digital component may include distribution keywords that the digital component must match (e.g., with electronic documents, document keywords, or terms specified in the component request 112) to be eligible for presentation. Additionally or alternatively, the distribution parameters may include embeddings that can use various different dimensions of data, such as website details and / or consumption details (e.g., page viewports, user scrolling speed, or other information related to data consumption). The distribution parameters may also require that the component request 112 include information specifying a particular geographic region (e.g., a country or state) and / or information specifying that the component request 112 originated from a particular type of client device (e.g., a mobile device or tablet device) for the digital component to be eligible for presentation. The distribution parameters may also specify an eligibility value (e.g., a ranking score or some other specified value) used to evaluate the eligibility of the digital component for distribution / transmission (e.g., among other available digital components).

[0038] The identification of eligible digital components can be segmented into multiple tasks 117a-117c, which are then allocated among computing devices in a set of multiple computing devices 114. For example, different computing devices 114 in the set can each analyze a different portion of the digital component database 116 to identify various digital components having distribution parameters that match information included in the component request 112. In some implementations, each given computing device 114 in the set can analyze a different data dimension (or set of dimensions) and return (e.g., transmit) results (Result 1-Result 3) 118a-118c of the analysis to the service appliance 110. For example, the results 118a-118c provided by each of the computing devices 114 in the set can identify a subset of digital components that are eligible for distribution in response to the component request and / or a subset of digital components that have a particular distribution parameter. Identifying the subset of digital components can include, for example, comparing event data to the distribution parameter and identifying a subset of digital components that have distribution parameters that match at least some characteristics of the event data.

[0039] The service apparatus 110 aggregates the results 118a-118c received from the set of multiple computing devices 114 and uses information associated with the aggregated results to select one or more of the digital components to provide in response to the request 112. For example, the service apparatus 110 may select the set of acquired digital components (one or more digital components) based on the results of one or more content evaluation processes, as described below. The service apparatus 110 may then generate and transmit, via the network 102, reply data 120 (e.g., digital data representing a response) that enables the client device 106 to integrate the set of acquired digital components into a given electronic document, whereby the set of acquired digital components (e.g., acquired third-party content) and the content of the electronic document are presented together on the display of the client device 106.

[0040] In some implementations, the client device 106 executes instructions included in the reply data 120, thereby configuring and enabling the client device 106 to retrieve a set of acquired digital components from one or more digital component servers 108. For example, the instructions in the reply data 120 may include a network location (e.g., a uniform resource locator (URL)) and a script that causes the client device 106 to send a server request (SR) 121 to the digital component server 108 to retrieve the given acquired digital component from the digital component server 108. In response to this request, the digital component server 108 identifies the given acquired digital component specified in the server request 121 (e.g., within a database that stores multiple digital components) and sends digital component data (DC data) 122 to the client device 106, which presents the given acquired digital component of the electronic document at the client device 106.

[0041] When client device 106 receives digital component data 122, the client device renders the digital component (e.g., third-party content) and presents the digital component at a location specified by or assigned to script 154. For example, script 154 may create a walled garden environment, such as a frame, presented within, e.g., next to, native content 152 of electronic document 150. In some implementations, the digital component is overlaid on (or adjacent to) a portion of native content 152 of electronic document 150, and service appliance 110 may specify the presentation location within electronic document 150 in reply 120. For example, if native content 152 includes video content, service appliance 110 may specify a location or object within a scene depicted in the video content where the digital component is to be presented.

[0042] The service device 110 may also include an artificial intelligence system 160 configured to autonomously generate digital components either prior to a request 112 (e.g., offline) and / or in response to a request 112 (e.g., online or real-time). As described in more detail throughout this specification, the artificial intelligence ("AI") system 160 may collect online content related to a particular entity (e.g., a digital component provider or other entity) and summarize the collected online content using one or more language models 170. The one or more language models 170 may include large-scale language models.

[0043] A large-scale language model ("LLM") is a model trained to generate and understand human language. LLMs are trained on large datasets of text and code and can be used for a variety of tasks. For example, LLMs can be trained to translate text from one language to another; summarize text such as website content, search results, news articles, or research papers; answer questions about text such as "What is the capital of Georgia?"; create chatbots that can converse with humans; and generate creative text such as poetry, stories, and code.

[0044] Language model 170 can be any suitable language model neural network that receives an input sequence composed of text tokens selected from a vocabulary and autoregressively generates an output sequence composed of text tokens from the vocabulary. For example, language model 170 can be a Transformer-based language model neural network or a recurrent neural network-based language model.

[0045] In some circumstances, when the neural network used to implement language model 170 autoregressively generates an output sequence of tokens, language model 170 may be referred to as an autoregressive neural network. More specifically, the autoregressively generated output is created by generating each particular token in the output sequence conditioned on the current input sequence, including every token that precedes a particular text token in the output sequence, i.e., any tokens already generated for any previous positions in the output sequence that precede the particular position of the particular token, and contextual input that provides context for the output sequence.

[0046] For example, the current input sequence when generating a token at any given position in the output sequence may include the input sequence and any preceding tokens that precede the given position in the output sequence. As a particular example, the current input sequence may include the input sequence followed by tokens at any preceding positions that precede the given position in the output sequence. Optionally, the input sequence and the current output sequence may be separated by one or more predetermined tokens in the current input sequence.

[0047] More specifically, to generate a particular token at a particular position in the output sequence, the neural network of language model 170 can process the current input sequence to generate a score distribution (e.g., a probability distribution) that assigns a respective score, e.g., a respective probability, to each token in a vocabulary of tokens. The neural network of language model 170 can then use the score distribution to select a token from the vocabulary as the particular token. For example, the neural network of language model 170 can greedily select the token with the highest score, or can sample the token from the distribution using, for example, kernel sampling or other sampling techniques.

[0048] As a particular example, language model 170 may be an autoregressive transformer-based neural network that includes (i) multiple attention blocks, each applying a self-attention operation, and (ii) an output subnetwork that processes the output of the final attention block to generate a score distribution.

[0049] Takeaway 170 is a snowflake The snow-white squid These are the details of J.Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, DdLCasas, LAHendricks, J. Welbl, A. Clark, et al. Cai, K. Millican, J. Hoffmann, HFSong, J. Aslanides, S. Henderson, R. Ring, S. Young, E. Rutherford, T. Hennigan, J. Menick, A. Cassirer, R. Powell, G. van den Driessche, LAHendricks, M. Rauh, P. Huang, A. Glaese,J. Welbl , S. Dathathri , S. Huang , J. Uesato , J. Chem. Mellor, I. Higgins, A. Creswell, N. McAleese, A. Wu, E. Elsen, SM Jayakumar, E. Buchatskaya, E. Budden, E. Sutherland, K. Simonyan, M. Paganini, L. Sifre, L. Martens, XLLi, A. Kuncoro ,A.Nematzadeh,E.Gribovskaya,D.Donato,A.Lazaridou,A.Mensch,J.Lespiau,M.Tsimpoukelli,N.Grigorev,D.Fritz,T.Sottiaux,M.Pajarskas,T.Pohlen,Z.Gong,D.Toyama,C.de Masson d'Autume, Y. Li, T. Terzi, V. Mikulik, I. Babuschkin, A. Clark, D. de Las Casas, A. Guy, C. Jones, J. Bradbury, M. Johnson, BAHechtman, L. Weidinger, I. Gabriel, WSIsaac, E. Lockhart, S. Osindero, L. Rimell, C. Dyer, O. Vinyals, K. Ayoub, J. Stanway, L. Bennett, D. Hassabis, K. Kavukcuoglu, and G. Irving. Scaling language models: Methods, analysis & insights from training gopher. CoRR, abs / 2112.11446, 2021; Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019; Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. Towards a human-like open-domain chatbot. CoRR, abs / 2001.09977, 2020; and Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020 are described in.

[0050] In general, however, a Transformer-based neural network includes a sequence of attention blocks, and during processing of a given input sequence, each attention block in the sequence receives a respective input hidden state for each input token in the given input sequence. The attention blocks then at least partially update each of the hidden states by applying self-attention to generate a respective output hidden state for each input token. The input hidden state of the first attention block is the embedding of the input token in the input sequence, and the input hidden state of each subsequent attention block is the output hidden state generated by the preceding attention block.

[0051] In this example, the output sub-network processes the output hidden state produced by the last attention block in the sequence of the last input token in the input sequence to produce a score distribution.

[0052] In general, because language models are autoregressive, the service device 110 can use the same language model 170 to generate multiple different candidate output sequences in response to the same request, for example, by using a sample-and-rank decoding strategy, by using beam search decoding from the score distribution generated by the language model 170, by using different random number seeds for the pseudorandom number generator used for sampling for different runs through the language model 170, or by using other decoding strategies that exploit the autoregressive nature of the language model.

[0053] In some implementations, language model 170 is pre-trained, i.e., trained for language modeling tasks that do not require providing evidence in response to user questions, and service device 110 (e.g., using AI system 160) includes having language model 170 generate output sequences according to a predetermined syntax via natural language prompts in the input sequence.

[0054] For example, service device 110 (e.g., AI system 160), or a separate training system, pre-trains language model 170 (e.g., a neural network) on a language modeling task, such as a task requiring, given a current sequence of text tokens, to predict the next token that follows the current sequence in training data. As a particular example, language model 170 may be pre-trained with a maximum likelihood objective on a large dataset of text, e.g., publicly available text from the Internet or other corpus of text.

[0055] In some implementations, AI system 160 can generate prompts 172 that are submitted to language models 170, causing language models 170 to generate output sequences 174, also referred to as passages or simply “output.” AI system 160 can identify a list of online information sources, such as a list of websites or data repositories, and generate the prompts in a (e.g., structured) manner that specifies a set of constraints that language model 160 must use to generate a summary of the information found in the online sources specified in prompt 172. To begin creating output sequences 174, AI system 160 submits prompts 172 to one or more language models 170, which use prompts 172 to evaluate the information found in the online sources specified in prompt 172 and generate output 174 that summarizes the information according to the constraints specified in prompt 172.

[0056] The AI ​​system 160 can use the generated summary as part of another prompt 172 sent to the language model 170. For example, the AI ​​system 160 can insert the generated summary into an additional prompt 172 (e.g., a prompt generated after receiving the summary), which is submitted to the language model 170 as a constraint for generating a clause for use in a digital component generated by the AI ​​system 160. More specifically, assume that the AI ​​system 160 is generating a digital component to provide in response to a request 112 including a keyword / query. In this example, the AI ​​system 160 can generate the additional prompt 172 to include the query and a set of constraints that include the summary received in the previous output 174. The set of constraints of the additional prompt 172 can also include instructions on how to format, style, semantically style, among other things, the clause generated by the language model 170 using the additional prompt 172 (e.g., specifying content that should be excluded from the clause, such as detailed content such as numbers). For example, the additional prompt 172 can take the following form: Write a search digital component called good_output for the query "10g network" and the entity "example_network_provider." The good_output is based on only the essential parts of this summary: "200Mbps Internet on example_network_provider's 10G network with WiFi equipment - $50 / month for 2 years. example_network_provider's 10G network is getting faster and more reliable every day. example_network_provider offers all of this and more. example_network_provider's mobile offers the fastest mobile service with two unlimited lines at the lowest price. Stream the latest series wherever you are. Storm-ready WiFi with wireless connectivity, unlimited data, and battery life." The good_output must be in bullet point format. The good_output must contain exactly three bullet points. Each bullet point must be less than 90 characters. The good_output must not have nested bullet points. The good_output must be catchy and demonstrate value. A good_output should be useful and informative, avoiding tedious details such as numbers.

[0057] In this example prompt, AI system 160 provides the following constraints to language model 170: The query constraint specifies the query "10G network" to which the output clause should relate. -The entity constraint specifies "example_network_provider" as the entity name to use in the output clause. - The summary constraint specifies the content summary to use during clause generation, i.e. "200Mbps Internet over WiFi...". -The style constraint "Must be in bulleted format...must contain exactly three bullets. Each bullet must be less than 90 characters...must not have nested bullets" specifies the format that the output clause must use. - "It must be catchy and show something of value. good_output must be useful and informative and avoid boring details such as numbers." Semantic / tone control constraints define the tone and content of the output clauses generated using the prompt.

[0058] Submitting this additional prompt 172 to language model 170 causes the language model to generate additional output 174, which includes a set of clauses generated according to the query and constraints, and which is electronically communicated to AI system 160. AI system 160 receives the clauses in additional output 174 and generates a plurality of candidate digital components that can be provided in response to request 112. In some implementations, each different candidate digital component includes a different combination of clauses received from language model 170 in additional output 174. For example, assuming additional output 174 includes 12 different clauses and the format of the digital components generated by AI system 160 each includes space for three different clauses, AI system 160 can create 220 different candidate digital components using the three different clauses in each candidate digital component (e.g., 12! / (3!(12-3)!)=220). In some situations, the AI ​​system 160 may also create candidate digital components using different sets of links to online content (e.g., second level domain links to web pages that describe the candidate digital component's topic, phone numbers, etc.) This can continue to exponentially increase the number of different candidate digital components that the AI ​​system 160 can create using additional output 174 clauses of the language model 170.

[0059] The AI ​​system 160 can perform one or more post-processing operations to evaluate one or more characteristics of a plurality of candidate digital components. In some implementations, the post-processing operations can include generating a grounding score for each clause from the additional output 174. The grounding score is a value that specifies the likelihood that the clause is true. The generation of the grounding score is described in further detail with reference to FIG. 2.

[0060] The post-processing operations may also include evaluating the relevance of the clause to the query constraints, the level of completeness of the clause relative to the content located in the links included in the candidate digital components, and / or evaluating the tone (e.g., positive or negative) of the clause. As described in further detail in FIG. 2, the post-processing operations may be used to score or otherwise assign a level of priority to each of the candidate digital components. The AI ​​system 160 may thereby rank multiple candidate digital components relative to one another and ultimately provide one or more of the highest-ranked candidate digital components as output digital components in reply 120 to the request 112. While the operations of the AI ​​system 160 and language model 170 are described above as being performed in response to receiving the request 112, it should be noted that at least some of the operations may be performed prior to receiving the request 112, as described in further detail below with reference to FIG. 2.

[0061] Further, although a single language model 170 is shown in FIG. 1 , different language models can be trained specifically to process different prompts at different stages of the processing pipeline. For example, a more general (e.g., larger) language model can be used to generate a summary of online content as an offline process (e.g., independent of receiving request 112), and then that summary can be inserted into a prompt that is input to a more specialized, faster language model in an online process (e.g., in real time in response to receiving request 112). Furthermore, AI system 160 can generate a set of candidate digital components as an offline process (e.g., before receiving request 112) and store the set of candidate digital components in a database. In this scenario, when AI system 160 receives request 112, AI system 160 can further evaluate and rank the stored candidate digital components based on additional information included in the request and other contextual data (e.g., time of day, day of the week, weather conditions, etc.).

[0062] FIG. 2 is a block diagram 200 illustrating interactions between an artificial intelligence system 106, a language model 202, and a client device 204. In some situations, the language model 202 and the client device 204 may be the same as or similar to the language model 170 and the client device 106, respectively, of FIG. 1. While a single language model 202 is shown in FIG. 2, the language model 202 may be a set of different language models that can be invoked for different tasks, with the different language models being specially trained for the tasks. For example, one language model in the set of language models may be specially trained to perform a content summarization task, while other models may be specially trained to generate highly factual output, e.g., using the summary output of the specially trained summarizing language model. Additionally, the set of models may include generalized language models that are larger in size and capable of generating large, diverse datasets, but the generalized models may have higher latency than specialized models and may be less suitable for use in real-time operation depending on the time latency constraints required to generate the content.

[0063] The artificial intelligence system 160 includes a data collection unit 206, a summary unit 208, a prompt unit 210, and a post-processing unit 212. Although the following description refers to these different units being implemented independently and each configured to perform a set of operations, any of these units can be combined to perform the operations described below.

[0064] The artificial intelligence system 160 is in communication with a memory structure 214. The memory structure 214 may include one or more databases. As shown, the memory structure includes a collection data database 216, a clause database 218, and a digital component database 220. Each of these databases 216, 218, and 220 may be implemented in the same hardware memory device, separate hardware memory devices, and / or may be implemented in a distributed cloud computing environment.

[0065] The data collector 206 is implemented using at least one computing device (e.g., one or more processors) and may include one or more language models. The data collector 206 is configured to collect information from online data sources. In some implementations, the collected information includes passages collected from a set of online resources. To obtain the passages, the data collector 206 may issue / submit queries to a search system, which responds to the queries with information about topics and / or entities. In some implementations, the collected data obtained by the data collector 206 may include search result snippets returned by the search system in response to submitting a query to the data collector 206.

[0066] In some implementations, the data collector 206 can be configured to rewrite / expand the query, for example, using the language model 202 or an internal language model to submit the rewritten query to a search system. Performing this additional query rewriting and search process increases the diversity of information collected, which is then summarized and used to generate clauses that are ultimately used to create the candidate digital component. Increasing the diversity of information collected and ultimately used to generate clauses can enhance the creative properties of the candidate digital component by providing more output options for the language model, while still adhering to specified constraints.

[0067] If an entity (e.g., a company) has an online presence, e.g., a website, that provides information about the entity, the query submitted by the data collector 206 may be a site-constrained query that causes the search system to only respond to site-constrained queries that have information contained on a specified site (e.g., the company's website). Of course, multiple site-constrained queries can be issued to multiple different sites, or multiple different sites can be specified in the site-constrained query, causing the search system to collect information related to the query from multiple different specified sites (e.g., social networking sites, web answer sites, entity review sites, etc.). The site constraint can be specified, for example, as a second-level domain or as a specific page address depending on where the information is provided.

[0068] The data collector 206 may store the collected data in a database of collected data 216. For example, the data collector 206 may index the collected data to entities characterized by the query used to collect the data and / or the collected data, so that the collected data can be retrieved from the database of collected data 216 for further operations performed by the data collector 206 and / or any operations performed by the artificial intelligence system 160.

[0069] In some implementations, the collected data can be used to train or specialize the training of one or more language models. For example, prompts related to the collected data can be submitted to a general language model, and the output of the general language model can be evaluated based on how closely the output of the general language model matches the collected data. More specifically, a quality measure can be calculated based on the difference (e.g., factual difference) between the output of the general language model and the collected data associated with the prompt. The general language model can then be iteratively adjusted to reduce the difference between the output of the general language model and the collected data, resulting in a higher quality measure and a specialized language model. The specialized language model can then be used to generate output related to the collected data (e.g., topic or content categories for a subset of the collected data).

[0070] The summarizer 208 is implemented using at least one computing device (e.g., a device including one or more processors) and may include one or more language models. The summarizer 208 is configured to summarize information about a topic or entity (e.g., a person, place, thing, or concept). In some implementations, the summarizer 208 is configured to summarize data collected by the data collector 206, potentially stored in the database of collected data 216. For example, the summarizer 208 can be configured to accept the collected data as input and output a summary of a specified length (e.g., 200 words or some other number of words) of the content of the collected data.

[0071] The summary may be generated using a language model 202, which may be part of a summarizer 208, or in data communication with the summarizer 208. In some implementations, the summarizer 208 (or a prompter 210, described below) may generate a summary prompt that is submitted to the language model 202 as an input prompt 222. The summary prompt may specify one or more of the following: a set of sources to be used to generate the summary; -Details about the set of sources that the language model should consider when summarizing content, and - a factual instruction specifying that the language model should provide citations to the sources used to generate the summary; -summarization constraints that specify information that should not be included in the summary (e.g., information that is not directly supported by a set of sources); - formatting constraints that specify how the output of the language model should be formatted (e.g., bulleted or paragraph format, with or without an introductory summary, total length (e.g., character count or separate sections)); - Tone constraints that specify the tone of the output (e.g., creative, funny, sad, serious, or from the perspective of a specified entity such as an artist, engineer, or story writer).

[0072] An exemplary summary prompt may take the following form: "Given a question and a list of sources, prepare a short summary that cites each source and summarizes them all as comprehensively as possible. Each source is independent and may repeat or contradict content from other sources. The summary should be directly supported by a given source and properly cited using the [$i] notation after statements that are supported by $i. If a statement is based on multiple sources, all of these sources should be listed in parentheses (e.g., [$i,$j,$k]). The summary may begin with a general statement about the answer space. The summary should not include any information that is not supported by the given sources."

[0073] In this example, the notation $i can be a placeholder for the name of a source. The bolded "list of sources" can be replaced with the name of the actual source to be considered, or it can be a reference to the location of the source within a set of sources to consider when creating the summary. The set of sources can be network addresses (e.g., Universal Resource Indicator / Locator—URI / URL) of online data sources (e.g., second-level domains of websites, specific addresses of web pages, or network addresses of other data sources). In some implementations, the set of sources can include a database of collected data 216, thereby enabling summaries to be generated using data collected and stored by the data collector 206. The summarizer 208 uses the summary prompt to generate a summary that summarizes passages collected from one or more network locations of the set of sources into a passage summary. As noted above, the passage summary can be formatted as a set of bullet points or in paragraph format.

[0074] The summary is generated by a language model 202, which may be part of the summarizer 208, or in data communication with the summarizer 208, as described above. In either case, the summarizer 208 inputs a summary prompt to the language model 202 as an input prompt 222. The language model 202 (e.g., an LLM) processes the input prompt 222 and generates a natural language output ("NL output") 224 that summarizes the content of the set of sources according to the instructions / constraints specified in the summary prompt.

[0075] An example paragraph summary of a set of sources providing information about automobiles could take the following form: The 2.5-liter four-cylinder engine delivers enhanced horsepower and torque distribution for seamless acceleration. Cutting-edge all-wheel-drive technology provides confidence-inspiring driving in nearly any condition with six available drive modes (Eco, Normal, Tarmac, Gravel, Snow, and Mud). Active Spot Assist can help drivers avoid collisions by detecting other vehicles in surrounding lanes, even when they are invisible to the human eye.

[0076] An exemplary bullet summary of the same set of sources could take the following form: -The 2.5-liter four-cylinder engine delivers enhanced horsepower and torque distribution for seamless acceleration. -State-of-the-art all-wheel drive technology with six available drive modes (Eco, Normal, Pavement, Gravel, Snow and Mud) ensures confident driving in almost any condition. -Active Lined Spot Assist can help drivers avoid collisions by detecting other vehicles in surrounding lanes, even when they are invisible to the human eye.

[0077] The summary may be generated in response to receiving the query 226 from the client device 204 (e.g., in real-time or online mode), or may be generated in offline mode (e.g., independent of receiving an instance of the query 226 from the client device 204). In real-time / online mode, the query 226 received from the client device 204 may be passed to the summarizer 208 in parallel with other processing being performed on the query, such as obtaining search results or otherwise generating a language model response to the query. As a result, the summarizer 208 may generate the summary while other query processing operations are taking place, thereby reducing the latency associated with providing the client device 204 with a final response to the query 226.

[0078] In offline mode, the summarizer 208 can generate summaries for anticipated queries (e.g., a large number of queries) that can be stored in a memory structure for use when queries 226 are received from the client device 204. For example, the summarizer 208 can identify a set of highest-ranked queries where the set of highest-ranked queries is related to the frequency of the queries or related to other metrics, such as time constraints for response to the queries. In this example, the summarizer 208 can identify, for each query, a set of sources related to the query and / or stored collected data related to the query (e.g., data collected using the query or similar queries), perform operations similar to those described above, and generate a set of summaries (e.g., one or more summaries) for the query. This set of summaries can be stored in memory structure 214 (e.g., with an index to the query), and when client device 204 submits query 226, summarization unit 208 (or other device in AI system 160) can query memory structure 214 to obtain one or more of the summaries indexed to query 226 in order to facilitate operations performed using the summaries, as described in more detail below. Generating summaries in an offline mode can reduce the latency associated with responding to queries 226 submitted by client device 204, as the operations required to generate the summaries do not preclude downstream operations (e.g., prompt generation), described below, that depend on the summaries.

[0079] In some implementations, the summarizer 208 may perform one or more post-processing operations on the summary output by the language model. The post-processing operations may evaluate the summary based on one or more of its factuality, relevance, comprehensiveness, conciseness, tone, and clarity, as described in more detail below with respect to the post-processor 212. Of course, the summary post-processing operations may, in some implementations, be performed by the post-processor 212 itself.

[0080] The summary is provided to a prompt unit 210, which is implemented using at least one computing device (e.g., a device including one or more processors) and may include one or more language models. The prompt unit 210 is configured to generate a prompt that includes a query 226 and a set of constraints.

[0081] The query 226 can be received from the client device 204, for example. The query 226 can be entered through a search service, a chat interface, a gaming interface, a digital assistant interface, or other interface to a service provided either online or via a native application installed on the client device. The query 226 can be as simple as a single token or a series of tokens comprising a multi-token phrase. In this scenario, the query 226 can be received by the AI ​​system 160 and inserted into a prompt by the prompt device 210. Additionally or alternatively, the AI ​​system 160 can use the query 226 to search for or otherwise obtain information related to the query 226. For example, the AI ​​system 160 can use the query 226 to identify relevant information in the database 216 of stored collected data, to collect data related to the query 226 from various online locations as described above with respect to the data collector 206, or otherwise use the query 226 to generate or identify information that can provide additional context for the creation of the prompt (e.g., to collect weather information related to the query).

[0082] The set of constraints may include passage summaries of specified sources of online content that summarize passages of information collected from a set of online resources (e.g., as described above with respect to data collector 206). For example, prompter 210 may insert one or more of the summaries generated by summarizer 208 into the prompt. In some implementations, the passage summaries inserted into the prompt act as context constraints that restrict the content created by language model 202 in response to a prompt that includes the summaries. For example, the summaries may restrict the content created by the language model to subject matter specified by the summaries included in the prompt as context constraints, as described in more detail below.

[0083] When constructing a prompt, the prompt device 210 can insert a specified entity name into the prompt. The entity name can be pre-specified or derived. In some implementations, the entity name is pre-specified based on the entity for which the content is being created (e.g., a content distributor). For example, assume that a request to generate content is submitted to the AI ​​system 160 by Example_Entity_1 (“ET1”). In this example, the prompt device 210 can insert “ET1” into the prompt, which is submitted to the language model 202 as an input prompt 222 and informs the language model 202 of the entity referenced in the NL output 224 of the language model 202. For example, similar to the description of the example of the add prompt 172 in FIG. 1 , in this example, the prompt device 210 can insert the entity name “ET1” as an entity constraint of the input prompt 222 submitted to the language model 202. In this example, the entity constraint “ET1” acts as an instruction to the language model 202 that the NL output 224 should reference “ET1.” In some implementations, the prompt unit 210 can insert a specific instruction into the prompt that the NL output 224 generated by the language model 202 must include the entity name inserted in the prompt.

[0084] In some implementations, the entity name may be derived. For example, the prompt device 210 (or other device in the AI ​​system 160) can evaluate various data to determine the appropriate entity to specify in the entity constraint inserted in the prompt. The data evaluated may include, for example, the source of data specified in the summary prompt, the summary itself, the collected data stored in the collected data database 216, or other sources of information. For purposes of illustration, assume that the source used to generate the summary references ET1 more frequently than any other entity. In this example, the prompt device 210 can determine that ET1 should be explicitly mentioned by the NL output 224 of the language model based on the fact that ET1 is referenced more frequently than any other entity. In this example, the prompt device 210 derived the identity of the entity referenced in the prompt's entity constraint by analyzing the source of the information from which the summary was created, rather than being explicitly instructed to include a specific entity name in the prompt. This derived entity name can be inserted into the prompt, thereby instructing / causing the language model 202 to reference the entity in the NL output 224 (e.g., to specify the entity name textually).

[0085] The prompter 210 can be configured to insert one or more grounding constraints into the prompt. A grounding constraint instructs / causes the language model to generate output from a verifiable source (e.g., a set of accessible sources). Including a grounding constraint in the input prompt 222 may require that the content created by the language model 202 be present in a specified set of online resources or other data sources. To utilize a grounding constraint, the prompter 210 can insert an instruction into the prompt that the NL output 224 of the language model 202 only include information from summaries (e.g., information from summaries with corresponding citations) and / or is actually / currently present in the citations referenced in the summaries.

[0086] The inclusion of these grounding constraints causes the language model 202 to construct NL outputs 224 that are verifiable within a specified data source (e.g., an abstract, a website, or other data source). This can lead to more factual and / or accurate NL outputs 224, with less tendency for "hallucination," thereby improving the operation, accuracy, and / or precision of the language model 160 and the AI ​​system 160 overall. Language model hallucination refers to a situation in which a large-scale language model (LLM) generates factually incorrect or illogical text (e.g., indicating that dogs have horns) that is not supported by the source data. Due to the nature of language models, in the absence of constraints, inaccurate, ambiguous, or illogical outputs may be generated. However, by using the grounding constraints described herein, these types of erroneous or illogical outputs can be avoided, thereby making the language model's output more accurate, precise, and trustworthy. Furthermore, generating erroneous or illogical data wastes computing resources, network bandwidth, mobile client battery power, etc. Reducing the likelihood or occurrence of hallucinations (e.g., using grounding constraints) improves the operation of language models and systems relying on their outputs by, for example, reducing the distribution of non-factual information, reducing the number of network calls made to / from the language model to arrive at an appropriate answer, and reducing the use of computational power to generate erroneous information. Reducing the likelihood / occurrence of hallucinations in a model also leads to more efficient use of mobile device battery life because the mobile device does not waste processing power or battery life processing and displaying hallucinations, allowing for more user queries and more responses to be processed and displayed by the client device before arriving at a factual response.

[0087] In some implementations, the grounding constraint may specify a particular network location (e.g., a second-level domain such as example.com, or a full page address such as example.com / example_page) or a set of network locations. In these implementations, the prompt unit 210 can insert the particular network location or set of network locations into a prompt, which is provided to the language model as an input prompt 222. Including these network locations in the prompt can improve the likelihood that the NL output 224 generated by the language model is factually accurate by instructing / causing the language model 202 to generate content that resides in one or more of the specified network locations.

[0088] The prompter 210 (or other component of the AI ​​system 160) sends, communicates, communicates, or otherwise submits the constructed prompt to the language model 202. The language model 202 uses either the summary, the query, and the specified constraints to generate the NL output 224. The NL output 224 can be a set of clauses formatted according to the formatting constraints specified in the input prompt 222. For example, if the input prompt 222 includes a formatting constraint specifying a "bulleted list," the NL output 224 can have the form of a bulleted list of clauses. The number of clauses included in the NL output 224 can also be specified by the constraints included in the prompt.

[0089] In some implementations, the number of clauses generated by language model 202 and included in NL output 224 may be greater than the number of clauses used by AI system 160 to create each candidate digital component. For example, assume AI system 160 is attempting to create candidate digital components that each contain three clauses. In this example, AI system 160 may include in the prompt an instruction to have language model 202 generate at least 12 clauses.

[0090] By instructing the language model 202 to generate more clauses (e.g., sentences, bulleted phrases, etc.) than are required to generate each individual candidate digital component, the AI ​​system 160 can create multiple different digital components by submitting a single input prompt 222 and receiving / processing a single NL output 224. In this manner, the system is made more efficient than a system that requires multiple input prompts 222 and multiple single NL outputs 224 to create multiple candidate digital components. For example, by requesting 12 clauses in the single input prompt 222, the AI ​​system 160 receives 12 clauses in the single NL output 224, which it can use to create 220 different combinations of three clauses. Thus, while a minimum of 220 different candidate digital components can be created using the single NL output 224 in this example, the AI ​​system 160 will only use one set of clauses to create the candidate digital components if the NL output 224 contains only three clauses.

[0091] The AI ​​system 160 can also use other objects to create additional candidate digital components. For example, the AI ​​system 160 can use formatting options (e.g., font, font color, text emphasis, etc.) to create additional candidate digital components. The AI ​​system 160 can also use multiple different links to create different candidate digital components. For example, the AI ​​system 160 can combine the output of the language model 202 (e.g., a set of clauses) with links to a second-level domain or specific subpages within the second-level domain to create a candidate digital component. When the AI ​​system 160 has access to multiple links appropriate for a given set of clauses received from the language model 202, the AI ​​system 160 can generate multiple different candidate digital components, each containing the same given set of clauses but linking to different web pages.

[0092] For example, assume that AI system 160 identifies a link to a website's home page (e.g., example.com) as well as a product information page (e.g., example.com / product_info), and that a section of a digital component describes the product found on the product information page. In this example, AI system 160 can create one candidate digital component that links to the home page and another candidate digital component that links to the product information page, using the same section in each of the candidate digital components.

[0093] The clauses obtained from the language model 202 can be stored in a clause database 218 for further processing by the post-processor 212 .

[0094] The post-processing unit 212 of the AI ​​system 160 is implemented using at least one computing device (e.g., a device including one or more processors) and may include one or more language models. The post-processing unit 212 is configured (e.g., specially programmed with code) to perform one or more post-processing operations on the candidate digital component. In some implementations, the post-processing operations may occur after the digital component is constructed (e.g., after clauses, links, and / or other objects are attached to the candidate digital component). In some implementations, the post-processing operations may be performed before completing the construction of the candidate digital component. For example, one or more of the post-processing operations may be performed on clauses in the NL output 224 of the language model 202 before they are combined with links to create the completed candidate digital component. As used throughout this specification, performing post-processing operations on clauses before combining them into the completed candidate digital component is considered performing post-processing on the candidate digital component, unless otherwise specified.

[0095] Performing post-processing operations includes evaluating one or more characteristics of the candidate digital component. The evaluated characteristics may include, for example, the factual quality of the clause generated by the language model 202, the relevance of the clause to the query or summary included in the prompt, the level of completeness of the clause, and the tone of the clause. Post-processing operations can be performed on clauses individually, multiple clauses combined, and / or on completed candidate digital components that include other objects, such as links to online resources.

[0096] In some implementations, the post-processing unit 212 is configured to generate a grounding score for each clause output by the language model 202. The grounding score indicates the likelihood that the clause is factual. For example, a clause that exists in the current version of a specified online resource or data source can be considered more factual than a clause that does not exist in the current version of the specified online resource or data source. Because language models generate new text content using summaries, queries, and constraints, many clauses may not be found verbatim in the specified online resource. Therefore, assessment of factual quality can be performed using similarity measures and / or other language models. For example, a clause can be compared with the raw text of the specified online resource or data source to determine the semantic distance of the clause from the text of the specified online resource. Similarly, other language models can be used to determine the level of similarity between a clause and the text of an online resource or data source. The post-processing device 212 can generate a grounding score based on the similarity of the passage with the text of the specified online resource or data source, with a higher level of similarity potentially indicating higher factual quality and a higher grounding score, and a lower level of similarity potentially indicating lower factual quality and a lower grounding score.

[0097] The post-processing device 202 can be configured to filter the clauses output by the language model 202 based on their grounding scores. For example, the post-processing device 202 can remove one or more clauses from consideration for inclusion in the digital component if the grounding score of the clause or clauses does not meet a grounding threshold (e.g., a minimum specified grounding score). The grounding threshold can be specified by an administrator or designer of the AI ​​system 160 and can be a grounding score that distinguishes between clauses classified as factual and clauses classified as non-factual. Using a score to distinguish between factual and non-factual clauses eliminates the subjectivity associated with human assessments of fact and non-fact, thereby resulting in an objective assessment. When a clause is removed from consideration for inclusion in the digital component, the AI ​​system 160 can identify other available clauses or request another set of clauses from the language model 202 to consider for inclusion. For example, if clauses are first ranked based on relevance (e.g., in the NL output 224), the next highest ranked clause can replace the removed clause in creating the candidate digital component. Similarly, if a section is deleted from a completed candidate digital component, another section can be selected to replace the deleted section.

[0098] The post-processing unit 212 can be configured to evaluate the relevance of each of the clauses to the query or summary included in the prompt. In some implementations, the relevance of a clause to the query or summary can be performed by embedding the clause, query, and / or summary in a multidimensional semantic space and determining the cosine distance between the embeddings. The relevance of a clause to the query or summary can also be determined by inputting the clause, query, and / or summary to a machine learning model (e.g., a neural network) trained to determine semantic relevance between sets of text. The post-processing unit 212 can generate a relevance score for each clause based on the analysis (e.g., a higher score indicates a higher level of relevance) and rank the clauses based on their relevance.

[0099] The post-processing device 212 can be configured to evaluate the level of completeness of each of the sections. The level of completeness of a section or set of sections defines how comprehensively the section or set of sections describes one or more topics. In some implementations, the level of completeness defines how comprehensively the section or set of sections describes a topic in a second-level domain used to generate the summary. Sections may be evaluated and ranked before creating the candidate digital component and / or may be evaluated together as a group within the candidate digital component. In some implementations, the evaluation defines how comprehensively the set of sections of a candidate digital component describes a topic in a second-level domain (or of a particular page) linked by a given candidate digital component. The level of completeness may be higher if the set of sections (e.g., three sections) of a candidate digital component describes the topic of the second-level domain more completely (e.g., provides more detail), or lower if the set of sections describes the topic less thoroughly. The post-processing unit 212 may generate an integrity score for each set of clauses and / or each candidate digital component, and may rank the sets of clauses / candidate digital components by integrity score.

[0100] The post-processing device 212 can be configured to evaluate the tone of each clause or set of clauses. The tone of a clause indicates whether the clause characterizes an item positively or negatively. In some implementations, the level of positivity or negativity can be used to generate a tone score, e.g., a clause with a positive tone has a higher tone score (e.g., a positive score) than clauses with a neutral and negative tone, and a clause with a negative tone has a lower tone score (e.g., a negative score) than clauses with a neutral and positive tone. A clause with a neutral tone can be assigned a score of zero, for example, so that it does not contribute positively or negatively to the overall tone of the candidate digital component.

[0101] The tone of a passage can be generated, for example, by submitting the passage to a language model and querying the language model whether the tone is positive, neutral, or negative. Additionally, or alternatively, the passage can be input to a machine learning model that has been trained (e.g., using labeled data) to classify passages as positive, neutral, or negative in tone. The classification of the passages can be used to assign a tone score to each passage, and the overall tone of the candidate digital component can be determined by aggregating (e.g., summing) the tone scores of the individual passages. The post-processing unit 212 can rank the passages / candidate digital components based on the tone scores.

[0102] The post-processing device 212 can be configured to rank the candidate digital components based on one or more of the post-processing operations. For example, the post-processing device 212 can rank each of the candidate digital components based on any of the scores / ratings described above or a combination of the scores / ratings described above. For example, the post-processing device can sum or average multiple different scores to obtain a total score for a clause, a set of clauses, or a candidate digital component. In some implementations, the scores can be weighted based on the relative importance of each rating to obtain a total score (e.g., a weighted average). Using the total scores, the post-processing device 212 can rank the clauses, a set of clauses, or a candidate digital component, and one or more highest-ranked candidate digital components can be identified as one or more output digital components (“output DCs”) 228 to be provided to the client device by the AI ​​system 160. In some implementations, the creation and provision of the output digital components 228 is performed after receiving the query 226. In some implementations, the output digital components 228 may be generated in an offline process (e.g., before receiving the query 226) and stored in the digital component database 220 until receiving the query 226. At that point, one or more output digital components 228 may be retrieved from the digital component database 220 and provided to the client device 204.

[0103] 3 is a flowchart of an exemplary process 300 for creating and providing a digital component using artificial intelligence. The operations of process 300 may be performed by, for example, service device 110 of FIG. 1 (e.g., including AI system 160 and / or language model 170) or other data processing device. The operations of process 300 may also be implemented as instructions stored on a computer-readable medium, which may be non-transitory. Execution of the instructions by one or more data processing devices causes the one or more data processing devices to perform the operations of process 300.

[0104] Passages are collected from a set of online resources (302). As noted above, passages can be collected in several ways. For example, passages can be collected using a site-constrained query that requires the passages to be collected from one or more network locations specified in a site constraint. In a specific example, a site-constrained query can include a site constraint (e.g., a query parameter) that specifies that a content search using the query should be limited to locations within a second-level domain (e.g., example.com) specified in the site constraint, or to a particular page (e.g., an item detail page) within the second-level domain.

[0105] The passages can be collected in an offline process (e.g., independent of and / or before a query is received from a client device) or in an online process that is performed when processing a query received from a client device. Additional details on passage collection are provided above with respect to data collector 206.

[0106] The collected passages are summarized during passage summarization (304). As described above, summarization of passages collected from one or more network locations can be performed by a language model trained to summarize multiple passages of text, such as a large-scale language model. In some implementations, the passages are collected from a list of sources specified in a summary prompt generated by an artificial intelligence system and submitted to the large-scale language model, as described above.

[0107] Summary prompts submitted to the large-scale language model can specify one or more of the following: a set of sources to be used to generate the summary; -Details about the set of sources that the language model should consider when summarizing content, and - a factual instruction specifying that the language model should provide citations to the sources used to generate the summary; -summarization constraints that specify information that should not be included in the summary (e.g., information that is not directly supported by a set of sources); - formatting constraints that specify how the output of the language model should be formatted (e.g., bulleted or paragraph format, with or without an introductory summary, total length (e.g., number of characters or separate sections)); - Tone constraints that specify the tone of the output (e.g., creative, funny, sad, serious, or from the perspective of a specified entity such as an artist, engineer, or story writer).

[0108] Exemplary summary prompts and additional details regarding the generation of summaries are provided above with respect to the summarizer 208.

[0109] A prompt (e.g., an additional prompt) is generated / constructed (306) that includes the query and a set of constraints that restrict the clauses generated by the language model. The prompt that includes the query and the set of constraints differs from the summary prompt, and as described above, the set of constraints can include the summary generated using the summary prompt as a summary constraint. For example, generating the prompt can include inserting at least a portion of the passage summary into the prompt as a context constraint that restricts the content created by the language model to the subject matter specified in the context constraint.

[0110] Additionally, generating the prompt may include inserting entity names of entities referenced by one or more network locations into the prompt as entity constraints. The entity constraint specifies that content identifying the entity must be included in the content created by the language model. In other words, the entity constraint instructs / causes the language model to include content identifying the entity in the content (e.g., clause) created by the language model.

[0111] Generating the prompt may include inserting a grounding constraint into the prompt that requires that the content created by the language model reside in a specified set of online resources. For example, inserting a grounding constraint into the prompt may be accomplished by inserting a second-level domain (e.g., example.com) into the prompt, which requires that the content created by the language model reside in resources within the second-level domain. Of course, the network locations of other data sources (e.g., databases such as database of collected data 216, individual web pages, etc.) may also be inserted into the prompt as grounding constraints. Generating the prompt is described in more detail above with respect to prompt device 210.

[0112] A plurality of candidate digital components are generated / constructed (308) using the clauses generated by the language model. In some implementations, the clauses are generated using summaries of specified sources of online content. A candidate digital component may be a single clause obtained from the language model or a combination of clauses obtained from the language model. The candidate digital component may also include other objects / items, such as links to online resources, scripts that enable various user interactions with the digital component (e.g., creating a reservation, launching a game, launching an augmented reality environment, etc.). For example, one or more of the candidate digital components may be generated by combining the output of the language model (e.g., one or more clauses) with a link to a second-level domain (e.g., the home page of example.com) and / or a link to a specific page within the second-level domain (e.g., an item information page for the item described in the clause).

[0113] In some implementations, each different candidate digital component includes a different combination of clauses received from the language model. For example, as described above, if the output of the language model (e.g., generated using the prompt from act 306) includes 12 different clauses, and each of the digital components generated by the AI ​​system is formatted to include space for three different clauses, the AI ​​system can use three different clauses in each of the candidate digital components to create 220 different candidate digital components (e.g., 12! / (3!(12-3)!)=220). Of course, adding other sets of objects / items to the digital components would further increase the number of possible combinations.

[0114] One or more post-processing operations are performed (310). In some implementations, the one or more post-processing operations include evaluating one or more characteristics of each given candidate digital component among a plurality of different candidate digital components. As noted above, a candidate digital component may be a single clause output from a language model, a combination of clauses, and / or other objects combined with one or more of the clauses output from a language model. Thus, post-processing operations may be performed on any of these candidate digital components, including individual clauses.

[0115] The execution of one or more post-processing operations can be accomplished by evaluating how factually based the candidate digital component is. In some implementations, the evaluation of how factually based the candidate digital component is can be evaluated based on whether the information in the candidate digital component can be verified in one or more specified data sources.

[0116] For example, assume that a digital component describes an object using multiple clauses generated by a language model. In this example, information about the object is collected from a set of online resources as described above with reference to operation 302, summarized as described above with reference to operation 304, and used by the language model to generate clauses that are output by the language model. The clauses that are output from the language model may differ from the passages collected in operation 302, for example, to present information from the passages in a more creative manner. Thus, although the clauses may not be found verbatim in the set of online resources, the clauses may still be analyzed to determine whether the information conveyed by the clauses is consistent with the information conveyed by the original passages, as described in more detail above with reference to the post-processing unit 212.

[0117] In some implementations, the assessment of how factual a candidate digital component (e.g., a single clause or combination of clauses) is can be performed using a grounding score, as described above with respect to the post-processing unit 212. For example, for each clause in the output of the language model, a grounding score specifying the likelihood that the clause is factual can be generated based on the level of similarity / difference between the clause and the content of a specified online resource or data source.

[0118] Using the grounding score, one or more passages can be filtered (e.g., removed from consideration for a candidate digital component). For example, one or more passages with a grounding score that does not meet a grounding threshold can be excluded from consideration. The grounding threshold is specified to distinguish between passages classified as factual and non-factual. Using a specified grounding threshold (e.g., a minimum score) based on the semantic distance (e.g., cosine distance) between the passage and the reference content (e.g., in a specified online resource) removes the subjectivity of whether information is factual or non-factual, resulting in an objective classification system. Once one or more passages are removed for not meeting the grounding threshold, those one or more passages can be removed and replaced with other passages of the output having a grounding score that meets the grounding threshold, or other passages can be evaluated for inclusion in the set of passages considered for inclusion in the digital component.

[0119] As described above with reference to the post-processing unit 212, post-processing operations may include evaluation of other characteristics of the candidate digital components. For example, each given candidate digital component among the plurality of candidate digital components may be evaluated with respect to its relevance, completeness, and tone, among others. Evaluating relevance may include evaluating the relevance of passages in the given candidate digital component to one or more of the prompt's query, the prompt's summary, a search result snippet generated using the prompt's query, or the content of the set of online resources from which the passage was collected (or other specified online data source).

[0120] The assessment of the level of completeness specifies how comprehensively the sections of a given candidate digital component describe one or more topics. As mentioned above, the one or more topics may be topics found in a second-level domain linked by a given candidate digital component or used to generate a summary. If a set of sections (e.g., three sections) of a candidate digital component more completely describes the topic of the second-level domain (e.g., provides more detail found in the second-level domain), the level of completeness may be higher, and if the set of sections describes the topic less fully, the level of completeness may be lower. For example, an artificial intelligence agent / machine learning system can compare the semantic space (e.g., a multidimensional semantic space) covered by the content of the second-level domain with the semantic space covered by the set of sections. The difference (e.g., a mathematical difference or ratio) between the covered semantic spaces can be used to obtain a completeness score for the set of sections. The difference between the semantic spaces covered by different sets of content can be determined, for example, by embedding the text of the content (e.g., in a vector representation) and determining the distance between the embeddings (or the level of overlap between the embeddings). Additionally or alternatively, different sets of content can be input into a neural network trained to determine semantic similarity.

[0121] Post-processing operations may include evaluating the tone of the passages of the candidate digital component to determine whether the passages characterize the item with a positive tone or a negative tone. In some implementations, the level of positive or negative may be used to generate a tone score, e.g., a passage with a positive tone has a higher tone score (e.g., a positive score) than passages with a neutral and negative tone, and a passage with a negative tone has a lower tone score (e.g., a negative score) than passages with a neutral and positive tone. A passage with a neutral tone may be assigned a score of zero, for example, so as not to contribute positively or negatively to the overall tone of the candidate digital component.

[0122] The tone of a passage can be generated, for example, by submitting the passage to a language model and querying the language model whether the tone is positive, neutral, or negative. Additionally, or alternatively, the passage can be input to a machine learning model that has been trained (e.g., using labeled data) to classify passages as positive, neutral, or negative in tone. The passage classification can be used to assign a tone score to each passage, and the overall tone of the candidate digital component can be determined by aggregating (e.g., summing) the tone scores of the individual passages.

[0123] Each of the plurality of candidate digital components is ranked (312). In some implementations, the plurality of candidate digital components can be ranked based on the results of the post-processing operation. The candidate digital components can be ranked based on any of the scores / ratings described above or a combination of the scores / ratings described above. For example, the post-processing device can sum or average the plurality of different scores to obtain a total score for the clause, set of clauses, or candidate digital component. In some implementations, the scores can be weighted based on the relative importance of each rating to obtain a total score (e.g., a weighted average), which can be determined by a system administrator, system architect, and / or machine learning model evaluating the feedback performance of the candidate digital component. The total score can be used to rank the clause, set of clauses, or candidate digital component (e.g., from highest score to lowest score).

[0124] At least one output digital component is provided based on the ranking (314). The at least one output digital component can be selected, for example, from among the top candidate digital components that can be classified as the output digital component. More specifically, if one output digital component is provided, the highest-ranked output digital component can be provided. If multiple output digital components are provided, a set of multiple output digital components that are in the set of highest-ranked digital components can be provided. Providing the output digital component can include transmitting instructions causing presentation of the output digital component at the client device.

[0125] 4 is a block diagram of an example computer system 400 that may be used to perform the operations described above. System 400 includes a processor 410, a memory 420, a storage device 430, and an input / output device 440. Each of the components 410, 420, 430, and 440 may be interconnected using, for example, a system bus 450. Processor 410 is capable of processing instructions for execution within system 400. In one embodiment, processor 410 is a single-threaded processor. In another embodiment, processor 410 is a multi-threaded processor. Processor 410 may process instructions stored in memory 420 or storage device 430.

[0126] Memory 420 stores information within system 400. In one embodiment, memory 420 is a computer-readable medium. In one embodiment, memory 420 is a volatile memory unit. In other embodiments, memory 420 is a non-volatile memory unit.

[0127] Storage device 430 is capable of providing mass storage for system 400. In one implementation, storage device 430 is a computer-readable medium. In various different implementations, storage device 430 may include, for example, a hard disk device, an optical disk device, a storage device shared over a network by multiple computing devices (e.g., a cloud storage device), or some other mass storage device.

[0128] The input / output device(s) 440 provide input / output operations for the system 400. In one embodiment, the input / output device(s) 440 may include one or more of a network interface device, such as an Ethernet card, a serial communication device, such as an RS-232 port, and / or a wireless interface device, such as an 802.11 card. In other embodiments, the input / output device(s) may include a driver device configured to receive input data and send output data to other devices, such as keyboards, printers, displays, and other peripheral devices 460. However, other implementations, such as mobile computing devices, mobile communication devices, set-top boxes, television client devices, etc., may also be used.

[0129] Although an example processing system is described in FIG. 4, implementations of the subject matter and functional operations described herein can be implemented in other kinds of digital electronic circuitry, or computer software, firmware, or hardware, including the structures disclosed herein and structural equivalents thereof, or one or more combinations thereof.

[0130] An electronic document (for simplicity we will simply call it a document) does not necessarily correspond to a file: a document may be stored as part of a file that holds other documents, in a single file dedicated to that document, or in multiple linked files.

[0131] In situations where the systems described herein collect and / or use personal information about a user, the user may be provided with an opportunity to enable / disable or otherwise control programs or features that may collect and / or use personal information (e.g., information about the user's social networks, social actions or activities, the user's preferences, or the user's current location). Additionally, certain data may be handled in one or more ways such that personally identifiable information associated with the user is removed before it is stored or used. For example, the user's identity may be anonymized so that personally identifiable information about the user cannot be determined, or if location information is obtained (e.g., to the city, zip code, or state level), the user's geographic location may be generalized so that the user's specific location cannot be determined.

[0132] Embodiments of the subject matter and operations described herein can be implemented in digital electronic circuitry, or computer software, firmware, or hardware, including the structures disclosed herein and their structural equivalents, or one or more combinations thereof. Embodiments of the subject matter described herein can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on a computer storage medium for execution by or to control the operation of a data processing apparatus. Alternatively, or additionally, the program instructions can be encoded in an artificially generated propagated signal, such as a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to a suitable receiving device for execution by the data processing apparatus. The computer storage medium can be, or can be included in, a computer-readable storage device, a computer-readable storage substrate, or a random or serial access memory array or device, or one or more combinations thereof. Furthermore, while a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially generated propagated signal. A computer storage medium may also be, or may be included in, one or more separate physical components or media (such as multiple CDs, disks, or other storage devices).

[0133] The operations described herein may be implemented as operations performed by a data processing apparatus on data stored in one or more computer-readable storage devices or data received from other sources.

[0134] The term "data processing apparatus" encompasses all kinds of apparatuses, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, a system on a chip, or a plurality or combination thereof. The apparatus may include special-purpose logic circuitry such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program of interest, such as code comprising processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or one or more combinations thereof. The apparatus and execution environment may implement a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

[0135] This specification refers to a service device. As used herein, a service device is one or more data processing devices that perform operations to facilitate the delivery of content over a network. A service device is shown as a single block in a block diagram. However, while a service device may be a single device or a single set of devices, the present disclosure contemplates that a service device may also be a group of devices or even multiple different systems that communicate to provide various content to client devices. For example, a service device may include one or more of a search system, a video streaming service, an audio streaming service, an email service, a navigation service, an advertising service, a gaming service, or any other service.

[0136] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted, declarative or procedural, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program, or in multiple associated files (e.g., files storing one or more modules, subprograms, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communications network.

[0137] The processes and logic flows described herein may be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows may also be performed by, and apparatus may be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[0138] Processors suitable for executing a computer program include, for example, both general-purpose and special-purpose microprocessors and any one or more processors of any kind of digital computer. Generally, a processor receives instructions and data from a read-only memory, a random-access memory, or both. The essential elements of a computer are a processor, which performs actions in accordance with the instructions, and one or more memory devices for storing instructions and data. Typically, a computer also includes one or more mass storage devices for storing data, such as magnetic, magneto-optical, or optical disks, or is operatively coupled to receive data from or transfer data to them, or both. However, a computer need not include such devices. Furthermore, a computer may be incorporated into other devices, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (such as a universal serial bus (USB) flash drive). Suitable storage devices for storing computer program instructions and data include all types of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks and removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0139] To interact with a user, embodiments of the subject matter described herein may be implemented in a computer that has a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user. For example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user may be acoustic, verbal, or tactile input. Additionally, a computer may interact with a user by sending and receiving documents to a device used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0140] Embodiments of the subject matter described herein can be implemented in a computing system that includes back-end components, e.g., a data server, or includes middleware components, e.g., an application server, or includes front-end components, e.g., a client computer having a graphical user interface or web browser through which a user can interact with an implementation of the subject matter described herein, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communications network. Examples of communications networks include local area networks ("LANs") and wide area networks ("WANs"), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0141] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server sends data (e.g., HTML pages) to client devices (e.g., for the purpose of displaying the data to and receiving user input from a user interacting with the client device). Data generated at the client device (e.g., the result of user interaction) may be received from the client device by the server.

[0142] While this specification contains many specific implementation details, these should not be construed as limiting the scope or patentable content of any invention, but rather as descriptions of features specific to particular embodiments of a particular invention. Certain features described herein in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features of the invention that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, even if features may be described above as functioning in a particular combination and originally claimed as such, one or more features from a claimed combination may, in some cases, be deleted from the combination, and the claimed combination may be directed to subcombinations or variations of the subcombination.

[0143] Similarly, while operations are shown in the figures in a particular order, this should not be understood as requiring such operations to be performed in the particular order shown or in sequential order, or that all of the illustrated operations be performed, to achieve desirable results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the above embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated into a single software product or packaged into multiple software products.

[0144] Thus, specific embodiments of the present invention have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve desirable results. Furthermore, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

1. generating, by the artificial intelligence system, a prompt including a set of constraints that constrain the query and the clauses generated by the language model, the set of constraints including a summary of a specified source of online content; generating, by the artificial intelligence system, a plurality of candidate digital components using the summary of the specified source of the online content and using the clauses generated by the language model; performing, by the artificial intelligence system, one or more post-processing operations that evaluate one or more characteristics of the plurality of candidate digital components; ranking, by the artificial intelligence system, each of the plurality of candidate digital components based on the post-processing operation; providing, by the artificial intelligence system, at least one output digital component within a set of highest-ranked candidate digital components.

2. collecting passages from a set of online resources using a site-constrained query that requires collecting the passages from one or more network locations specified by a site constraint; The method of claim 1 , further comprising: summarizing the passages collected from the one or more network locations into a passage summary.

3. 3. The method of claim 2, wherein generating the prompt includes inserting at least a portion of the passage summary into the prompt as a context constraint that limits content produced by the language model to a subject matter specified in the context constraint.

4. 4. The method of claim 3, wherein generating the prompt includes inserting into the prompt an entity name of the entity referenced by the one or more network locations as an entity constraint specifying that content identifying the entity must be included in content created by the language model.

5. 5. The method of claim 4, wherein generating the prompt includes inserting a grounding constraint into the prompt requiring that content produced by the language model be present in a specified set of online resources.

6. 6. The method of claim 5, wherein inserting a grounding constraint into the prompt comprises inserting a second-level domain into the prompt that requires that content produced by the language model reside in a resource within the second-level domain.

7. The method of claim 6 , wherein generating the plurality of candidate digital components comprises combining an output of the language model with a link to the second level domain.

8. performing one or more post-processing operations to evaluate one or more characteristics of the plurality of candidate digital components, generating a grounding score for each clause in the output of the language model that specifies the likelihood that the clause is a fact; filtering the output of the language model by removing one or more clauses having a grounding score that does not satisfy a grounding threshold that distinguishes between clauses classified as factual and clauses classified as non-factual; and replacing the one or more removed clauses with other clauses of the output having a grounding score that satisfies the grounding threshold.

9. performing one or more post-processing operations to evaluate one or more characteristics of the plurality of candidate digital components, for each given candidate digital component among said plurality of candidate digital components: assessing the relevance of the clause of the given candidate digital component to the query of the prompt; assessing a level of completeness of the section of the given candidate digital component, which specifies how comprehensively the section describes the topic of the second-level domain linked by the given candidate digital component; and evaluating the tone of the passage of the candidate digital component to determine whether the passage characterizes an item as positive or negative.

10. One or more non-transitory computer-readable media storing instructions that, when executed by an artificial intelligence system, cause the artificial intelligence system to perform operations, the operations including: generating a prompt including a set of constraints that constrain the query and the clauses generated by the language model, the set of constraints including a summary of a specified source of online content; using the summary of the specified source of the online content to generate a plurality of candidate digital components using the clauses generated by the language model; performing one or more post-processing operations to evaluate one or more characteristics of the plurality of candidate digital components; ranking each of the plurality of candidate digital components based on the post-processing operation; and providing at least one output digital component that is within the set of highest-ranked candidate digital components.

11. The instructions cause the artificial intelligence system to perform operations, the operations including: collecting passages from a set of online resources using a site-constrained query that requires collecting the passages from one or more network locations specified by a site constraint; and summarizing the passages collected from the one or more network locations into a passage summary.

12. 12. The one or more non-transitory computer-readable media of claim 11, wherein generating the prompt includes inserting at least a portion of the passage summary into the prompt as a context constraint that limits content produced by the language model to a subject matter specified in the context constraint.

13. 13. The one or more non-transitory computer-readable media of claim 12, wherein generating the prompt includes inserting into the prompt an entity name of the entity referenced by the one or more network locations as an entity constraint specifying that content identifying the entity must be included in content created by the language model.

14. 14. The one or more non-transitory computer-readable media of claim 13, wherein generating the prompt includes inserting a grounding constraint into the prompt requiring that content produced by the language model be present in a specified set of online resources.

15. 15. The one or more non-transitory computer-readable media of claim 14, wherein inserting a grounding constraint into the prompt comprises inserting a second-level domain into the prompt that requires content produced by the language model to reside in a resource within the second-level domain.

16. 16. The one or more non-transitory computer-readable media of claim 15, wherein generating the plurality of candidate digital components comprises combining an output of the language model with a link to the second level domain.

17. performing one or more post-processing operations to evaluate one or more characteristics of the plurality of candidate digital components, generating a grounding score for each clause in the output of the language model that specifies the likelihood that the clause is a fact; filtering the output of the language model by removing one or more clauses having a grounding score that does not satisfy a grounding threshold that distinguishes between clauses classified as factual and clauses classified as non-factual; and replacing the one or more removed clauses with other clauses of the output having grounding scores that satisfy the grounding threshold.

18. performing one or more post-processing operations to evaluate one or more characteristics of the plurality of candidate digital components, for each given candidate digital component among said plurality of candidate digital components: assessing the relevance of the clause of the given candidate digital component to the query of the prompt; assessing a level of completeness of the section of the given candidate digital component, which specifies how comprehensively the section describes the topic of the second-level domain linked by the given candidate digital component; and evaluating the tone of the passage of the candidate digital component to determine whether the passage characterizes an item as positive or negative.

19. 1. An artificial intelligence system, comprising: one or more memory devices; one or more computing devices configured to execute code comprising a set of instructions, execution of the set of instructions causing the one or more computing devices to perform operations, the operations comprising: generating a prompt including a set of constraints that constrain the query and the clauses generated by the language model, the set of constraints including a summary of a specified source of online content; using the summary of the specified source of the online content to generate a plurality of candidate digital components using the clauses generated by the language model; performing one or more post-processing operations to evaluate one or more characteristics of the plurality of candidate digital components; ranking each of the plurality of candidate digital components based on the post-processing operation; and providing at least one output digital component that is within the set of highest-ranked candidate digital components.

20. The instructions cause the one or more computing devices to perform operations, the operations including: collecting passages from a set of online resources using a site-constrained query that requires collecting the passages from one or more network locations specified by a site constraint; 20. The system of claim 19, further comprising: summarizing the passages collected from the one or more network locations into a passage summary.

21. 21. The system of claim 20, wherein generating the prompt includes inserting at least a portion of the passage summary into the prompt as a context constraint that limits content produced by the language model to a subject matter specified in the context constraint.

22. 22. The system of claim 21, wherein generating the prompt includes inserting into the prompt an entity name of the entity referenced by the one or more network locations as an entity constraint specifying that content identifying the entity must be included in content created by the language model.

23. 23. The system of claim 22, wherein generating the prompt includes inserting a grounding constraint into the prompt requiring that content produced by the language model be present in a specified set of online resources.

24. 24. The system of claim 23, wherein inserting a grounding constraint into the prompt includes inserting a second-level domain into the prompt that requires content produced by the language model to reside in a resource within the second-level domain.

25. 25. The system of claim 24, wherein generating the plurality of candidate digital components comprises combining an output of the language model with a link to the second level domain.

26. performing one or more post-processing operations to evaluate one or more characteristics of the plurality of candidate digital components, generating a grounding score for each clause in the output of the language model that specifies the likelihood that the clause is a fact; filtering the output of the language model by removing one or more clauses having a grounding score that does not satisfy a grounding threshold that distinguishes between clauses classified as factual and clauses classified as non-factual; and replacing the one or more removed clauses with other clauses of the output having a grounding score that satisfies the grounding threshold.

27. performing one or more post-processing operations to evaluate one or more characteristics of the plurality of candidate digital components, for each given candidate digital component among said plurality of candidate digital components: assessing the relevance of the clause of the given candidate digital component to the query of the prompt; assessing a level of completeness of the section of the given candidate digital component, which specifies how comprehensively the section describes the topic of the second-level domain linked by the given candidate digital component; and evaluating the tone of the passage of the candidate digital component to determine whether the passage characterizes an item as positive or negative.

Images