Generative artificial intelligence

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating digital components for conversations between a user and an artificial intelligence (AI) system employing a language model. The method includes: receiving an input message from the user as part of the conversation; receiving, as part of the dialog, an output generated by the AI system in response to the input message and based on a language model; determining one or more keywords based on at least one of the input message and the output; generating one or more digital components based on the one or more keywords; determining that there is a pause in the output when the AI system continues to generate an additional output; and merging the one or more digital components into the conversation at a location corresponding to the determined pause for display on the user interface.

Description

Background Art

[0001] This specification relates to data processing and blending content from different domains into a combined visual presentation. Summary of the invention

[0002] This specification describes techniques for generating digital components for a conversation between a user and an artificial intelligence (AI) system using a language model. In general, an innovative aspect of the subject matter described in this specification can be embodied in a method comprising the following actions: receiving an input message from the user as part of the conversation; receiving output generated by the AI ​​system in response to the input message and based on a language model as part of the conversation; determining one or more keywords based on at least one of the input message and the output; generating one or more digital components based on the one or more keywords; determining that there is a pause in the output as the AI ​​system continues to generate additional output; and merging the one or more digital components into the conversation at a location corresponding to the determined pause for display on a user interface.

[0003] These and other embodiments may each optionally include one or more of the following features. For example, in some implementations, generating one or more digital components based on one or more keywords further includes sending the one or more keywords to a processing system, and receiving the one or more digital components from the processing system.

[0004] In some implementations, determining one or more keywords based on at least one of an input message and an output includes parsing at least one of the input message and the output.

[0005] In some implementations, determining the one or more keywords is further based on one or more of: an input message, an output, one or more previously received input messages, one or more previously generated outputs, one or more previously received input messages from a previous conversation with the user, one or more previously generated outputs from a previous conversation with the user, one or more previously received input messages from one or more previous conversations with the user, one or more previously generated outputs from one or more previous conversations with the user, a profile of the user, personalized information associated with the user, the location of the user or the location of a client device used by the user, or one or more attributes of the client device.

[0006] In some implementations, the output includes a plurality of text items, and wherein determining the one or more keywords includes determining at least one keyword for each text item in the plurality of text items.

[0007] In some implementations, the processing system includes one or more of a search engine, a digital component server, a reservation system, an assistant system that assists users in completing tasks, or a chatbot.

[0008] In some implementations, the method further includes displaying the dialog and the one or more digital components on a user interface.

[0009] In some implementations, generating the one or more digital components based on the one or more keywords includes individually generating each of the one or more digital components based on at least a subset of the one or more keywords.

[0010] In some implementations, the method further includes selecting a subset of the one or more digital content items; and wherein merging the one or more digital components into the conversation includes merging the subset of the one or more digital content items into the conversation.

[0011] In some implementations, selecting a subset of one or more digital content items is based on a combination of one or more of: a measure of relevance of one or more of: an input message, an output, one or more previously received input messages, one or more previously generated outputs, one or more previously received input messages from a previous conversation with the user, one or more previously generated outputs from a previous conversation with the user, one or more previously received input messages from one or more previous conversations with the user, one or more previously generated outputs from one or more previous conversations with the user, a profile of the user, personalized information associated with the user, a location of the user or a location of a client device used by the user, or one or more attributes of the client device; expected user satisfaction, wherein the expected user satisfaction includes one or more of a received user satisfaction quantification of the user and a measured user satisfaction quantification of the user, wherein the measured user satisfaction quantification of the user is determined based on one or more behaviors of the user; expected short-term profitability; expected long-term profitability, wherein the expected long-term profitability is configured to take into account both short-term profitability and long-term behavioral changes.

[0012] In some implementations, the method further includes determining a user interaction related to a respective digital content item of the one or more digital content items.

[0013] In some implementations, the method further includes determining a respective score for each of the one or more digital content items based on the one or more respective user interactions, wherein selecting the subset of the one or more digital content items is based on the respective scores of the one or more digital content items.

[0014] In some implementations, the method further includes: determining that there is an interruption in the output while the AI ​​system is waiting for the next input message; determining one or more updated keywords based on the output received before the interruption, the output including one or more pauses; generating one or more updated digital components corresponding to the one or more updated keywords; and merging the one or more updated digital components into the dialogue at a location corresponding to the determined interruption for display on the user interface.

[0015] In some implementations, at least one of parsing the input message and the output is performed as a lightweight parsing.

[0016] In some implementations, determining one or more keywords based on at least one of an input message and an output and generating at least one of one or more digital components based on the one or more keywords are performed in response to one of: receiving an input message from a user as part of a conversation; and receiving an output generated by the AI ​​system as part of a conversation.

[0017] Other implementations of this and other aspects include corresponding systems, devices, and computer programs that are configured to perform the actions of these methods encoded on computer storage devices. The system of one or more computers can be so configured by means of software, firmware, hardware, or a combination thereof installed on the system that causes the system to perform actions during operation. One or more computer programs can be so configured by having instructions that, when executed by a data processing device, cause the device to perform actions.

[0018] Specific embodiments of the subject matter described in this specification may be implemented to achieve one or more of the following advantages. The techniques discussed in this specification may improve the generation of digital components for a conversation between a user and an AI system, so that the conversation may be enhanced by digital components. This may improve user engagement based on the conversation. For example, an enhanced conversation or portion of a conversation may have one or more digital components associated with the conversation or portion of the conversation. Digital components may help to further enrich the conversation and provide visualization, richness, and interaction options for the user. Digital components may be generated in an organic manner as the conversation progresses, such as corresponding to user input or output generated by an AI system. Specifically, the output generated by the AI ​​system may be provided in an intermittent manner, rather than continuously or all at once. The processing characteristics of intermittent output may be used to provide one or more corresponding digital components corresponding to a portion of the output continuously provided by the AI ​​system. Based on the techniques described in this specification, the generation of digital components may be performed in a manner that does not add additional delays and or waiting times to the normal progress of the conversation. Enhancing the conversation using digital components may be performed in the same or substantially similar time as the normal progress of the conversation.

[0019] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a block diagram of an example environment in which generating digital components for a conversation between a user and an AI system may be performed.

[0021] Figure 2 Showing an example portion of a conversation between a user and an AI system.

[0022] Figure 3 is a flow chart of an example method for generating a digital component for a conversation between a user and an AI system.

[0023] Figure 4 An example portion showing an augmented conversation between a user and an AI system.

[0024] Figure 5 It can be used to perform Figure 3 A block diagram of an example computer system 400 in which the method is described.

[0025] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0026] This specification describes techniques for generating digital components for a conversation between a user and an artificial intelligence (AI) system, such as a generative AI system or a "chatbot." The digital components may be generated by a processing system, such as a system that parses at least a portion of the conversation and generates the digital components based on the parsing, or by another AI system that generates output based on at least a portion of the conversation and generates the digital components based on the output.

[0027] AI generally refers to the theory and development of computer systems that can perform tasks that traditionally require human intelligence. For example, tasks can include visual perception, speech recognition, decision making, and natural language processing. AI systems can utilize machine learning, such as by using algorithms and statistical models to analyze and draw inferences from patterns and structures in training data, enabling the AI ​​system to generate new data with similar characteristics. Generative AI refers to AI systems that can generate text, images, or other content in response to user input or "prompts."

[0028] The technology described in this specification enables a processing system (e.g., an AI system) to generate digital components for a conversation between a user and another (e.g., generative) AI system. The processing system may include, for example, a search engine, a digital component server, a reservation system, an assistant system that assists users in completing personal or professional tasks, a chatbot (independent or embedded in another service), or an interactive system that interacts with a user through a combination of text and rich content. When a user interacts with an AI system (particularly a generative AI system), the response of the AI ​​system to the user input is generally limited to text output. In some implementations, the AI ​​system is configured to generate text output corresponding to a natural language so that the user can read and understand the content as easily as in another conversation (e.g., in a conversation with another user). For purposes such as visualization, interaction, and / or other purposes, it may be beneficial to enhance text output. For example, enhanced text output may provide users with additional instructions for the content referred to by the output of an AI system or chatbot. This is particularly useful when the user is not familiar with a language dedicated to a particular field, for example, when a user who is not particularly interested in jewelry and is not familiar with gemstone names, cuts, jewelry types, etc. wants to make recommendations about jewelry as a gift. In yet another example, enhanced text can help improve user experience and / or engagement, particularly when chatbot output is generated over time and sent / output to a user device. These and further examples are described below.

[0029] AI systems typically utilize input prompts from a language model such as a large language model (LLM), which outputs one or more clauses in the form of text output in response to user input such as a "prompt". In the context of AI, a prompt typically refers to a user input in the form of text, spoken language, or other forms that communicates a user's question or query to the AI ​​system. The specific output from the AI ​​system in response to the user input varies based on, for example, the way the user input is phrased. The processing of the AI ​​system typically exhibits processing delays and / or intermittent output when generating output, such as pauses between consecutive clauses. Continuous clauses may also be referred to as partial outputs, or in-process outputs. One or more consecutive clauses or partial outputs typically form the complete output of the AI ​​system generated in response to a prompt. In some cases, when the AI ​​system no longer generates output in response to the user input or prompt, the response to the user input or prompt may be considered a complete output.

[0030] Delays and / or pauses in the output of the AI ​​system can be used to perform processing of available information, particularly "lightweight" processing. Lightweight processing can refer to non-resource-intensive processing, such as simple parsing of text information and / or the application of simple models. The available information to be processed can include, for example, user input or prompts, one or more clauses or partial outputs, and / or complete outputs, as well as previously received prompts and / or outputs (e.g., in the form of context or "state" of a conversation). For example, processing can be performed to enhance, strengthen, annotate or otherwise process prompts, clauses, partial outputs, complete outputs and / or contexts. The processing can be configured to be performed partially or entirely during the generation of the complete output by the AI ​​system. This may have the effect of providing or displaying the (enhanced) complete output with very little or no delay.

[0031] The processing may be performed by an AI system that utilizes one or more clauses or partial inputs in conjunction with a corresponding model configured to provide an output based on which one or more clauses or partial inputs or the complete output may be annotated. Such a corresponding model may be a lightweight or simple model, a specialized model, or a general model.

[0032] While the AI ​​systems and / or underlying models used in the (lightweight) processing may be stateless, they may alternatively be context-sensitive to the entire conversation (e.g., enabling processing of "stateful" conversations). For example, the processing may take into account the current context of the conversation, where the processing is based at least in part on information about the current state of the conversation. This may include taking into account the textual content of one or more previously prompted and / or the textual content of one or more previously outputted for processing one or more clauses or partial outputs. The output of the AI ​​system performing the (lightweight) processing may include one or more keywords corresponding to the one or more clauses or partial outputs. Context-sensitive processing of the (stateful) conversation may, for example, improve the annotation of the conversation in terms of accuracy and / or completeness.

[0033] The (lightweight) processing may additionally or alternatively be based on parsing one or more clauses or partial outputs, e.g., to determine one or more keywords corresponding to the one or more clauses or partial outputs. Parsing may be optimized for lightweight processing of one or more clauses or partial outputs, either individually or taking into account the overall context of the conversation (a "stateful" conversation). Context-sensitive parsing may further improve the annotation of the conversation, e.g., in terms of accuracy and / or completeness.

[0034] In some implementations, to enhance the output, the AI ​​system collects information from various sources, such as web pages or other online resources, and combines the collected information in different ways to create different candidate digital components. The AI ​​system uses one or more clauses or partial outputs to generate one or more digital components corresponding to the one or more clauses or partial outputs. In some examples, generating one or more digital components corresponding to the one or more clauses or partial outputs may include determining one or more keywords, and determining the one or more digital components based on the determined one or more keywords.

[0035] The AI ​​system may further perform post-processing to select one or more digital components to be used to annotate the conversation from among the one or more digital components that may be considered candidate digital components. The one or more digital components may be provided for output, such as for display on a user interface.

[0036] In some implementations, the (generative) AI system is configured to generate text output corresponding to natural language in response to a user's prompt, so that the user can easily read and understand the content as in another conversation (e.g., in a conversation with another user). The text output is then enhanced for purposes such as visualization and / or interaction.

[0037] In some implementations, visualization includes retrieving digital components related to the conversation, including digital content or digital information (e.g., video clips, audio clips, multimedia clips, game content, images, text, highlights, artificial intelligence output, language model output, or another content unit), to introduce one or more of the retrieved digital components into the conversation for display. The digital content / information related to the conversation may include, for example, one or more of a list of paid or free products / services, such as items that a user may be interested in purchasing, places that a user may be interested in booking, services that a user may be interested in purchasing, or other products and services, particularly products and services that a provider of a product or service is interested in promoting. One or more digital components may be displayed in combination with or near user input and / or AI system clauses or outputs, or displayed as an integrated part of user input and / or AI system clauses or outputs. Displaying one or more digital components includes displaying corresponding digital content or digital information and further associated elements, such as one or more narrations, hyperlinks, and / or icons.

[0038] In some implementations, the interaction includes providing an interactive digital component with which the user can interact. The interactive digital component may include digital content or digital information associated with one or more actions, including accessing a hyperlink, launching an app or application, performing a function associated with the user device, or performing another action.

[0039] In some implementations, a (generative) AI system is configured to intermittently generate text output over a period of time, wherein the AI ​​system generates an initial output for display, and during the period of time, continuously generates additional output for display until it is determined that the text output in response to the user input is complete. For example, the additional output may be appended to previously generated output. Typically, the AI ​​system does not process additional user input (e.g., one or more prompts) during the period of time when the output is continuously or intermittently generated and displayed. In some implementations, the AI ​​system is configured to buffer additional user input received while generating output in response to previous user input.

[0040] As used throughout this document, the phrase "digital component" refers to a discrete unit of digital content or digital information (e.g., a video clip, an audio clip, a multimedia clip, game content, an image, text, a gist, an artificial intelligence output, a language model output, or another unit of content). A digital component may be electronically stored in a physical memory device as a single file or in a collection of files, and a digital component may take the form of a video file, an audio file, a multimedia file, an image file, or a text file and include advertising information, such that an advertisement is a type of digital component.

[0041] Figure 1 1 is a block diagram of an example environment 100 in which generation of a digital component for a conversation 200 between a user and an AI system 168 may be performed. The example environment 100 includes a network 102, such as a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof. The network 102 connects an electronic document server 104, a user device 106, a digital component server 108, and a service device 110. The example environment 100 may include many different electronic document servers 104, user devices 106, and digital component servers 108.

[0042] Client devices 106 are electronic devices capable of requesting and receiving online resources over network 102. Example client devices 106 include personal computers, gaming devices, mobile communication devices, digital assistant devices, augmented reality devices, virtual reality devices, and other devices that can send and receive data over network 102. Client devices 106 typically include user applications (such as web browsers) to facilitate sending and receiving data over network 102, but native applications (other than browsers) executed by client devices 106 may also facilitate sending and receiving data over network 102.

[0043] A game device is a device that enables a user to participate in a game application, for example, in which a user can control one or more characters, avatars or other rendered content presented in a game application. A game device typically includes a computer processor, a memory device and a controller interface (physically or visually rendered) that enables a user to control the content rendered by the game application. The game device can store and execute the game application locally, or execute a game application (e.g., an online game application) that is at least partially stored and / or served by a cloud server. Similarly, the game device can be docked with a game server that executes the game application and "streams" the game application to the game device. The game device can be a tablet device, a mobile telecommunications device, a computer, or another device that performs other functions in addition to executing the game application.

[0044] The digital assistant device includes a device with a microphone and a speaker. The digital assistant device is generally capable of receiving input by voice, and responding to the content using audible feedback, and can present other audible information. In some cases, the digital assistant device also includes a visual display or communicates with a visual display (e.g., by wireless or wired connection). When there is a visual display, feedback or other information can also be provided visually. In some cases, the digital assistant device can also control other devices, such as lights, locks, cameras, climate control devices, alarm systems, and other devices registered with the digital assistant device.

[0045] As shown, client device 106 communicates data with AI system 168 via network 102. AI system 168 may be a generative AI system. Client device 106 is configured to receive prompts or user inputs, such as text inputs via a user interface, and send prompts or user inputs 162 to AI system 168 (see, for example, request 112 sent from client device 106 to AI system 168 via network 102). The request includes user input (or prompt) 162.

[0046] AI system 168 is configured to receive user input 162 and generate output 164 based on received user input 162. Output 164 is transmitted to client device 106 via network 102 (see, e.g., replies 120 sent from AI system 168 to client device 106 via network 102). Replies 120 include one or more sub-clauses or partial outputs (e.g., portions of a complete output) or the complete output generated by AI system 168. Replies 120 are transmitted to client device 106 via network 102. For example, one or more replies 120 may include one or more sub-clauses or partial outputs, or the complete output.

[0047] The AI ​​system 168 is configured to provide a prompt or user input 162 to a language model such as an LLM, which provides an output 164 in response to the prompt or user input 162. The output may be a text output that includes, for example, one or more clauses. The AI ​​system 168 processes the prompt 162 and generates an output 164. Generating the output may introduce processing delays because the output 164 is provided intermittently in the form of one or more clauses or partial outputs, rather than continuously (e.g., as a continuous stream of data) or instantaneously (e.g., as a block of data). The output 164 may include pauses between consecutive clauses or partial outputs when the AI ​​system generates one or more additional clauses. One or more consecutive clauses or partial outputs generally constitute the complete output of the AI ​​system to the prompt or user input. The output is considered "complete" because it represents the response of the AI ​​system to the prompt (e.g., the AI ​​system no longer generates additional clauses to respond to the prompt). Once the AI ​​system provides the complete output, the AI ​​system is generally configured to wait for the next prompt.

[0048] Figure 2 An example portion 202 of a conversation 200 between a user and an AI system 168 is shown. The user provides a prompt 162 to the AI ​​system 168, and the AI ​​system 168 provides an output 164 in response. As described above, the output 164 may be provided in a discrete manner as a complete output 164, or intermittently in the form of several consecutively provided clauses or partial outputs 164-1, 164-2, and 164-3, such as with a pause after each partial output. In a similar manner, the itemized list of the partial output 164-2 may include consecutively provided clauses, each clause corresponding to an item in the itemized list.

[0049] For example Figure 1 The processing system of AI system 166 shown in can be configured to perform processing during the time period when AI system 168 generates output in response to the prompt. The processing system can be configured to incorporate the digital components generated by the processing system into the conversation. AI system 166 can be referred to as a "processing AI system" and AI system 168 can be referred to as a "generative AI system."

[0050] Figure 3 is a flow chart of an example method 300 for generating a digital component for a conversation between a user and an AI system. In some implementations, the method is performed by a processing system integrated into or running on a client device 106 and in communication with the AI ​​system 168.

[0051] The processing system may include Figure 1In other implementations, the method is performed by a processing system that operates independently of the client device 106. The processing communicates with both the client device 106 and the AI ​​system 168. The processing system may be Figure 1 The service device 110 shown implements the method 300. In the following, without prejudice, the method 300 is described as being performed by a processing system integrated into the client device 106 and employing the AI ​​system 166.

[0052] At step 302, the AI ​​system 166 receives an input message (e.g., prompt 162) from a user. In some implementations, the input message is input by the user as part of a conversation 200 between the user and the AI ​​system 168 using a user interface of the client device 106. The user input may include, for example, text input from the user. The user may use any form of input supported by the client device 106, such as keyboard input, voice input, gesture input, and other forms. The AI ​​system 166 or the client device 106 may be configured to send the input message 162 to the AI ​​system 168 for processing.

[0053] At step 304, as part of dialog 200, AI system 166 receives output 164 generated by AI system 168 in response to input message 162. In some implementations, AI system 168 utilizes LLM to generate text output in response to input message 162 in the form of one or more clauses.

[0054] If the LLM continuously generates a series of intermittent clauses or partial outputs, e.g., includes pauses between consecutive clauses or partial outputs, then AI system 166 continuously receives from AI system 168 several clauses or partial outputs (e.g., Figure 2 One or more consecutive clauses or partial outputs 164 - 1 , 164 - 2 , and 164 - 3 typically form output 164 of AI system 168 in response to prompt or input message 162 .

[0055] At step 306, AI system 166 determines one or more keywords based on at least one of input message 162 received at step 302 and / or output 164 received at step 304. Output 164 may include one or more clauses 164-1, 164-2, and 164-3 generated by AI system 168.

[0056] In some implementations, AI system 166 is configured to determine one or more updated keywords in response to receiving further output 164 from AI system 168 and / or further user input 162. This enables AI system 166 to determine one or more keywords based on a large amount of information. This can improve the quality of one or more keywords. Figure 2 In the example dialogue 200 shown, if AI system 166 determines one or more keywords based only on prompt 162, AI system 166 may determine that the context of the one or more keywords is relatively limited. Similar considerations apply to the case where AI system 166 determines one or more keywords based on prompt 162 and clause 164-1. The keywords so determined may be relatively more general and less closely related to the current dialogue.

[0057] In general, the more information or context the AI ​​system 166 can use to determine one or more keywords, the better the correspondence of the one or more keywords to the conversation. Figure 2 In the example dialogue 200 shown, if the AI ​​system 166 can determine one or more keywords based on the prompt 162 and the clauses 164-1 and 164-2, the AI ​​system 166 can determine that the context of the one or more keywords is relatively richer than the case where the AI ​​system 166 determines the one or more keywords based only on the prompt 162 and / or the clause 164-1. In some implementations, the context may include one or more of the following in addition to the prompt 162 and / or the clause 164-1: one or more previously received input messages, one or more previously generated outputs, one or more previously received input messages from a previous dialogue of the user, one or more previously generated outputs from a previous dialogue of the user, one or more previously received input messages from one or more previous dialogues of the user, one or more previously generated outputs from one or more previous dialogues of the user, a profile of the user, personalized information associated with the user, the location of the user or the location of a client device used by the user, and one or more attributes of the client device. The user profile may include one or more of the following: information shared by the user, information inferred based on the user's usage data (e.g., demographics, interests, preferences, previous commercial and non-commercial activities). Personalized information may include, inter alia, information concerning commercial and non-commercial personalization.

[0058] At step 308, the AI ​​system 166 generates one or more digital components 161 based on the one or more keywords. Each digital component 161 is determined based on the one or more keywords or based on a subset of the one or more keywords. In some implementations, the AI ​​system 166 applies the one or more keywords to a simple model to generate the one or more digital components. Additionally or alternatively, the AI ​​system may send a request 112 to the service device 110 to query the digital component database 116, such as Figure 1 Additionally or alternatively, AI system 166 may send request 112 to one or more digital component servers 108, one or more electronic document servers 104, and / or other servers via network 102, such as Figure 1 The AI ​​server 166 receives corresponding responses 120 in response to the one or more requests 112, including one or more digital components matching the one or more queries.

[0059] At step 310, the AI ​​system 166 determines that there is a pause in the output 164 received from the AI ​​system 168. For example, the pause can be determined as a semantic pause and / or a delay-based pause. The semantic pause can be identified by (simple) parsing the output. For example, a period or semicolon (or other punctuation, which can be referred to as an "output pause token") that ends a sentence can indicate a semantic pause. The same applies to, for example, line breaks (usually bullets or after bullets) or other format-related characters or codes, bullets or molecules, and other printable or unprintable character encodings. In other examples, more complex parsing can be used to identify semantic pauses that can insert digital components before continuing the output of the AI ​​system. The pause can be further determined as a delay-based pause, which exhibits a delay in output while further output is expected, such as when the AI ​​system 168 is expected to continue to produce additional output. Typically, the AI ​​system 168 generates an output 164 that includes one or more pauses, during which the AI ​​system 168 temporarily pauses to provide further output. During the pause in providing output, AI system 168 continues to process input messages 162 .

[0060] Once the AI ​​system 168 generates further output 164, the AI ​​system resumes providing further output after the pause. In some implementations, each of the one or more pauses has a duration of a fraction of a second (e.g., ranging from 0.1 seconds to less than 1.0 seconds) or a duration of one or more seconds. In some implementations, a semantic pause can trigger an output delay while the AI ​​system 168 continues to generate further output. This can help improve the overall responsiveness of the AI ​​system 168. In some implementations, each of the one or more pauses can have a different duration. In some implementations, the generated output includes an interruption, after which the AI ​​system 168 will not generate further output and wait for one or more further prompts or input messages. The AI ​​system 168 can, for example, indicate an interruption (indicating that the most recent output is complete) using an "end of output" word (e.g., a predetermined printable or non-printable character encoding).

[0061] At step 312 , AI system 166 incorporates one or more digital components 161 into dialog 200 for display on the user interface. Figure 4 An example portion of an augmented conversation 200′ between a user and an AI system 168 is shown, wherein one or more digital components 161 (see digital components 161-a and 161-b) are incorporated into the conversation 200. As shown, the AI ​​system 166 determines one or more keywords based on the clause or partial output 164-a, and generates the digital component 161-a based on the one or more keywords corresponding to the clause or partial output 164-a. The AI ​​system 166 incorporates the generated one or more digital components 161-a into the conversation 200 (at Figure 4 The enhanced dialog 200' is referred to as enhanced dialog 200' for display on the user interface.

[0062] The generated digital component 161-a is associated with the clause or partial output 164-a and is incorporated into the dialogue 200' in close proximity to the clause or partial output 164-a, such that at least a portion of the dialogue 200' is enhanced relative to the clause or partial output 164-b. In a similar manner, the generated digital component 161-b is associated with the clause or partial output 164-b and is incorporated into the dialogue 200' in close proximity to the clause or partial output 164-b, such that at least a portion of the dialogue 200' is enhanced relative to the clause or partial output 164-b. In some implementations, incorporating one or more digital components 161 (see, e.g., digital components 161-a and 161-b) into the dialog 200 immediately adjacent to the corresponding clause or partial outputs 164-a and 164-b includes incorporating one or more digital components 161 into the dialog 200 immediately before or after the corresponding clause or partial outputs 164-a and 164-b, or directly adjacent to the corresponding clause or partial outputs 164-a and 164-b. Figure 4 In the example shown, the number components 161 - a and 161 - b have been incorporated into the dialog 200 ′ directly after the clause or portion output 164 - a and 164 - b , respectively, to be displayed immediately below the clause or portion output 164 - a and 164 - b .

[0063] It may be beneficial to enhance text output for purposes such as visualization and / or interactivity. Figure 4 In the example shown, the digital component 161-a visualizes the clause or partial output 164-a and facilitates further interaction based on the digital component 161-a. In some implementations, the digital component includes a snippet, a photo, a hyperlink, etc. For example, the digital component 161 may include one or more of the following bounded by a box and linked to a destination: text may include a title, a representation of the URL to which the link will go, one or more of a description, and other annotations, images, videos, prices, promotional indications valid for the content mentioned, and other information to help the user understand what the digital component refers to and why the user may be interested. The digital component 161 may refer to, for example, (1) a product, such as a piece of jewelry (see, e.g. Figure 4 elements 161-a), a specific vehicle, an item of clothing, an item of household goods, a food or drink, or any other product, (2) a product category, such as a type of jewelry (e.g., “earring studs,” “diamond earring studs”), a type of vehicle (e.g., “n-row SUV, sedan, compact,” model, model and year, model and year, and trim), a type of clothing (e.g., “shirt,” “dress shirt,” “dress shirt with a fun print”), (3) a physical location, such as a restaurant (see, e.g., Figure 4 Elements 161-b in (a), stores, healthcare providers (e.g., doctors, dentists, hospitals), other service providers (e.g., hair salons or barbershops, accountants, law firms), entertainment (e.g., theaters, amusement parks, zoos), (4) brands, websites, chains, companies, artists, sports teams, (5) services. Services can be, for example, (i) bookable, purchasable, or tradable services (see, e.g., (3) or (4); e.g., insurance services, restaurant reservations, entertainment tickets, venue events), (ii) travel accommodations, flight reservations, transportation services, or other travel-related services, (iii) ongoing services or services that are typically provided over an extended period of time (e.g., tax advice, legal services), (iv) virtual services (e.g., cloud services, subscriptions), or any other services.

[0064] Typically, AI system 166 is configured to determine one or more keywords based on the first output 164 received from AI system 168, regardless of whether the received output 164 is a complete output or a partial output (e.g., partial outputs 164-1, 164-2, and 164-3). This has the effect that while AI system 166 is waiting for further output 164 generated by AI system 168, AI system 166 can determine one or more keywords based on the output 164 that has already been received.

[0065] In some implementations, AI system 166 is configured to determine one or more keywords based on input message 162 in response to receiving input message 162 at step 302. This has the effect that AI system 166 can determine one or more keywords corresponding to input message 162 while AI system 166 is awaiting output 164 generated by AI system 168.

[0066] In some implementations, AI system 166 determines one or more selected keywords from one or more keywords determined based on received input message 162 and one or more keywords determined based on one or more received outputs 164 - n received from AI system 168 .

[0067] In some implementations, latencies and pauses and / or interruptions exhibited by AI system 168 in generating output 164 are used by AI system 166 to generate one or more digital components 161, such that AI system 166 generates one or more digital components 161 while AI system 168 continues to generate additional outputs and / or during pauses before AI system 168 continues to generate additional outputs. This may result in improved system performance and / or reduced response time.

[0068] Reference again Figure 3In some implementations, after executing step 312, AI system 166 optionally continues to execute steps 302 or 204, or continues to step 314. Once AI system 166 incorporates one or more digital components into dialog 200, AI system 166 may receive (further) output 164 generated by AI system 168, or further input message 162 from the user. In some implementations, AI system 166 is configured to prioritize further output provided by AI system 168 over further input message from the user, in which case it will continue to execute step 304. AI system 166 may be configured to buffer further input message from the user until an interruption in output is determined (see step 314). In some implementations, without providing further input message to AI system 168, AI system 166 considers further input message when executing step 304 and subsequent steps (e.g., adds further input message to context). This has the effect that AI system 166 may perform step 306 based on further output 164 generated by AI system 168 and further input messages from the user, which may enrich the context in which step 306 is performed.

[0069] At optional step 314, AI system 166 determines that there is a break in output from AI system 168, such that AI system 166 may continue at step 306, e.g., if the most recent output from AI system 168 has been received, but not yet processed. Alternatively, AI system 166 continues at step 302, awaiting further input messages from the user.

[0070] The conversation may be a "stateful" conversation. The "state" of the conversation may be maintained as, for example, one or more of: a textual summary of the conversation so far (whether from the current session or including previous conversations between the user and the AI ​​system 168), a structured input store (e.g., digital components 161 presented so far, digital components 161 to which the user has reacted positively / negatively), scores for user responses to digital components 161, a "concept embedding" of the conversation so far (e.g., a learned multi-dimensional representation of the concepts discussed so far).

[0071] In some implementations, only a subset of clauses is enhanced with digital components. The number of clauses to be enhanced can be limited and the candidate digital components can be sorted to determine the corresponding limited number of candidate digital components. The ranking can be based on the corresponding scores determined for each candidate digital component or a subset thereof. For example, how many pauses or interruptions can trigger the presentation of the digital component according to a rule-based approach or a modeling-based approach. In a rule-based approach, for example, the maximum number of pauses or scores within a conversation turn, or the interruption within a conversation is determined to receive the digital component 161. In another example, a higher maximum number of pauses / interruptions or scores is determined to receive the digital component 161. The rules can be adjusted or adapted based on data collected about user behavior and / or responses. In other examples, user research can be based on field experiments or other methods to maximize user satisfaction, observations of user behavior (e.g., a tendency to continue to interact with the AI ​​system 168 or a tendency to maintain interaction with the digital component 161). In a modeling-based approach, user satisfaction can be improved, optimized or maximized based on, for example, the quality of the digital component 161. In this way, a greater number of pauses or interruptions determined to be receiving digital components 161 may be permitted, for example, if those digital components 161 are more likely to result in increased user satisfaction and / or if those digital components 161 exhibit particular attributes (e.g., a higher likelihood of interaction).

[0072] In some implementations, the client device 106 is configured to provide additional functionality, such as presenting an electronic document 150. An electronic document is data that presents a set of content at the client device 106. Examples of electronic documents include web pages, word processing documents, portable document format (PDF) documents, images, videos, search result pages, and feeds. Native applications (e.g., "apps" and / or game applications) (such as applications installed on mobile, tablet, or desktop computing devices) are also examples of electronic documents. Electronic documents can be provided to the client device 106 by the electronic document server 104 ("Electronic DocServers").

[0073] For example, electronic document server 104 may include a server hosting a publisher website. In this example, client device 106 may initiate a request for a given publisher web page, and electronic server 104 hosting the given publisher web page may respond to the request by sending a machine executable instruction that initiates presentation of the given web page at client device 106.

[0074] In another example, the electronic document server 104 may include an app server from which the client device 106 may download the app. In this example, the client device 106 may download the files required to install the app at the client device 106, and then execute the downloaded app locally (i.e., on the client device). Alternatively or in addition, the client device 106 may initiate a request to execute the app, which is transmitted to the cloud server. In response to receiving the request, the cloud server may execute the application and stream the user interface of the application to the client device 106, so that the client device 106 does not have to execute the app itself. Instead, the client device 106 may present the user interface generated by the cloud server executing the app, and transmit any user interaction with the user interface back to the cloud server for processing.

[0075] An electronic document may include a variety of content. For example, an electronic document 150 may include native content 152 that is located within the electronic document 150 itself and / or does not change over time. An electronic document may also include dynamic content that may change over time or on a per-request basis. For example, a publisher of a given electronic document (e.g., electronic document 150) may maintain data sources for populating various portions of the electronic document. In this example, a given electronic document may include a script, such as script 154, that causes a client device 106 to request content (e.g., a digital component) from a data source when the given electronic document is processed (e.g., rendered or executed) by a client device 106 (or a cloud server). The client device 106 (or cloud server) integrates the content (e.g., a digital component) obtained from the data source into a given electronic document to create a composite electronic document that includes the content obtained from the data source.

[0076] In some cases, a given electronic document (e.g., electronic document 150) may include a digital component script (e.g., script 154) that references a service device 110 or a particular service provided by the service device 110. In these cases, the digital component script is executed by the client device 106 when the given electronic document is processed by the client device 106. The execution of the digital component script configures the client device 106 to generate a request for a digital component 112 (referred to as a "component request"), which is transmitted to the service device 110 via the network 102. For example, the digital component script may enable the client device 106 to generate a packetized data request including header and payload data. The component request 112 may include event data specifying characteristics, such as the name (or network location) of the server for which the digital component is being requested, the name (or network location) of the requesting device (e.g., client device 106), and / or information that the service device 110 may use to select one or more digital components or other content to be provided in response to the request. The component request 112 is transmitted by the client device 106 to a server of the service device 110 via the network 102 (e.g., a telecommunications network).

[0077] The component request 112 may include event data specifying other event characteristics, such as characteristics of the electronic document being requested and the location of the electronic document at which the digital component may be presented. For example, event data specifying a reference (e.g., a URL) to an electronic document (e.g., a web page) in which the digital component is to be presented, an available location of the electronic document that can be used to present the digital component, the size of the available location, and / or a media type that is eligible for presentation at the location may be provided to the service device 110. Similarly, event data specifying keywords associated with the electronic document ("document keywords") or entities referenced by the electronic document (e.g., people, places, or things) may also be included in the component request 112 (e.g., as payload data) and provided to the service device 110 to facilitate identifying digital components that are eligible for presentation with the electronic document. The event data may also include a search query submitted from the client device 106 to obtain a search results page.

[0078] The component request 112 may also include event data related to other information, such as information that has been provided by a user of the client device, geographic information indicating the state or region in which the component request was submitted, or other information providing the context of the environment in which the digital component will be displayed (e.g., the time of day of the component request, the day of the week of the component request, the type of device (such as a mobile device or a tablet device) that will display the digital component. The component request 112 may be transmitted, for example, over a packetized network, and the component request 112 itself may be formatted as packetized data having a header and payload data. The header may specify a destination for the packet, and the payload data may include any of the information discussed above.

[0079] The service device 110 determines or generates a digital component 161 (e.g., third-party content, such as video files, audio files, images, text, game content, enhanced implementation content, and combinations thereof, all of which may take the form of advertising content or non-advertising content) to be incorporated into the conversation in response to receiving the component request 112 and / or using information included in the component request 112. In some implementations, the method 300 is performed by the service device 110 and / or the AI ​​system 160, wherein the client device 106 is primarily used as a user interface for receiving prompts and displaying conversations. In addition, the digital component 161 generated by the service device 110 can be presented with a given electronic document (e.g., at a location specified by the script 154).

[0080] In some implementations, the digital component 161 is selected in less than one second to avoid errors that may result from delayed selection of the digital component. For example, a delay in providing the digital component in response to the component request 112 may result in a delay in displaying the (enhanced) dialog 200, 200'. In other examples, a delay in providing the digital component in response to the component request 112 may result in page loading errors at the client device 106 or result in portions of the electronic document not being populated even after other portions of the electronic document are presented at the client device 106.

[0081] Moreover, as the delay in incorporating one or more digital components 161 into the conversation 200 increases, the delay is more likely to negatively impact the flow of the conversation, thereby negatively impacting the user's experience of using the AI ​​system 168. The service device 110, AI system 160, client device 106, and / or AI system 166 are configured to process prompts 162 and outputs 164 in a manner that does not add any delay to the AI ​​system 168's regular processing of the conversation. Additionally, increased delays in providing digital components 161 to the client device 106 may result in the electronic document no longer being present at the client device 106 when the digital component is delivered to the client device 106, thereby negatively impacting the user's experience of the electronic document. Additionally, delays in providing the digital component may cause the delivery of the digital component to fail, for example, if the electronic document is no longer present at the client device 106 when the digital component is provided.

[0082] In some implementations, the service device 110 is implemented in a distributed computing system that includes, for example, a server and a set of multiple computing devices 114 that are interconnected and identify and distribute digital components in response to the request 112. The set of multiple computing devices 114 operate together to select from millions of available digital components (DC 1-x) that are eligible for presentation in an electronic document. Millions of available digital components may be indexed, for example, in the digital component database 116. Each digital component index entry may reference a corresponding digital component and / or include distribution parameters (DP1 to DP2) that facilitate (e.g., trigger, constrain, or limit) the distribution / transmission of the corresponding digital component. x For example, the distribution parameters may facilitate (e.g., trigger) the transmission of the digital component by requiring the component request to include at least one criterion that matches (e.g., completely or with some pre-specified level of similarity) one of the distribution parameters of the digital component.

[0083] In some implementations, the distribution parameters for a particular digital component may include a distribution keyword that must be matched (e.g., to an electronic document, a document keyword, or a term specified in a component request 112) in order for the digital component to be eligible for presentation. Additionally or alternatively, the distribution parameters may include embeddings that may use a variety of different data dimensions, such as website details and / or consumption details (e.g., page viewports, user scrolling speed, or other information about data consumption). The distribution parameters may also require that the component request 112 include information specifying a particular geographic region (e.g., a country or state) and / or information specifying that the component request 112 originated from a particular type of client device (e.g., a mobile device or a tablet device) in order for the digital component to be eligible for presentation. The distribution parameters may also specify an eligibility value (e.g., a ranking score, or some other specified value) that is used to evaluate the eligibility of the digital component for distribution / transmission (e.g., with other available digital components).

[0084] The identification of one or more digital components 161 may be divided into a plurality of tasks 117a to 117c, which are then assigned among computing devices within a group of a plurality of computing devices 114. For example, different computing devices in the group 114 may each analyze different portions of the digital component database 116 to identify various digital components having distribution parameters that match the information included in the component request 112. In some implementations, each given computing device in the group 114 may analyze different data dimensions (or groups of dimensions) and communicate (e.g., transmit) the results of the analysis (Res 1 to Res 3) 118a to 118c back to the service device 110. For example, the results 118a to 118c provided by each of the computing devices in the group 114 may identify a subset of digital components that are eligible for distribution in response to the component request and / or a subset of digital components that have certain distribution parameters. The identification of the subset of digital components may include, for example, comparing the event data to the distribution parameters, and identifying the subset of digital components having distribution parameters that match at least some features of the event data. In some implementations, selecting a subset of one or more digital content items is based on a combination of one or more of: (a) a measure of relevance of one or more of an input message, an output, one or more previously received input messages, one or more previously generated outputs, one or more previously received input messages from a previous conversation with a user, one or more previously generated outputs from a previous conversation with a user, one or more previously received input messages from one or more previous conversations with a user, one or more previously generated outputs from one or more previous conversations with a user, a profile of the user, personalized information associated with the user, a location of the user or a location of a client device used by the user and one or more attributes of the client device; (b) expected user satisfaction, wherein the expected user satisfaction includes one or more of a received user satisfaction quantification of the user and a measured user satisfaction quantification of the user, wherein the measured user satisfaction quantification of the user is determined based on one or more behaviors of the user; (c) expected short-term profitability (e.g., within a specified time period such as a day, a week, or a month); and (d) expected long-term profitability (e.g., over a specified time period). The expected long-term profitability is configured to take into account both short-term profitability and long-term behavioral changes.

[0085] The service device 110 aggregates the results 118a-118c received from a set of multiple computing devices 114 and uses information associated with the aggregated results to select one or more digital components to be provided in response to the request 112. For example, the service device 110 can select a set of winning digital components (one or more digital components) based on the results of one or more content evaluation processes, as described below. In turn, the service device 110 can generate and transmit reply data 120 (e.g., digital data representing the reply) over the network 102, which enables the client device 106 to integrate the set of winning digital components into a given electronic document, so that the set of winning digital components (e.g., winning third-party content) and the content of the electronic document are presented at the display of the client device 106 together.

[0086] In some implementations, the client device 106 executes instructions included in the reply data 120 that configure the client device 106 and enable it to obtain a set of winning digital components from one or more digital component servers 108. For example, the instructions in the reply data 120 may include a network location (e.g., a uniform resource locator (URL)) and a script that causes the client device 106 to transmit a server request (SR) 121 to the digital component server 108 to obtain a given winning digital component from the digital component server 108. In response to the request, the digital component server 108 will identify the given winning digital component specified in the server request 121 (e.g., within a database storing a plurality of digital components) and transmit digital component data (DC data) 122 to the client device 106 that presents the given winning digital component in an electronic document at the client device 106.

[0087] When the client device 106 receives the digital component data 122, the client device will render the digital component (e.g., third-party content) and present the digital component at a location specified by or assigned to the script 154. For example, the script 154 may create a walled garden environment (such as a frame) that is presented within, e.g., next to, the native content 152 of the electronic document 150. In some implementations, the digital component is overlaid on (or adjacent to) a portion of the native content 152 of the electronic document 150, and the service device 110 may specify the presentation location within the electronic document 150 in the reply 120. For example, when the native content 152 includes video content, the service device 110 may specify a location or object within a scene depicted in the video content above which the digital component is to be presented.

[0088] In some implementations, the service device 110 includes an AI system 160 configured to perform the method 300 described above. The AI ​​system 160 is configured to autonomously generate a digital component before (e.g., offline) and / or in response to the request 112 (e.g., online or in real time). As described in more detail herein, the artificial intelligence (“AI”) system 160 may collect online content about a particular entity (e.g., a digital component provider or another entity) and summarize the collected online content using one or more language models 170, which may include large language models.

[0089] A large language model (LLM) is a model that is trained to generate and understand human language. LLMs are trained on large datasets of text and code, and they can be used for a variety of tasks. For example, LLMs can be trained to translate text from one language to another; summarize text, such as website content, search results, news articles, or research papers; answer questions about text, such as "What is the capital of Georgia?"; create chatbots that can have conversations with humans; and generate creative text, such as poetry, stories, and code.

[0090] The language model 170 may be any suitable language model neural network that receives an input sequence consisting of text tokens selected from a vocabulary and autoregressively generates an output sequence consisting of text tokens from the vocabulary. For example, the language model 170 may be a Transformer-based language model neural network or a recurrent neural network-based language model.

[0091] In some cases, language model 170 may be referred to as an autoregressive neural network when the neural network used to implement language model 170 autoregressively generates an output sequence of tokens. More specifically, the autoregressively generated output is created by generating each particular token in the output sequence conditioned on the current input sequence, including any tokens that precede the particular text token in the output sequence (i.e., tokens that have been generated for any previous positions in the output sequence that precede the particular position of the particular token), and contextual input that provides context for the output sequence.

[0092] For example, when generating a word-gram at any given position in the output sequence, the current input sequence may include the word-gram at any previous position in the input sequence and the output sequence that precedes the given position. As a specific example, the current input sequence may include the input sequence followed by the word-gram at any previous position in the output sequence that precedes the given position. Optionally, the input and current output sequences may be separated by one or more predetermined word-grams within the current input sequence.

[0093] More specifically, in order to generate a specific word-gram at a specific position in the output sequence, the neural network of the language model 170 may process the current input sequence to generate a score distribution (e.g., a probability distribution) that assigns a corresponding score (e.g., a corresponding probability) to each word-gram in the word-gram vocabulary. The neural network of the language model 170 may then use the score distribution to select a word-gram from the vocabulary as the specific word-gram. For example, the neural network of the language model 170 may greedily select the word-gram with the highest score, or may sample the word-gram from the distribution, for example, using kernel sampling or another sampling technique.

[0094] As a specific example, the language model 170 can be an autoregressive Transformer-based neural network that includes (i) multiple attention blocks, each of which applies a self-attention operation; and (ii) an output subnetwork that processes the output of the last attention block to generate a score distribution.

[0095] The language model 170 may have any of a variety of Transformer-based neural network architectures. Examples of such architectures include those described in J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al., Training compute-optimal large language models, arXiv preprint arXiv:2203.15556, 2022; J. W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, HF Song, J. Aslanides, S. Henderson, R. Ring, S. Young, E. Rutherford, T. Hennigan, J. Menick, A. Cassirer, R. Powell, G. van den Driessche, L. A. Hendricks, M. Rauh, P. Huang, A. Glaese, J. Welbl, S. Dathathri, S. Huang, J.Uesato, J. Mellor, I. Higgins, A. Creswell, N. McAleese, A.Wu, E. Elsen, SMJayakumar, E. Buchatskaya, D. Budden, E. Sutherland, K. Simonyan, M. Paganini, L.Sifre, L. Martens, XL Li, A. Kuncoro, A. Nematzadeh, E. Gribovskaya,D. Donato, A. Lazaridou, A. Mensch, J. Lespiau, M. Tsimpoukelli, N. Grigorev, D. Fritz, T. Sottiaux, M. Pajarskas, T. Pohlen, Z. Gong, D. Toyama, C. de Masson d'Autume, Y. Li, T. Terzi, V. Mikulik, I. Babuschkin, A. Clark, D. de Las Casas, A. Guy, C.Jones,J. Bradbury, M.Johnson, B.A. Hechtman, L. Weidinger, I. Gabriel, W.S. Isaac, E. Lockhart, S. Osindero, L. Rimell, C. Dyer, O. Vinyals, K. Ayoub, J. Stanway, L. Bennett, D. Hassabis, K. Kavukcuoglu, and G. Irving, “Scaling language models: Methods, analysis & insights from training gopher,” CoRR, abs / 2112.11446, 2021; Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” (Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer), arXiv preprint arXiv:1910.10683, 2019; Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, ApoorvKulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le, Towards a human-like open-domain chatbot, CoRR, abs / 2001.09977, 2020; and Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell et al., “Language models are few-shot learners,” arXiv preprint arXiv:2005.14165, 2020. .

[0096] However, in general, a Transformer-based neural network includes a sequence of attention blocks, and during processing of a given input sequence, each attention block in the sequence receives a corresponding input hidden state for each input token in the given input sequence. The attention block then updates each of the hidden states at least in part by applying self-attention to generate a corresponding output hidden state for each of the input tokens. The input hidden state of the first attention block is an embedding of the input token in the input sequence, and the input hidden state of each subsequent attention block is an output hidden state generated by the previous attention block.

[0097] In this example, the output subnetwork processes the output hidden state generated by the last attention block in the sequence for the last input token in the input sequence to generate a score distribution.

[0098] Typically, because language models are autoregressive, the service device 110 can use the same language model 170 to generate multiple different candidate output sequences in response to the same request, for example, by using beam search decoding from a score distribution generated by the language model 170, using a sampling and sorting decoding strategy, by using different random seeds for the pseudo-random number generator used when sampling different runs of the language model 170, or using another decoding strategy that exploits the autoregressive properties of the language model.

[0099] In some implementations, language model 170 is pre-trained, i.e., trained on language modeling tasks that do not require providing evidence in response to user questions, and service device 110 (e.g., using AI system 160) causes language model 170 to generate an output sequence according to a predetermined grammar through natural language prompts in the input sequence.

[0100] For example, service device 110 (e.g., AI system 160) or a separate training system pre-trains language model 170 (e.g., a neural network) on a language modeling task, such as a task that requires predicting the next token that follows the current sequence in the training data given a current text token sequence. As a specific example, language model 170 can be pre-trained on a large text dataset (e.g., text publicly available from the Internet or another text corpus) according to a maximum likelihood objective.

[0101] In some implementations, method 300 as described above is performed by AI system 160. In such implementations, client device 106 primarily provides a user interface for interaction between a user and AI system 168. Prompt 162 and output 164 are relayed via service device 110, which employs AI system 160 as described above with respect to AI system 166 running on client device 106. In some implementations, client 106 includes a basic renderer, where the client includes, for example, a web browser. In such a case, service device 110 creates a web page containing enhanced AI system 168 responses, which can be updated incrementally (e.g., using AJAX), such as in Figure 2 and Figure 4 161 . In the user interface shown. In an alternative, client 106 comprises a chat client, in which case service device 110 can send both (partial) output from AI system 168 and a message containing digital component 161. In some implementations, client 106 can comprise a simple client without AI functionality but running logic (e.g., a web browser running JavaScript that can parse output from AI system 168). Such client 106 can include logic to determine when to request digital component 161 and to request the corresponding digital component 161. In some implementations, client 106 comprises a complete system, such as an app running complex logic and / or AI system.

[0102] The use of AI system 160 to perform method 300 may have the effect that service device 110 may provide more system resources to run AI system 160 than those or client devices 106 running AI system 166. As a result, AI system 160 may apply more complex and / or more comprehensive models and / or may be trained based on more comprehensive training data or training data not available to AI system 166 used by client device 106. Additionally or alternatively, service system 110 and / or AI system 160 may access more network resources to generate digital components 161 and / or may maintain a library of pre-processed or cached digital components locally for reuse during processing of one or more conversations.

[0103] The AI ​​system 160 may perform one or more post-processing operations that evaluate one or more characteristics of the plurality of candidate digital components. The post-processing operations may also include evaluating the relevance of the clause to the query constraints, the level of completeness of the clause relative to the content at the link included in the candidate digital component, and / or evaluating the tone of the clause (e.g., positive or negative). The post-processing operations may be used to score or otherwise assign a priority level to each candidate digital component so that the AI ​​system 160 may rank the plurality of candidate digital components relative to each other and ultimately use one or more of the highest-ranked candidate digital components as output digital components as a reply 120 to the request 112. Note that although the operations of the AI ​​system 160 and the language model 170 are described above as being performed in response to receiving the request 112, at least some of the operations may be performed before receiving the request 112.

[0104] In addition, despite Figure 1 112, but different language models may be specifically trained to handle different prompts at different stages of the processing pipeline. For example, a more general (e.g., larger) language model may be used to generate a summary of online content as an offline process (e.g., independent of receipt of request 112), which may then be inserted into a prompt that is input into a more specialized and faster language model in an online process (e.g., in real time in response to receiving request 112). Additionally, AI system 160 may generate a set of candidate digital components as an offline process (e.g., prior to receiving request 112) and store the set of candidate digital components in a database. In this scenario, when AI system 160 receives request 112, AI system 160 may further evaluate and rank the stored candidate digital components based on additional information and other contextual data (e.g., time of day, date of week, weather conditions, etc.) included in the request.

[0105] Figure 5 4 is a block diagram of an example computer system 400 that can be used to perform the operations described above. System 400 includes a processor 410, a memory 420, a storage device 430, and an input / output device 440. Each of components 410, 420, 430, and 440 can be interconnected, for example, using a system bus 450. Processor 410 is capable of processing instructions for execution within system 400. In one implementation, processor 410 is a single-threaded processor. In another implementation, processor 410 is a multi-threaded processor. Processor 410 is capable of processing instructions stored in memory 420 or on storage device 430.

[0106] The memory 420 stores information within the system 400. In one implementation, the memory 420 is a computer readable medium. In one implementation, the memory 420 is a volatile memory unit. In another implementation, the memory 420 is a non-volatile memory unit.

[0107] The storage device 430 can provide mass storage for the system 400. In one implementation, the storage device 430 is a computer-readable medium. In various implementations, the storage device 430 may include, for example, a hard disk device, an optical disk device, a storage device shared by multiple computing devices over a network (e.g., a cloud storage device), or some other mass storage device.

[0108] The input / output device 440 provides input / output operations for the system 400. In one implementation, the input / output device 440 may include one or more of a network interface device (e.g., an Ethernet card), a serial communication device (e.g., an RS-232 port), and / or a wireless interface device (e.g., an 802.11 card). In another implementation, the input / output device may include a driver device configured to receive input data and send output data to other devices (e.g., a keyboard, a printer, a display, and other peripheral devices 460). However, other implementations may also be used, such as mobile computing devices, mobile communication devices, set-top TV client devices, etc.

[0109] Although in Figure 5 An example processing system is described in the specification, but the subject matter and implementation of the functional operations described in this specification may be implemented in other types of digital electronic circuit systems or in computer software, firmware or hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them.

[0110] An electronic document (referred to simply as a document for brevity) need not correspond to a file. A document may be stored in a portion of a file that holds other documents, in a single file dedicated to the document in question, or in multiple coordinated files.

[0111] With respect to situations in which the systems discussed herein collect and / or use personal information about a user, the user may be provided with the opportunity to enable / disable or otherwise control programs or features that may collect and / or use personal information (e.g., information about the user's social network, social actions or activities, the user's preferences, or the user's current location). Additionally, certain data may be disposed of in one or more ways prior to its storage or use such that personally identifiable information associated with the user is deleted. For example, the user's identity may be anonymized such that personally identifiable information about the user cannot be determined, or the user's geographic location may be generalized (such as to a city, zip code, or state level) if location information is available such that the user's specific location cannot be determined.

[0112] The embodiments of the subject matter and operations described in this specification may be implemented in a digital electronic circuit system or in computer software, firmware or hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them. The embodiments of the subject matter described in this specification may be implemented as one or more computer programs, that is, one or more modules of computer program instructions, which are encoded on a computer storage medium for execution by a data processing device or for controlling the operation of a data processing device. Alternatively or in addition, program instructions may be encoded on an artificially generated propagation signal (e.g., a machine-generated electrical, optical or electromagnetic signal), which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device. The computer storage medium may be a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them, or included in them. In addition, although a computer storage medium is not a propagation signal, a computer storage medium may be a source or destination of computer program instructions encoded in an artificially generated propagation signal. The computer storage medium can also be, or be included in, one or more separate physical components or media (eg, multiple CDs, disks, or other storage devices).

[0113] The operations described in this specification may be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0114] The term "data processing equipment" includes all types of equipment, devices and machines for processing data, for example, including programmable processors, computers, systems on chip, or a plurality of or a combination of the foregoing items. The equipment may include a dedicated logic circuit system, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the equipment may also include code for creating an execution environment for the computer program involved, for example, code constituting a processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. The equipment and execution environment may implement various different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

[0115] This document relates to service devices. As used herein, a service device is one or more data processing devices that perform operations to facilitate the distribution of content over a network. A service device is depicted as a single block in a block diagram. However, although a service device can be a single device or a single group of devices, the present disclosure contemplates that a service device can also be a group of devices, or even a plurality of different systems that communicate to provide various content to a client device. For example, a service device can encompass one or more of a search system, a video streaming service, an audio streaming service, an email service, a navigation service, an advertising service, a gaming service, or any other service.

[0116] A computer program (also referred to as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages ​​or declarative or procedural languages; and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or portions of code). A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communications network.

[0117] The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by a dedicated logic circuit system, and the device can also be implemented as a dedicated logic circuit system, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[0118] For example, processors suitable for executing computer programs include general and special purpose microprocessors, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for performing actions according to instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or be operatively coupled to receive data from it or transfer data to it or both. However, a computer does not have to have such devices. In addition, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name a few. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks (e.g., internal hard disks or removable disks); magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0119] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0120] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component, such as a data server, or includes a middleware component, such as an application server, or includes a front-end component, such as a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), interconnected networks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0121] A computing system may include a client and a server. The client and server are usually remote from each other and typically interact through a communication network. The relationship of client and server arises from computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data (e.g., an HTML web page) to a client device (e.g., for the purpose of displaying data to a user interacting with the client device and receiving user input from the user). Data generated at the client device (e.g., the result of a user interaction) may be received from the client device at the server.

[0122] Although this specification contains many specific implementation details, these details should not be interpreted as limitations on the scope of any invention or the scope that may be claimed, but should be interpreted as descriptions of features that are peculiar to a particular embodiment of a particular invention. Certain features described in the context of a separate embodiment in this specification may also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. In addition, although features may be described above as acting in certain combinations and even initially claimed as such, in some cases, one or more features from the claimed combination may be deleted from the combination, and the claimed combination may involve a sub-combination or a variation of the sub-combination.

[0123] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in a continuous order, or that all of the operations shown be performed, to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0124] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions set forth in the claims can be performed in different orders and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

Claims

1. A computer-implemented method for generating digital components for a conversation between a user and an artificial intelligence (AI) system using a language model, the method comprising: receiving an input message from the user as part of the conversation; receiving as part of the conversation output generated by the AI ​​system in response to the input message and based on the language model; determining one or more keywords based on at least one of the input message and the output; generating one or more digital components based on the one or more keywords; determining that there is a pause in the output as the AI ​​system continues to generate additional output; as well as The one or more digital components are merged into the dialog at a location corresponding to the determined pause for display on a user interface.

2. The method of claim 1, wherein generating the one or more digital components based on the one or more keywords further comprises: sending the one or more keywords to a processing system; as well as The one or more digital components are received from the processing system. 3 . The method of claim 1 , wherein determining the one or more keywords based on at least one of the input message and the output comprises parsing the at least one of the input message and the output.

4. The method of claim 1, wherein determining the one or more keywords is further based on one or more of: the input message, the output, one or more previously received input messages, one or more previously generated outputs, one or more previously received input messages from a previous conversation with the user, one or more previously generated outputs from a previous conversation with the user, one or more previously received input messages from one or more previous conversations with the user, one or more previously generated outputs from one or more previous conversations with the user, a profile of the user, personalized information associated with the user, a location of the user or a location of a client device used by the user, or one or more attributes of the client device.

5. The method of claim 1, wherein the output comprises a plurality of text items, and wherein determining the one or more keywords comprises determining at least one keyword for each text item in the plurality of text items.

6. The method of claim 1, wherein the processing system comprises one or more of a search engine, a digital component server, a reservation system, an assistant system that assists the user in completing a task, or a chatbot.

7. The method of claim 1, further comprising displaying the dialog and the one or more digital components on the user interface.

8. The method of claim 1, wherein generating the one or more digital components based on the one or more keywords comprises individually generating each of the one or more digital components based on at least a subset of the one or more keywords.

9. The method of claim 1, further comprising selecting a subset of one or more digital content items; and wherein merging the one or more digital components into the conversation comprises merging the subset of the one or more digital content items into the conversation.

10. The method of claim 9, wherein selecting the subset of the one or more digital content items is based on a combination of one or more of the following: a measure of relevance of one or more of: the input message, the output, one or more previously received input messages, one or more previously generated outputs, one or more previously received input messages from a previous conversation with the user, one or more previously generated outputs from a previous conversation with the user, one or more previously received input messages from one or more previous conversations with the user, one or more previously generated outputs from one or more previous conversations with the user, a profile of the user, personalized information associated with the user, a location of the user or a location of the client device used by the user, or one or more attributes of the client device, an expected user satisfaction, wherein the expected user satisfaction comprises one or more of a received user satisfaction quantification of the user and a measured user satisfaction quantification of the user, wherein the measured user satisfaction quantification of the user is determined based on one or more behaviors of the user, Expected short-term profitability, Expected long-term profitability, wherein the expected long-term profitability is configured to account for both short-term profitability and long-term behavioral changes.

11. The method of claim 1, further comprising: A user interaction related to a respective one of the one or more digital content items is determined.

12. The method of claim 11, further comprising: A respective score is determined for each of the one or more digital content items based on one or more respective user interactions, wherein selecting a subset of the one or more digital content items is based on the respective scores of the one or more digital content items.

13. The method of claim 1, further comprising: determining that there is a break in the output while the AI ​​system is waiting for a next input message; determining one or more updated keywords based on output received prior to the interruption, the output including one or more pauses; generating one or more updated digital components corresponding to the one or more updated keywords; as well as The one or more updated digital components are merged into the dialog at a location corresponding to the determined interruption for display on the user interface.

14. The method of claim 3, wherein parsing the at least one of the input message and the output is performed as a lightweight parsing.

15. The method of claim 1, wherein determining the one or more keywords based on at least one of the input message and the output and generating at least one of the one or more digital components based on the one or more keywords are performed in response to one of: receiving the input message from the user as part of the conversation; and receiving the output generated by the AI ​​system as part of the conversation.

16. A system for generating digital components for a conversation between a user and an artificial intelligence (AI) system using a language model, comprising: Data storage device; as well as One or more processors configured to interact with the data storage device and perform operations when executing instructions, the operations comprising receiving an input message from the user as part of the conversation, receiving as part of the conversation an output generated by the AI ​​system in response to the input message and based on the language model, determining one or more keywords based on at least one of the input message and the output, generating one or more digital components based on the one or more keywords, determining that there is a pause in the output as the AI ​​system continues to generate additional output, and The one or more digital components are merged into the dialog at a location corresponding to the determined pause for display on a user interface.

17. The system of claim 16, wherein generating the one or more digital components based on the one or more keywords further comprises: sending the one or more keywords to a processing system; as well as The one or more digital components are received from the processing system.

18. The system of claim 16, wherein determining the one or more keywords based on at least one of the input message and the output comprises parsing the at least one of the input message and the output.

19. A system as described in claim 16, wherein determining the one or more keywords is further based on one or more of the following: the input message, the output, one or more previously received input messages, one or more previously generated outputs, one or more previously received input messages from a previous conversation with the user, one or more previously generated outputs from a previous conversation with the user, one or more previously received input messages from one or more previous conversations with the user, one or more previously generated outputs from one or more previous conversations with the user, a profile of the user, personalized information associated with the user, the location of the user or the location of a client device used by the user, or one or more attributes of the client device.

20. The system of claim 16, wherein the output comprises a plurality of text items, and wherein determining the one or more keywords comprises determining at least one keyword for each text item in the plurality of text items.

21. The system of claim 16, wherein the processing system comprises one or more of a search engine, a digital component server, a reservation system, an assistant system that assists the user in completing a task, or a chatbot.

22. The system of claim 16, further comprising displaying the dialog and the one or more digital components on the user interface.

23. The system of claim 16, wherein generating the one or more digital components based on the one or more keywords comprises individually generating each of the one or more digital components based on at least a subset of the one or more keywords.

24. The system of claim 16, wherein the operation further comprises selecting a subset of one or more digital content items; and wherein merging the one or more digital components into the conversation comprises merging the subset of the one or more digital content items into the conversation.

25. The system of claim 24, wherein selecting the subset of the one or more digital content items is based on a combination of one or more of: a measure of relevance of one or more of: the input message, the output, one or more previously received input messages, one or more previously generated outputs, one or more previously received input messages from a previous conversation with the user, one or more previously generated outputs from a previous conversation with the user, one or more previously received input messages from one or more previous conversations with the user, one or more previously generated outputs from one or more previous conversations with the user, a profile of the user, personalized information associated with the user, a location of the user or a location of the client device used by the user, or one or more attributes of the client device, an expected user satisfaction, wherein the expected user satisfaction comprises one or more of a received user satisfaction quantification of the user and a measured user satisfaction quantification of the user, wherein the measured user satisfaction quantification of the user is determined based on one or more behaviors of the user, Expected short-term profitability, Expected long-term profitability, wherein the expected long-term profitability is configured to account for both short-term profitability and long-term behavioral changes.

26. The system of claim 16, wherein the operations further comprise: A user interaction related to a respective one of the one or more digital content items is determined.

27. The system of claim 26, wherein the operations further comprise: A respective score is determined for each of the one or more digital content items based on one or more respective user interactions, wherein selecting a subset of the one or more digital content items is based on the respective scores of the one or more digital content items.

28. The system of claim 16, wherein the operations further comprise: determining that there is a break in the output while the AI ​​system is waiting for a next input message; determining one or more updated keywords based on output received prior to the interruption, the output including one or more pauses; generating one or more updated digital components corresponding to the one or more updated keywords; as well as The one or more updated digital components are merged into the dialog at a location corresponding to the determined interruption for display on the user interface.

29. The system of claim 18, wherein parsing the at least one of the input message and the output is performed as a lightweight parsing.

30. The system of claim 16, wherein determining the one or more keywords based on at least one of the input message and the output and generating at least one of the one or more digital components based on the one or more keywords are performed in response to one of: receiving the input message from the user as part of the conversation; and receiving the output generated by the AI ​​system as part of the conversation.

31. A non-transitory computer-readable medium storing instructions for using a language model to generate digital components for a conversation between a user and an artificial intelligence (AI) system, wherein the instructions, when executed, cause one or more processors to perform operations, the operations comprising receiving an input message from the user as part of the conversation, receiving as part of the conversation an output generated by the AI ​​system in response to the input message and based on the language model, determining one or more keywords based on at least one of the input message and the output, generating one or more digital components based on the one or more keywords, determining that there is a pause in the output as the AI ​​system continues to generate additional output, and The one or more digital components are merged into the dialog at a location corresponding to the determined pause for display on a user interface.

32. The non-transitory computer readable medium of claim 31 , wherein generating the one or more digital components based on the one or more keywords further comprises: sending the one or more keywords to a processing system; as well as The one or more digital components are received from the processing system.

33. The non-transitory computer-readable medium of claim 31, wherein determining the one or more keywords based on at least one of the input message and the output comprises parsing the at least one of the input message and the output.

34. A non-transitory computer-readable medium as described in claim 31, wherein determining the one or more keywords is further based on one or more of the following: the input message, the output, one or more previously received input messages, one or more previously generated outputs, one or more previously received input messages from a previous conversation with the user, one or more previously generated outputs from a previous conversation with the user, one or more previously received input messages from one or more previous conversations with the user, one or more previously generated outputs from one or more previous conversations with the user, a profile of the user, personalized information associated with the user, the location of the user or the location of a client device used by the user, or one or more attributes of the client device.

35. The non-transitory computer-readable medium of claim 31, wherein the output comprises a plurality of text items, and wherein determining the one or more keywords comprises determining at least one keyword for each text item in the plurality of text items.

36. The non-transitory computer-readable medium of claim 31, wherein the processing system comprises one or more of a search engine, a digital component server, a reservation system, an assistant system that assists the user in completing a task, or a chatbot.

37. The non-transitory computer-readable medium of claim 31 , further comprising displaying the dialog and the one or more digital components on the user interface.

38. The non-transitory computer-readable medium of claim 31, wherein generating the one or more digital components based on the one or more keywords comprises individually generating each of the one or more digital components based on at least a subset of the one or more keywords.

39. A non-transitory computer-readable medium as described in claim 31, wherein the operation further includes selecting a subset of one or more digital content items; and wherein merging the one or more digital components into the conversation includes merging the subset of the one or more digital content items into the conversation.

40. The non-transitory computer-readable medium of claim 39, wherein selecting the subset of the one or more digital content items is based on a combination of one or more of: a measure of relevance of one or more of: the input message, the output, one or more previously received input messages, one or more previously generated outputs, one or more previously received input messages from a previous conversation with the user, one or more previously generated outputs from a previous conversation with the user, one or more previously received input messages from one or more previous conversations with the user, one or more previously generated outputs from one or more previous conversations with the user, a profile of the user, personalized information associated with the user, a location of the user or a location of the client device used by the user, or one or more attributes of the client device, an expected user satisfaction, wherein the expected user satisfaction comprises one or more of a received user satisfaction quantification of the user and a measured user satisfaction quantification of the user, wherein the measured user satisfaction quantification of the user is determined based on one or more behaviors of the user, Expected short-term profitability, Expected long-term profitability, wherein the expected long-term profitability is configured to account for both short-term profitability and long-term behavioral changes.

41. The non-transitory computer readable medium of claim 31 , wherein the operations further comprise: A user interaction related to a respective one of the one or more digital content items is determined.

42. The non-transitory computer readable medium of claim 41, wherein the operations further comprise: A respective score is determined for each of the one or more digital content items based on one or more respective user interactions, wherein selecting a subset of the one or more digital content items is based on the respective scores of the one or more digital content items.

43. The non-transitory computer readable medium of claim 31 , wherein the operations further comprise: determining that there is a break in the output while the AI ​​system is waiting for a next input message; determining one or more updated keywords based on output received prior to the interruption, the output including one or more pauses; generating one or more updated digital components corresponding to the one or more updated keywords; as well as The one or more updated digital components are merged into the dialog at a location corresponding to the determined interruption for display on the user interface.

44. The non-transitory computer-readable medium of claim 33, wherein parsing the at least one of the input message and the output is performed as a lightweight parsing.

45. The method of claim 31 , wherein determining the one or more keywords based on at least one of the input message and the output and generating at least one of the one or more digital components based on the one or more keywords are performed in response to one of: receiving the input message from the user as part of the conversation; and receiving the output generated by the AI ​​system as part of the conversation.

Images