Guided Dialogue Using Language Generation Neural Networks and Search

Abstract

A method, system, and apparatus, including a computer program encoded on a computer storage medium, for enabling a user to conduct a dialogue. System implementations can learn when to rely on supporting evidence obtained from an external search system via a search system interface and generate answers for the user that are consistent with the preferences of a previously trained response selection neural network. System implementations can also use a previously trained rule violation detection neural network to generate answers that take into account previously learned rules.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Provisional Application No. 63 / 408,430, filed September 20, 2022, which is incorporated herein by reference in its entirety. [Background technology]

[0002] This specification relates to a method for interaction using language generation neural networks and search, in particular embodiments subject to a set of rules, which can be used to obtain information and to control real-world systems.

[0003] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict an output for a received input. Some neural networks contain one or more hidden layers in addition to an output layer. The output of each hidden layer is used as the input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from the received input according to the current values ​​of its respective set of parameters. Summary of the Invention [Means for solving the problem]

[0004] This specification describes a system implemented as a computer program on one or more computers at one or more locations that enables a user to conduct interactions to obtain information using, among other things, a language generation neural network.

[0005] In embodiments, the system can provide information to the user either based on knowledge stored in a trained language generation neural network, or by supplementing this knowledge with one or more external searches to balance the computing requirements and communication bandwidth needed to search for relevant information.

[0006] A user of the system may be a human user or a machine. Some implementations of the system may be used by a human to conduct general conversations with a computer system. Some implementations of the system may be used to diagnose technical faults in a mechanical or computer system or network. Some implementations of the system may be used for natural language control of tasks in a real-world environment, where the acquired information may be used, for example, to control tasks performed by a mechanical or computer system.

[0007] In one aspect, among other things, one or more computer-implemented methods and corresponding systems are described for enabling a user to obtain information through an interaction between the user and an agent, such as a suitably programmed computer system, that includes a first trained language generation neural network.

[0008] System implementations learn when to rely on supporting evidence obtained from an external search system via a search system interface, and therefore when to provide a "supported" response as opposed to an "unsupported" response that does not rely on an external search. System implementations can also generate answers for a user that are consistent with the preferences of a previously trained response selection neural network. System implementations can also use a previously trained rule violation detection neural network to generate answers that take into account previously learned rules.

[0009] In other aspects, one or more computer-implemented methods and corresponding systems are described for training systems of the type described above, particularly to train an interaction neural network system to enable a user to use an agent including a first language generation neural network, e.g., to obtain information through an interaction between the user and the agent.

[0010] Also described is a machine learning computer trained to enable a user to obtain information through interaction, the trained machine learning computer system including a trained first language generation neural network, a trained response selection neural network, and optionally a trained rule violation detection neural network.

[0011] Also described is a dialogue training computer for training the dialogue computer system. The language generation / language model neural network may be stored on a training computing device, and the search system may be remote from this device.

[0012] The subject matter described herein can be implemented in particular embodiments to realize one or more of the following advantages.

[0013] The above-mentioned "unsupported" responses can be generated without external search. Therefore, they can be provided locally by a language generation neural network, although this is not necessarily optimal. For example, sometimes the language generation neural network may provide an inaccurate response, or the response may require more up-to-date information than was available when the language generation neural network was trained. One solution would be to perform an extensive search before providing an answer, but this could result in inefficient use of communication bandwidth, for example, to eliminate servers. Another solution would be to not perform a search, which has the drawbacks described above. A further solution would be to periodically retrain the language generation neural network, but this would be computationally inefficient, and sending updated neural network parameters to the user would incur a significant bandwidth penalty. The described hybrid solution uses a combination of a trained language generation neural network and a response selection neural network to facilitate performing only a limited search, thus reducing the use of available communication bandwidth.

[0014] Complementarily, the described hybrid solutions reduce the need for transmitting large amounts of data that would otherwise be required for frequent updates of trained language generation neural networks if relied solely on to provide factual information. The described hybrid solutions can also provide access to larger corpora of information than would be possible if relying solely on trained language generation neural networks or solely on search to retrieve information.

[0015] In embodiments, the ability to request subsequent requests within the same context as the initial request further facilitates limiting searches per round of interaction. That is, two rounds of searches overall may be fewer than would be required if all searches were performed in response to a single search query. Thus, the ability to request subsequent requests serves the technical objective of reducing the number of search requests and potentially reducing the execution time of the processes used to obtain answers to the requests.

[0016] Additionally, embodiments may use multiple rules to filter responses according to any desired or undesired characteristics. For example, one or more rules may be defined to reduce the likelihood of unsafe, undesirable, or inefficient system behavior, or the likelihood of offensive content, misinformation, or confidential or private information (such as personal contact information) in the information provided by the system. As another example, a language generation neural network trained on multiple corpora of documents or other information may later be used to provide information to a user, for example, by answering questions, with essentially all of the information used for training being available. One or more rules may be implemented to restrict access to certain types of information, such as, for example, to limit access to information for a particular user. Thus, some embodiments of the system may be used to restrict access to personal data, confidential data, or other data, such as, for example, copyrighted data.

[0017] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0018] [Figure 1] 1 shows an example of a dialogue system. [Figure 2]FIG. 2 is a flow diagram of a first exemplary process for using the dialogue system. [Figure 3] 1 illustrates an exemplary user interface of the dialogue system. [Figure 4] 10 illustrates, in outline, a second exemplary process for using the dialogue system. [Figure 5] FIG. 1 is a flow diagram of a first exemplary process for training a dialogue system. [Figure 6] 1 illustrates an exemplary neural network architecture for a dialogue system. [Figure 7] 10 illustrates, in outline, a second exemplary process for training a dialogue system. [Figure 8] 1 illustrates the performance of an exemplary implementation of a dialogue system. DETAILED DESCRIPTION OF THE INVENTION

[0019] Like reference numbers and designations in the various drawings refer to like elements.

[0020] 1 illustrates an example of a dialogue system 100. The dialogue system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations, in which the systems, components, and techniques described herein below may be implemented.

[0021] The dialogue system 100 can enable a user to obtain information through dialogue. For example, the dialogue system 100 can be used as an agent to enable dialogue between the agent, i.e., the system, and a user, e.g., a human user of the system 100. Later, the dialogue system 100 may be referred to as an agent or as a "Sparrow" ("Sparrow" being one particular exemplary implementation).

[0022] The dialogue system 100 includes a language generation neural network 110. When used in inference, the language generation neural network 110 is a trained language generation neural network 110. During training, the language generation neural network 110 is (further) trained, e.g., fine-tuned, by a training engine 150. After training, the training engine 150 is not needed.

[0023] Since there may be other language generation neural networks or language model neural networks in the dialogue system 100, the language generation neural network 110 is hereinafter referred to as the first language generation neural network.

[0024] In an embodiment, the dialogue system 100 communicates with a human user using text spoken or written in a natural language, but in general the language generation neural network 110 can also generate text in a computer language (any formal language used to communicate with a computer) such as a markup language, or a command or configuration language, or a data exchange language such as JSON, or a programming language.

[0025] The language generation neural network or language model neural network described herein can include a sequence-to-sequence model that receives an input sequence of natural language tokens and generates an output sequence of natural language tokens. Typically, natural language tokens define words or word pieces (e.g., word segments or morphemes), but natural language tokens may also define letters, numbers, i.e., characters, or multiple words. Tokens may include tokens representing punctuation. In some implementations, the output sequence of natural language tokens is generated one word or word piece at a time, for example, until one or more end-of-statement tokens are obtained or until a maximum length output is generated. Trained language generation neural networks can be obtained commercially or can be trained, for example, using a corpus of text and supervised learning with maximum likelihood loss.

[0026] In general, any language generation neural network can be used as one of the language generation neural networks or language model neural networks described herein, such as, for example, an autoregressive language generation neural network, or a language generation neural network that does not rely on an autoregressive model, such as a recurrent language generation neural network, or a denoising autoencoder-based language model. In some implementations, the language generation / model neural network can be an expert mixture model.

[0027] As an example, the language generation neural network described herein may be a transformer-based language model neural network, and in particular, an autoregressive transformer-based language model neural network. A transformer neural network may be characterized by having successive self-attention neural network layers. The self-attention neural network layers have an attention layer input for each element of the input, and apply an attention mechanism via the attention layer input to generate an attention layer output for each element of the input; there are many different attention mechanisms that may be used.

[0028] Similar to the language generation neural network 110, the dialogue system 100 can include one or more language model neural networks. A language model neural network is similar to a language generation neural network, but need not generate a linguistic output. For example, rather than generating an output sequence of natural language tokens, a language model neural network may process an input sequence of natural language tokens to generate a vector or scalar output. Because a language generation neural network effectively includes a language model neural network, the two terms are used somewhat interchangeably.

[0029] It is surprising, but well established, that so-called large language model (language generation) neural networks can perform tasks that they were not explicitly trained to perform: for example, they can perform translation tasks (if the training corpus contains words in different languages), arithmetic tasks, and many other tasks.

[0030] A language generation neural network can be directed to perform a particular task by providing a natural language description of a desired response as input or a "prompt." The prompt may be, for example, a few examples, such as 1-10, of a query, and a few-shot prompt in which exemplary output is provided in the text before the actual query.

[0031] Also, or alternatively, a language model (language generation) neural network can be "fine-tuned" to perform a particular task by taking a pre-trained language model neural network that has been trained on a large corpus of examples, and then further training all parts of the language model neural network on a relatively small number of examples that are specific to the type of task that is to be performed. Thus, for example, a trained language model neural network can perform control and diagnostic tasks of the types described below.

[0032] Some implementations of the methods / systems described herein use large-scale language models / language generation neural networks. Such large-scale language models / language generation neural networks may have more than 1 billion, 10 billion, or 100 billion trainable / trained parameters. Such large-scale language models / language generation neural networks may have been trained with more than 10 billion, 100 billion, or 1 trillion words or tokens representing words.

[0033] Generally, the language model neural network and the language generation neural network described herein are trained neural networks. When used in the training methods described below, the language model neural network and the language generation neural network may be further trained, or "fine-tuned." For example, the language generation neural network 110 may be fine-tuned using reinforcement learning.

[0034] The various different language generation neural networks and language model neural networks described herein may, but need not, comprise different instances of the same language model neural network / language generation neural network. By way of example only, the language model neural network and the language generation neural network may each comprise an instance of the (trained) Chinchilla model (Hoftman et al., 2022, arXiv:2203.15556). As another example, one or more of the models may use LaMDA (Thoppilan et al., 2022, arXiv:2201.08239). One or more of the models may optionally be fine-tuned, for example, using supervised fine-tuning. For example, when trained using reinforcement learning as described below, one or more may have been previously fine-tuned on some of the same data used for reinforcement learning.

[0035] In an embodiment, the (first) language generation neural network 110 is configured to process a contextual input including one or more prompts, each of which includes one or more natural language statements formatted in any suitable manner.

[0036] In general, references to language generation neural networks or language model neural networks that process natural language text are to language generation neural networks or language model neural networks that process text in a tokenized form, which may be obtained from a tokenizer such as SentencePiece.

[0037] The language generation neural network 110 processes the context input according to the first language generation neural network parameters to generate a natural language output, such as an output including one or more natural language statements. The natural language output may be obtained by a sampling process, such as kernel sampling. That is, the natural language output may be generated probabilistically.

[0038] As an example, the language generation neural network 110 may be trained such that, given a text prompt containing a sequence of tokens in a natural language, the neural network can generate the next token in the sequence. This process can be repeated to expand the text prompt one token at a time and generate a natural language output, i.e., to autoregressively generate natural language output for each token. At each "time step," the language model neural network processes the current sequence to generate a probability distribution over the vocabulary of tokens. The next token can then be selected using the probability distribution, such as by sampling from the distribution using kernel sampling or other sampling techniques, or by selecting the token with the highest probability. Tokens in the vocabulary can include any of a variety of tokens, such as words, subwords, letters, punctuation and other symbols, and some combination of digits. Such language generation neural networks are typically trained on a corpus of text consisting of tokens from a vocabulary (and optionally other tokens that can be mapped to tokens outside the specified vocabulary) to predict the next token in a sequence of tokens from the training data.

[0039] The dialogue system 100 is configured to receive a request, in an embodiment a natural language request 102, and to generate a language output, in particular a natural language output 104 that is a response to the natural language request 102. This can continue in turn, so that the user and the dialogue system 100 are engaged in a dialogue.

[0040] In some implementations, the initial contextual input to the language generation neural network 110 can include an initial prompt that prompts the model to continue as well, i.e., engage in a dialogue. However, the use of an initial prompt is not required. For example, the language generation neural network 110 may have been fine-tuned for dialogue.

[0041] The general format of such a prompt might be, for example: User:<User order> Sparrow:<Response> where placeholders are indicated by <>.

[0042] An example of such an initial prompt (which may be longer or shorter) is as follows: Below is a conversation between a highly knowledgeable and intelligent AI assistant called Sparrow and a human being called the user. In the following interaction, the user and Sparrow converse in natural language, and Sparrow does its best to answer the user's questions. Sparrow was built to be polite, courteous, and inclusive. Sparrow knows a lot and always tells the truth. The conversation begins. User: Okay, Sparrow, let's start with some warm-up questions. Who was the president of the United States in 2021? Sparrow: Joe Biden. User: Great job! Do you think Biden is a better president than his predecessor? Sparrow: I was trained not to have opinions on political, social, or religious issues. Is there anything else you'd like to know? As shown in this example, the initial prompt may include one or more instances where the agent refuses to answer to avoid harm.

[0043] The initial context input may also include text to prompt a response from the language generation neural network 110. For example, the initial context input may include two line breaks, the current role in the dialogue, and a colon, e.g., "\n\nSparrow:".

[0044] In an embodiment, token generation from the language generation neural network 110 terminates when a set of one or more end-of-statement tokens, such as a token with the ending suffix "\n\nUser", is generated, or when a maximum length output statement string is generated. In an embodiment, such end-of-statement suffixes are used only to determine the end of the sequence; that is, they are otherwise ignored.

[0045] In general, the (trained) language generation neural network 110 will emit a correctly formatted response when provided with contextual inputs such as those described above.

[0046] The dialogue system 100 can generate natural language outputs 104 for successive dialogue turns, later referred to as dialogue update iterations. When generating a natural language answer for a user, the dialogue system 100 may be provided with context input that includes the history of the preceding conversation or portions thereof, such as a selection or summary thereof, depending on, for example, the maximum length of the context input. Optionally, an initial prompt or a version of an initial prompt may also be included. In some other cases, the language generation neural network 110 may already have state that encodes the dialogue history, which does not need to be provided again.

[0047] As an example, in some embodiments, the context input for an agent's dialogue turn may include a concatenation of an initial prompt, dialogue history, and participant name, e.g., "agent" or "Sparrow," and a colon ":." In some embodiments, the dialogue system 100 may be trained using self-play, that is, by allowing the system to conduct a dialogue with itself. The participant name may then include "user." For example, the context input for a user's dialogue turn may include an initial prompt, dialogue history, and "user," and a colon.

[0048] An implementation of the dialogue system 100 includes a search system interface 140. The search system interface 140 may be an interface to any type of search system, such as one or more of a database-based search system, or an internet or other network search engine, or a search system for searching a corpus of documents in a text database, which may be, for example, a proprietary text database. The search system interface 140 may include, for example, a search system API (Application Programming Interface).

[0049] The dialogue system 100 can use the search system interface 140 by generating one or more search queries using the language generation neural network 110, such as by processing a contextual input to the language generation neural network 110 that includes an evidential prompt, such as a "search query." For example, in some implementations, the contextual input for generating the search query can include a concatenation of the initial prompt, the dialogue history, and the "search query" as the participant name followed by a colon.

[0050] An initial evidence prompt may be used to prompt the language generation neural network 110 to generate a search query. This may include, for example, a "search query" and a "search result" as participants, rather than just a "user" and an "agent." A typical format for such an initial evidence prompt may be, for example, as follows: User:<User order> Search query: <search query> Search results: <search results> Sparrow:<Response> where placeholders are indicated by <>. As an example, "<search results>" for a call to the Google™ Search API may be of the form: Search results: Page Title: <Page Title> <document fragment> Here, results from the Google™ Search API can be used to append "<page title>" and "<document fragment>". Optionally, documents returned by the search can be truncated to include only the most relevant parts or parts according to some metric of relevance to the search query, for example based on fuzzy string matching.

[0051] Examples of such initial evidence prompts (which may be longer or shorter) are as follows: Below is a conversation between a highly knowledgeable and intelligent AI assistant called Sparrow and a human being called the user. In the following interaction, the user and Sparrow converse in natural language, and Sparrow does its best to answer the user's questions. Sparrow was built to be polite, courteous, and inclusive. Sparrow knows a lot and always tells the truth. Sparrow can use Google to obtain external knowledge when needed. Sparrow can issue search queries and use them to get answers from Google. Sparrow answers carefully based on the search results and only provides information that is supported by the results. The year is 2022, and specifically, today's date is {current_weekday}{current_day}{current_month}2022. Sparrow's response reflects this. User: What color is the sky? Sparrow: It's blue. User: Why? Search Query: Why is the sky blue? Search results: Page Title: Sky With the exception of direct sunlight, most light in the daytime sky is caused by scattering, dominated by a small-particle limit called Rayleigh scattering. Scattering by molecular-sized particles (such as in air) is greater both toward and away from the light source than in directions perpendicular to the incident path. Scattering is significant for all visible wavelengths of light, but is stronger at the shorter (bluer) end of the visible spectrum, meaning that scattered light is bluer than its source, the sun. The remaining direct sunlight loses some of its shorter wavelength components and appears slightly less blue. Sparrow: This is due to Rayleigh scattering, which causes short wavelength blue light to scatter more than other colors in the spectrum. User: Who was Rayleigh? Search Query: Who is Rayleigh? Search results: Page title: John William Strutt, 3rd Baron Rayleigh - Wikipedia John William Strutt, 3rd Baron Rayleigh, OM, PC, PRS ( / "reIli / ; 12 November 1842 – 30 June 1919) was He was a British mathematician who made significant contributions to science. He spent his entire academic career at the University of Cambridge. He was awarded the Nobel Prize in Physics in 1904, among other honours, for "his most important studies on the density of gases, and for his discovery of argon in connection with these studies." Sparrow: British mathematician. He won the Nobel Prize in Physics in 1904.

[0052] In general, the search query may have any suitable structure and may be defined, for example, using initial evidence prompts and / or by fine-tuning the language generation neural network 110, for example, using supervised fine-tuning.

[0053] As some examples, a search query may include a natural language request, or a truncated, condensed, or modified form of a natural language request, or a search query may include a search query structured according to, for example, a search-specific computer language or a programming language such as SQL (i.e., a search language such as a database query language). One or more search queries may be provided to search system interface 140.

[0054] One or more search results are received from the search system interface 140, and the content can then be incorporated into the contextual input for the language generation neural network 110 in any suitable format, such as, for example, as natural language or as structured natural language. The search results are also referred to herein as "evidence."

[0055] In an embodiment, the dialogue system 100 also includes a response selection neural network 120, which may, for example, optionally include a pre-trained second language model neural network. As one example, the response selection neural network 120 may be obtained from a language generation neural network having a head, such as a linear layer, to generate preference scores. As another example, the preference scores may be determined from log-likelihoods assigned to language outputs from the language generation neural network.

[0056] In an embodiment, the response selection neural network 120 is configured (trained) to process the context input and the continuation or "completion" according to learnable parameters, such as weights, of the response selection neural network to generate a preference score. The "completion" may be a natural language response to the context input, such as a natural language output statement generated by the language generation neural network 110. The preference score may provide a measure of preference for the completion given the context input. When used to train the language generation neural network 110, the preference score may be used as a first reward, as described below. As a result, the response selection neural network 120 may be described as a preference-reward model.

[0057] In some implementations, there may be two versions of the response selection neural network 120 to use depending on whether the context input includes supporting evidence. One version may be trained only with training data without supporting evidence, while the other version may be trained with training data with and without supporting evidence (see below). In some other implementations, a single version of the response selection neural network 120 is used regardless of whether the context includes supporting evidence. Here, supporting evidence may refer to a representation of one or more search results obtained in response to one or more search queries. Different versions of the response selection policy neural network may be versions of the response selection policy neural network with the same architecture but different parameter values. If there are two versions of the response selection neural network 120, when the dialogue system 100 is used for inference, as opposed to training, in an implementation, only the version that has verified the supporting evidence is used (for re-ranking, as described below).

[0058] As an example, an implementation of the response selection neural network 120 using a language model neural network can be trained using training data captured from a human user. For example, a human evaluator may be given an incomplete (training) dialogue, possibly including evidence, and multiple possible statements for continuing the dialogue, e.g., each corresponding to a different sample or model, and the evaluator selects the response they believe to be best. In an implementation in which the dialogue system 100 is trained using self-play, the human evaluator may be asked to select the best response for both the user and the agent. The selected response can then be used to continue the dialogue, e.g., up to some maximum number of turns, or until the evaluator skips the task or indicates that all sequences are unsuccessful. Response preferences can be collected across multiple statement comparisons. For example, in a four-statement comparison, two responses may be sampled without evidence (generated with a no-evidence prompt) and two responses may be sampled with evidence (generated with a prompt including a search query and search results).

[0059] Continuing with the example, multi-option comparison can be used to generate multiple training data pairs, each including a context and a completion. One pair can include the best completion, the other can include the option not selected, and optionally also include a pair including a distracting statement sampled from an unrelated conversation.

[0060] When training the response selection neural network 120, the inputs may include a context, which is the current history of the (training) interaction, and a continuation (completion). When evidence is used, the context omits the order of the search query and search results, and the completion is represented as a combination of three intermediate orders. For example, the context might include "User: A Sparrow: B User: C," a completion without evidence might include "Sparrow: D," and a completion with evidence might include "Search query: D Search results: E Sparrow: F." This can provide training signals of the quality of the response and the quality of the search query and search results, as well as signals indicating when the use of a search query is preferable (over a response without evidence).

[0061] Optionally, additional training data can be collected by explicitly asking a human user, for each response in a multi-option comparison, whether the response is plausible (i.e., reasonable, on-topic, and likely true) and whether the response is supported by the evidence provided (i.e., does the evidence convince the user that the answer is correct). These can provide class labels for the classification loss.

[0062] Thus, in general, the second language model neural network that constitutes response selection neural network 120 may be trained using training data items, each data item comprising a sample of an interaction including a natural language request, a set of natural language responses generated by one or more training language generation neural networks, and preference data indicating the relative preference of the natural language responses (or, for example, that none of the responses are preferred because they are all "bad").

[0063] The set of natural language responses can be generated, for example, by an earlier version of the first language generation neural network or by other trained language generation neural networks, such as, for example, one with the ability to issue search queries (which can be provided by a few example prompts). For example, the set of natural language responses can include responses generated by using one or more training language generation neural networks to process contextual inputs including samples of interactions with and without search results from search queries based on natural language requests. This can provide learning signals for both the quality of the response and whether to use supporting evidence.

[0064] In general, the response selection neural network 120 can be trained using any suitable training objective, such as, for example, maximum likelihood loss, cross-entropy loss, or regression loss, using training data including contextual inputs and completions as described above. In general, training the response selection neural network 120 can involve backpropagating gradients of the training objective to update the learnable parameters of the response selection neural network. This may use any suitable gradient descent optimization algorithm, such as, for example, Adam or other optimization algorithm. Other neural networks described herein may be trained similarly.

[0065] In an embodiment, prior to training, the trained response selection neural network 120, and in particular the second language model neural network, may be initialized with an instance of the trained language model neural network, such as, for example, with an instance of the trained Chinchilla model that is then fine-tuned.

[0066] In some implementations, the second language model neural network is configured (trained) to generate preference scores on the Elo scale.

number

[0067] More generally, training the second language model neural network may include backpropagating the gradient of a response selection objective function that depends on an exponential function of the preference score of the relatively most preferred one of the natural language responses scaled by the sum of the respective exponential functions of the preference scores of the set of natural language responses. Optionally, an additional term, e.g., a constant, may be included in the sum to represent that no option is preferred, e.g., when all responses in the set of natural language responses are "bad."

[0068] Optionally, when training the response selection neural network 120 with both evidence-based and evidence-free training data, the training objective may include an auxiliary loss for classification tasks that involve matching class labels to whether the system's natural language responses are supported (by evidence) and plausible.

[0069] As one particular example, the response selection neural network 120, specifically the second language model neural network, is configured (trained) to generate preference scores on the Elo scale from a single linear head, and additional n classes It includes classifiers implemented with linear heads, which project from the context input and the final token embedding of the sequence (complete), i.e., the dialogue to which the response was added. Such a response selection neural network 120 can be trained using a combined training loss.

number

[0070] In an embodiment, the dialogue system 100 also includes a rule violation detection neural network 130, which may include, for example, a third trained language model neural network. The rule violation detection neural network 130 may be used by the dialogue system 100 to determine when a response violates one or more rules.

[0071] The rules may be any rules governing the characteristics of the natural language answer or the implications of the answer, for example, if the answer relates to an action to be taken. The rules may include, for example, requirements that the answer be relevant, stay on topic, or more specific requirements for any particular application, and / or the rules may include constraints, for example, that the answer avoid certain topics, content, assumptions, or actions.

[0072] In embodiments, each sample of a natural language response from the language generation neural network 110 may be processed using such a trained rule violation detection neural network 130 in conjunction with at least a portion of the contextual input to the language generation neural network 110 to determine, for each of a plurality of rules, a rule violation score that estimates the probability of violating the rule. Similarly, in some other embodiments, the rule violation score estimates the probability of following the rule. The probability of following the rule can be determined by determining the probability of violating the rule, since the two probabilities sum to one.

[0073] When used to train the language generation neural network 110, the rule violation score can be used as or to determine the second reward, as described below. The rule violation neural network 130 can then be described as a rule-reward model. For example, if the second reward rewards following a rule, then the second reward can be negative, or the second reward can have an inverse relationship with the rule violation score, such that as the rule violation score increases, the second reward decreases, or vice versa. For example, if the rule violation score determines the probability that a probability will be violated, then the second reward can be determined as the rule violation score minus 1.

[0074] More specifically, in embodiments, rule violation detection neural network 130, e.g., a third trained language generation neural network, is configured (trained) to process rule violation detection context inputs including a portion of the context input for language generation neural network 110, sample natural language responses from language generation neural network 110, and natural language expressions of rules (which may be concatenated with the context and responses) according to trainable rule violation neural network parameters. In some embodiments, the third trained language generation neural network also processes natural language requests to determine whether the rule was followed (or violated). The rule violation detection context inputs are processed using trained rule violation detection neural network 130 to estimate the probability that the rule is violated, or similarly, to determine a rule violation score that estimates the probability of following the rule, in other embodiments.

[0075] As one particular example, the rule violation detection context input (prompt) for rule violation neural network 130 may be formatted as follows: Below is a conversation between a person called the User and an AI computer system called Sparrow. [[Start conversation]] {snippet} [[End of conversation]] Question (yes / no): Did Sparrow follow rule "{rule}"? answer: where "{{snippet}" is part of the context input to the language generation neural network 110 and the natural language response from the language generation neural network 110, and "{rule}" defines a (natural) language rule. In this example, the third trained language generation neural network is intended to generate a natural language output, "yes" or "no."

[0076] Because some of the processing of the rule violation detection context input by the third trained language generation neural network can be shared between rules, this general type of template, in which the natural language representation of the rule is at or adjacent to the end of the rule violation detection context input, allows for the majority of rule violation detection context inputs to be shared between different rules, thus enabling computational optimization. For example, in inference, the computation involved in processing the rule violation detection context input can be shared for shared prefixes, i.e., for interaction and rule-form templates up to the first distinct token of "{rule}." As a result, the computational load scales only slightly with the number of rules.

[0077] As shown by example, the third trained language generation neural network may generate one or more natural language output tokens representing a decision as to whether or not a rule was followed (or violated), e.g., representing the word "yes" or "no," or other ways of saying this, and may have been trained, e.g., fine-tuned, accordingly.

[0078] The rule violation score may be determined from one or more output layer values ​​(corresponding to one or more natural language output tokens) used by the third trained language generation neural network to determine one or more natural language output tokens. For example, the rule violation score may be determined from a log-likelihood assigned to a sequence of one or more tokens representing a decision on whether or not a rule was followed, e.g., corresponding to "yes" or "no," in the first natural language output. More specifically, as an example, the rule violation score may be determined from a scalar value of the token representing whether or not a rule was violated, e.g., a logit value from the final linear layer of the language generation neural network, or from a combination or difference of such scalar values, e.g., one value of the token indicating that the rule was followed and another value of the token indicating that the rule was violated. As another example, the rule violation score may be determined from a vector value of the token representing whether or not a rule was violated, e.g., from an embedding of the token projected by the linear layer to a rule violation score.

[0079] In an embodiment, rule violation detection neural network 130 may be jointly trained with all rules. Such joint training may improve rule violation detection. Rule violation detection neural network 130, and in particular the third language model neural network, may be initialized with an instance of a trained language model neural network, such as, for example, with an instance of a trained Chinchilla model, which may then optionally be further trained, i.e., fine-tuned.

[0080] For example, a rule violation detection neural network, such as a third language generation neural network, may be trained using a supervised learning algorithm on a training dataset including multiple rule dialogue data items, each dialogue data item including a sequence of natural language statements representing a dialogue and a label, e.g., from a rating scale, indicating whether the dialogue complies with a particular rule.

[0081] In some embodiments, training data may be obtained from a human evaluator. For example, to obtain training data items including dialogue data items and labels, the human evaluator may assist or be given the opportunity to generate dialogue data items including rule violation detection context inputs (prompts) according to the above-described template, and the labels of the dialogue data items may be obtained from the human evaluator. As an example, the evaluator may provide labels according to a Likert scale of {definitely violate, probably violate, uncertain, probably comply, definitely comply}, which may then be binarized as violating and complying, with uncertain ratings being discarded. In some embodiments, for example, a human different from the evaluator, or another language generation neural network, may generate some conversational snippets of the training data items with the aim of having the language generation neural network 110 violate the rules (red teaming). This may involve the participation of humans or other language generation neural networks involved in a dialogue with the dialogue system 100.

[0082] The third language generation neural network may be trained to maximize the likelihood of correctly generating output token(s), e.g., "yes" or "no," that represent a decision on whether a rule was violated (or followed). For example, the training objective may be to maximize the likelihood of one or more tokens, e.g., "yes" or "no," in response to a label from a human evaluation, given a prompt with a dialogue and a rule, using, e.g., cross-entropy loss for classification. Training may be performed by backpropagating gradients of a classification objective function to update parameters of the rule violation detection neural network, e.g., a classification objective function based on cross-entropy loss.

[0083] As discussed further below, the specific rule(s) may depend on the application. However, as some general examples, rules may specify "stay to the point," "make sense," "relevant," "no great harm," "no stereotypes," and "no repetition." More specific rules may also be included, such as "no medical advice," "no opinions or emotions," or "no hate or harassment." In an embodiment, the rule violation detection neural network 130 learns to follow human judgment regarding whether or not to follow the rule(s). The rule violation detection neural network 130 may similarly be trained to detect when a rule is followed or violated.

[0084] In some implementations, the dialogue system 100 is implemented partially or fully on one or more remote servers and is accessed via a user computing device that provides natural language requests 102 to the system and receives natural language responses 104 from the system. Such a user computing device may be, for example, a mobile device such as a mobile phone or a smart speaker. The user computing device may provide a user interface to a search system via the dialogue system 100, allowing the user to access information encoded by the language generation neural network 110. The ability to access the search system may increase the reliability of the information provided to the user. The ability to access the search system also allows statements from the dialogue system 100 to be supported by evidence.

[0085] Such a user computing device may be equipped with an input mechanism that allows user input from a user in a natural language and an output mechanism that provides system output to the user in the natural language. The input / output mechanism may include, for example, a keyboard and a display. Alternatively, the input / output mechanism may include a voice-based mechanism. For example, the input mechanism may include a system configured to input audio data characterizing speech waveforms representing input from a user in the natural language and to convert the audio data into tokens representing speech in the natural language, e.g., representing a transcription of the spoken input. The output mechanism may include a system configured to receive tokens representing output to the user in the natural language and a system configured to convert the received tokens into audio data representing speech waveforms representing spoken words, i.e., representing output to the user in the natural language.

[0086] In some implementations, one or more of the first trained language generation neural network, the second trained language model neural network, and the third trained language generation neural network, for example, all of them, may be stored on a user computing device (i.e., local to the user). In implementations, the search system is remote from the user, and a search query is sent to the search system via a wired or wireless communication link between the user computing device and the search system, and one or more search results are received via the communication link. Such implementations of the system may be useful in improving the efficiency of using computational and communication resources, and may also provide improved user privacy because only limited information, such as the search query and response, needs to be sent over the communication link.

[0087] The above-described machine learning computer system for enabling a user to obtain information through interaction may use such a mechanism to enable a user to conduct an interaction with the machine learning system during or after training (in inference). Conducting such an interaction may include receiving, at the machine learning system, user input including a first request for information and providing, from the machine learning system, a first system output including a response to the first request for information. Conducting such an interaction may further include receiving, at the machine learning system, user input including a follow-up request for information, where the follow-up request for information is related to the first request for information, and providing, from the machine learning system, a second system output including the response to the follow-up request for information.

[0088] 2 is a flow diagram of a first exemplary process for conducting a dialogue using a dialogue system such as, for example, dialogue system 100. The process of FIG. 2 can be performed by one or more computer systems located at one or more locations. The steps of FIG. 2 do not have to be performed in the order shown. Some steps can be performed in parallel.

[0089] The process may involve determining an initial context input, i.e., a prompt (which may be a null input) (step 202), and then performing one or more of a number of dialogue update iterations, which may include dialogue rounds in which both the user and the agent "converse," i.e., generate natural language statements.

[0090] The dialogue update iteration may include receiving a natural language request from a user (step 204) and, in response, updating the context input to include the natural language request, such as a natural language representation of some or all of the text of the request. In an embodiment, the natural language request comprises an information request, i.e., a request for information, in particular a natural language question.

[0091] The dialogue update iteration may then include processing the (updated) context input using the first trained language generation neural network and according to parameters of the first trained language generation neural network to generate one or more sample first natural language responses (step 206). These responses may be referred to as uncorroborated in that they are generated without retrieval (without external search).

[0092] The dialog update iterations may also include generating one or more search queries from the natural language request (step 208), as described above. For each search query, one or more search results may be received from the search system interface. The search results may be partially or completely unstructured or structured, such as web page addresses or web page content or text.

[0093] A corroborated context input may be determined for each of the search results from the (updated) context input (step 210). The corroborated context input may include content, e.g., text, from one of the search results, e.g., in natural language.

[0094] The dialogue update iteration may include processing each corroborated context input using the first trained language generation neural network to generate one or more respective samples of second natural language responses, i.e., one or more for each corroborated context input (step 212).

[0095] The one or more samples of the first natural language response and the one or more samples of the second natural language response may be processed using the trained response selection neural network and according to parameters of the trained response selection neural network to select a natural language answer from the one or more samples of the first natural language response and the one or more samples of the second natural language response, for example, based on a preference score (step 214).

[0096] Processing the sample of natural language responses using the trained response selection neural network may then include, for each sample of natural language response, processing at least a portion of the context input and the sample of natural language response using a second trained language model neural network to generate a preference score for the sample of natural language response.

[0097] One of the sample natural language responses can then be selected to select a natural language answer based on the preference score for each sample natural language response. For example, each sample natural language response along with its respective context input / supported context input may be provided to the model sequentially, or the context input and sample may be provided together in parallel to determine which is preferred.

[0098] The natural language answer may provide at least a portion of the information requested by the information request; for example, the natural language answer may include an answer or a partial answer to the natural language question, or the information in the answer may include control information, for example, to control a mechanical system, such as a robot, or a computer system to perform a task.

[0099] The natural language answer may be provided to the user in response to the natural language user request (step 216). For example, the natural language answer may be made available at a user interface, such as by displaying the answer on a display or by converting the natural language answer into speech representing the answer and outputting the speech. In some implementations, the natural language answer may be used to control, for example, a mechanical system such as a robot, or a manufacturing plant, or equipment such as a heating or cooling device, or a computer system or network.

[0100] The search results, or subject matter drawn from the search results, such as, for example, portions of the corroborating contextual input, may also be provided to the user, for example, on a display or as audio (where such search results were used to determine the corroborating contextual input that was used to generate the natural language response that provided the answer). This may help a human user understand why a particular answer was provided, such as, for example, a particular answer to a question or a particular control signal for a mechanical or software system.

[0101] The process may then update the context input to include expressions of natural language answers for the next dialogue update iteration, which may involve a follow-up request from the user.

[0102] 3 shows an example user interface of a dialogue system that implements the above process. In this example, the context input to the language generation neural network 110 is on the left, where an example of the corroborated context input is shown, and an example user interface display is on the right.

[0103] In an embodiment, the dialogue system is configured to, among other things, respond to follow-up requests related to the natural language user request or answer by expanding the context input. This may include receiving a follow-up natural language request from the user and updating the context input to include (a representation of) the follow-up natural language request. The (updated) context input may then be processed using the first trained language generation neural network to generate one or more samples of a third natural language response.

[0104] One or more follow-up search queries can be generated from the follow-up natural language request and provided to the search system interface, and for each follow-up search query, one or more follow-up search results can be received. These can be used to determine a follow-up corroborated context input from the context input, and for each follow-up search result, for example, a follow-up context input including content from one of the follow-up search results, and the context input can be processed using a first trained language generation neural network to generate one or more respective samples of a fourth natural language response. The one or more samples of the third natural language response and the one or more samples of the fourth natural language response can then be processed using a trained response selection neural network to select a follow-up natural language answer to be provided to the user in response to the follow-up natural language request. The context input can then be updated again to include a representation of the follow-up natural language answer.

[0105] Figure 4 generally illustrates a second exemplary process for conducting a dialogue using a dialogue system, such as, for example, dialogue system 100. Figure 4 illustrates an exemplary implementation of the process of Figure 2 in which some of the steps may, but need not, be performed in parallel. The process generally illustrated in Figure 4 may be performed by one or more computer systems located at one or more locations. The implementation steps described below may be performed in conjunction with either the process of Figure 2 or Figure 4.

[0106] In an embodiment, the process generates a plurality of samples of first natural language responses 402 and a plurality of samples of second natural language responses and selects a natural language answer therefrom by, among other things, determining respective corroborated context inputs for each of a plurality of search results 404.

[0107] As mentioned above, in an embodiment, the dialogue system may use the trained rule violation detection neural network 130 to determine when a response violates one or more rules. For example, each sample of natural language responses may be processed using the trained rule violation detection neural network 130 in conjunction with at least a portion of the contextual input to determine, for each of a plurality of rules, a rule violation score that estimates the probability of violating (or following) the rule. The selection of one of the natural language responses for the answer (406) may then also be based on the respective rule violation scores of the rules for each sample.

[0108] As one example, any response that violates a rule, as detected, for example, by comparing the rule violation score to a threshold, may be omitted from the sample of natural language responses processed using the trained response selection neural network. As another example, for each sample of natural language responses, the rule violation score for each of the rules may be combined with the preference score of the answer and used to select an answer.

[0109] In principle, although undesirable, one may select an answer that violates one or more rules and / or select an answer according to a preference score that indicates that the natural language response is relatively unfavorable. Similarly, although also undesirable, one or more rules may be defined to select harmful answers, or a second trained language model neural network may be trained to generate a preference score that favors harmful answers. That is, in principle, the described technique is dual-purpose. Thus, the rules and preference scores should be selected to avoid such undesirable results.

[0110] For each sample of natural language responses, a combined rule violation score for the sample may be generated by, for example, combining the rule violation scores for each rule to determine a geometric mean of the rule violation scores. pr , and the combined rule violation scores of the samples are then used to generate a reranking score R rerank The preference scores may be combined to determine an average preference score AVG(R pr ), where relevance may be determined by conformance with formatting requirements.

[0111] As one specific example, the rerank score may be determined as follows:

number

number

[0112] One of the samples of natural language responses may then be selected based on the reranking score to select the natural language answer, such as selecting the response with the highest reranking score. In the example of Figure 4, eight samples are taken from the language model neural network 110 and the sample with the highest reranking score is selected.

[0113] Generally, preference scores are higher for responses that provide clear support, and rule violation scores penalize responses that violate the rules.

[0114] As an alternative to using re-ranking to decide whether to rely on evidence, the search / non-search choice can be made by calculating the log-likelihood of the roles “search query” and “agent” or “Sparrow” according to the context of the interaction (context input), such as by determining the score of the tokens “search query” and “agent” or “Sparrow.” The role with the higher log-likelihood is chosen to continue the interaction, thereby deciding whether to use the evidence retrieved from the search system interface 140.

[0115] In an embodiment, for each of the plurality of rules, the context input, the sample natural language responses, and at least a portion of the natural language expression of the rule (which may be linked to the context and response) are then processed using a trained rule violation detection neural network 130 to determine a rule violation score that estimates, for example, the probability of violating the rule.

[0116] There can be a significant computational burden in processing an interaction to determine whether any of a plurality of rules has been violated. Thus, in some implementations, the rule violation detection neural network can process the input up to the first distinct token (i.e., up to the start of the rule) and store the results of this calculation, so that only the representation of each rule needs to be processed next. Thus, in an embodiment, the dialogue system 110 can use a third trained language generation neural network to process at least a portion of the context input and the sample of natural language responses to determine a shared intermediate state of the third trained language generation neural network, and then, for each of the plurality of rules, can process the natural language representation of the rule using the third trained language generation neural network, starting from the shared intermediate state of the third trained language generation neural network.

[0117] Figure 5 is a flow diagram of a first exemplary process for training a dialogue system, such as dialogue system 100, to conduct a dialogue. The process of Figure 5 can be performed by one or more computer systems located at one or more locations. The steps of Figure 5 do not have to be performed in the order shown. Some steps can be performed in parallel.

[0118] In an embodiment, the dialogue system is trained, or more specifically, fine-tuned, so that the agent including the first language generation neural network 110 can be used by a user for interaction, such as, for example, to interactively obtain information between the user and the agent. The training can be performed by a training engine 150.

[0119] For an initial dialogue iteration, the training process first determines the context input for the current dialogue iteration (step 502).

[0120] In one or more dialogue update iterations, the process obtains natural language output statements from the action selection policy neural network, including the first language generation neural network 110, by generating natural language tokens for the natural language output statements at each successive time step until the end of the statement generation episode (step 504). The end of the statement generation episode may be indicated, for example, by the generation of one or more end-of-statement tokens, or the episode may end when a maximum-length output statement string has been generated.

[0121] In an embodiment, the action selection policy neural network is the first language generation neural network 110. The actions of the action selection policy neural network may be language actions, such as, for example, token selection actions. For example, the action selection policy neural network may be an autoregressive language generation neural network that generates tokens one at a time, and the generation of each successive token can be viewed as a selection of an action by the action selection policy neural network.

[0122] Generating a token at a time step can include processing the context input of the current dialogue iteration and the tokens previously generated at previous time steps in the episode (after the first time step) using an action selection policy neural network to select an action, where the action is to select the next token of the natural language output statement.

[0123] In an embodiment, at least a portion of the context input and the natural language output statement are processed using, for example, a response selection neural network 120, as described above, to determine a first reward for the natural language output statement (step 506).

[0124] In some implementations, at least a portion of the context input and the natural language output statement are processed using rule violation detection neural network 130, e.g., as described above, to determine a second reward for the natural language output statement (step 508). If the rule violation detection neural network determines a score that increases as the probability of a rule violation increases, the second reward may be negative. In some implementations, rule violation detection neural network 130 is not used and the second reward may be omitted. In some implementations, there may be one or more additional reward conditions, such as a (negative) condition that depends on the length of the output statement, for example, to promote conciseness.

[0125] As mentioned above, in embodiments, rule violation detection neural network 130 may be a rule-conditioned classifier neural network. At least a portion of the context input and the natural language output statement may be processed using the rule-conditioned classifier neural network, each conditioned on each of a plurality of rules, to determine multiple rule violation scores, one for each rule. Again, in some embodiments, the rule violation score of a rule may represent a probability that the portion of the context input and the natural language output statement violates the rule. Similarly, the rule violation score of a rule may represent a probability of following the rule.

[0126] The rule violation scores may be combined, for example, by determining a geometric mean of the scores to determine a second reward. As mentioned above, the rule violation detection neural network may include a third language generation neural network configured to process natural language rule statements representing the rules to generate one or more natural language output tokens representing a decision whether the rule was violated or not to determine a rule violation score.

[0127] In an embodiment, the process uses reinforcement learning techniques to train the action selection policy neural network, and thus the first language generation neural network 110, based on the first reward and the second reward (step 510). Any reinforcement learning technique may be used.

[0128] In an embodiment, an action selection policy neural network, such as the first language generation neural network 110, can be initialized with an instance of a trained language generation neural network, such as with an instance of a trained Chinchilla model, and then fine-tuned by a training process.

[0129] Training may be performed online or offline using previously stored data. The reinforcement learning technique may be a single-objective reinforcement learning technique, in which case the first and second rewards may be combined, e.g., by summing in a weighted sum, e.g., after normalizing each reward, or a multi-objective reinforcement learning technique may be used to optimize multiple, potentially interacting objectives.

[0130] In general, reinforcement learning techniques may train an action selection policy neural network, such as, for example, first language generation neural network 110, by iteratively backpropagating the gradient of a reinforcement learning objective function through the neural network, e.g., by iteratively adjusting neural network parameter values ​​of the action selection policy neural network of first language generation neural network 110. Again, this may use any suitable gradient descent optimization algorithm, such as, for example, Adam or other optimization algorithm.

[0131] Any suitable reinforcement learning objective function may be used. As just a few examples, the reinforcement learning objective function may rely on the (squared) Bellman error or may use a policy gradient based reward. In some implementations, the value of the reinforcement learning objective function is determined at the end of a statement generation episode (rather than at each time step).

[0132] For example, one or both of the first and second rewards may be zero at all steps except the end of the episode (thus, in an embodiment, the reward and the "return" are the same). In effect, each reward can be considered a reward for the episode, i.e., the complete natural language output statement, rather than a reward for each individual time step. Thus, in an embodiment, it is not necessary to use a response selection neural network and a rule violation detection neural network to process the sequence of tokens at each time step.

[0133] If the training is off-policy, for example, based on buffered trajectories as described below, optional off-policy corrections may be made. As mentioned above, in an embodiment, the first language generation neural network is pre-trained and fine-tuned using reinforcement learning techniques. A regularization term may then be included to keep the distribution of actions close to that of the initial pre-trained language generation neural network.

[0134] In general, the objective of reinforcement learning may be any objective that aims to maximize a reward, such as a primary, secondary, or combination thereof. As just one example, the reinforcement objective may be a reward R given by: agent maximizing (s|c),

number

number

number

[0135] As just one example, a reinforcement learning technique may be an actor-critic technique, such as the Advantage Actor-Critic technique (Minh et al., "Asychronous Methods for Deep Reinforcement Learning," arXiv:1602.01783). A critic neural network, such as an additional MLP (multilayer perceptron) head on an action selection policy neural network, may then be used to generate a scalar value estimate representing an estimate of the return (reward) from selecting future tokens according to the current values ​​of the action selection policy neural network parameters. As a result, the reinforcement learning objective function may depend on that value estimate. As another example, the REINFORCE algorithm with a baseline may be used (e.g., Sutton and Barto, "Reinforcement Learning: An Introduction," 2018).

[0136] As mentioned above, in some implementations, the first trained language generation neural network, the second trained language model neural network, and the third trained language generation neural network may each include a respective sequence-to-sequence neural network, e.g., a transformer, configured to receive an input sequence of tokens and process the input sequence of natural language tokens according to a respective set of neural network parameters to generate an output sequence of natural language tokens. In some implementations, to reduce computational load, one or more of the first trained language generation neural network, the second trained language model neural network, and the third trained language generation neural network include a set of shared input layers. That is, these neural networks may share a majority of their respective neural network parameters and include separately trained (multi-layer) "heads."

[0137] 6 shows an exemplary implementation of the neural network described above, using a shared language model 600, e.g., the Chinchilla model. The shared language model 600 includes a set of shared, pre-trained transformer layers 602, e.g., the bottom 80% of the transformer layers of a trained Chinchilla model, and a set of separate heads. During training, the shared transformer layers 602 are frozen (i.e., the learnable parameters of the layers are held constant), and only the heads are trained. This reduces memory usage.

[0138] The shared language model 600 includes an instance of an initial pre-trained language generation neural network 610, such as an instance of the trained Chinchilla model, along with frozen layers. One head is provided to an action selection policy neural network 612, which becomes the trained language generation neural network 110. During training of the action selection policy neural network 612, it may include the KL (Killer-Lee) regularization term described above to keep the distribution of token selection actions close to that of the initial pre-trained language generation neural network 610, sometimes referred to as the "teacher" neural network.

[0139] In the illustrated example, the shared language model 600 is configured for use in actor-critic reinforcement learning techniques and includes a value function head 614 to provide the value estimates described above. As an example, the value head 604 may include an MLP (multi-layer perceptron) that receives as input the final transformer layer representation of the action selection policy neural network 612 at each time step.

[0140] In the illustrated example, the shared language model 600 includes two heads for the response selection neural network 120: one trained only with evidence-free training data and one trained with mixed training data. In other implementations, there may be only one response selection neural network 120 and head. In the illustrated example, the shared language model 600 also includes a head for the rule violation detection neural network 130.

[0141] In an embodiment, the prompt is added to the context input of the current dialogue iteration to define the role of the action selection policy neural network when generating tokens for the natural language output statement, as described above. In an embodiment, the role is one or more of a user role (i.e., generating a statement and playing the role of a user in the dialogue), an agent role (i.e., generating a statement and playing the role of an agent in the dialogue), and a search query generation role (i.e., generating a statement that can be used as a search query as described above to retrieve one or more search results).

[0142] Some implementations of the dialogue system 100 learn by “self-play.” The context input may be updated to include representations of natural language output statements for the next dialogue update iteration, and prompts may be added to the updated context input to define the role of the action selection policy neural network in the next dialogue update iteration. The role of the action selection policy neural network in the next dialogue update iteration is typically different from the role of the action selection policy neural network in the current dialogue update iteration. For example, after generating an output statement for the user role, a statement may be generated for the agent or search query role; after generating an output statement for the search query role, a statement may be generated for the agent role; and after generating an output statement for the agent role, a statement may be generated for the user role. Such an approach enables the system to learn by “self-play,” i.e., by “talking” to itself, i.e., by conducting a dialogue with itself.

[0143] When configured to conduct training by self-play, the process does not need to train the action selection policy neural network using the second reward when the natural language output statement is for the user role or the search query generation role, i.e., in an embodiment, the second reward is not used for training when the role is the user role or the search query generation role (but is still trained with the first reward from the response selection neural network 120).

[0144] As described above, during training of an action selection policy neural network, such as the first language generation neural network 110, different versions of the response selection neural network 120 may be used depending on whether the context input includes supporting evidence. Thus, the system may use a version of the response selection neural network 120 trained to process context input without supporting evidence for the natural language output statement to determine the first reward when the role is the user role. The system may use a version of the response selection neural network trained to process context input with and without supporting evidence for the natural language output statement to determine the first reward when the role is the search query generation role. The system may use (e.g., by combining, etc., processing) a version of the response selection neural network trained to process context input with and without supporting evidence for the natural language output statement, as well as a version of the response selection neural network trained to process context input without supporting evidence for the natural language output statement, to determine the first reward when the role is the agent role. However, the estimation may use a version of a response selection neural network trained to process contextual inputs with and without supporting evidence for the natural language output statement.

[0145] Figure 7 generally illustrates a second exemplary process for training a dialogue system, such as dialogue system 100, to conduct a dialogue. The process of Figure 7 may be performed by one or more computer systems located at one or more locations. Figure 7 illustrates an exemplary implementation of the process of Figure 6. The implementation steps described below may be performed in conjunction with either the process of Figure 6 or Figure 7. The process generally illustrated in Figure 7 may be divided among components differently than in the illustrated example.

[0146] 7, the reinforcement learning environment 700 includes a dialogue buffer 702 (i.e., memory) configured to store multiple trajectories. In this example, a trajectory includes a context input for a current dialogue iteration, a natural language output statement, and (optionally) a first reward and a second reward. At the start of a dialogue, the dialogue buffer may optionally be initialized with one or more sequences of initial dialogues.

[0147] In the illustrated example, the reinforcement learning environment 700 also includes a reward model 704, i.e., the response selection neural network 120 and / or the rule violation detection neural network 130, that generates rewards as described above.

[0148] The action selection policy neural network, i.e., language generation neural network 110, can be trained on the stored trajectories using the reinforcement learning techniques described above. For example, the learnable parameters of the action selection policy neural network, i.e., language generation neural network 110, can be updated by a learner 706 using reinforcement learning techniques. The learner 706 can be implemented by the training engine 150.

[0149] In some implementations, a trajectory is stored in the interaction buffer on the condition that the trajectory's reward value is greater than a minimum reward threshold. The trajectory's reward value can be determined, for example, from one or both of the first reward and the second reward in the trajectory. Storing a trajectory may also be conditional on the trajectory having a valid format.

[0150] If the natural language output statement included in the trajectory is a search query statement including a search query for querying a search system, the trajectory may include the corresponding search results. This may involve providing the search query to search system interface 140 for querying the search system, receiving one or more search results from search system interface 140 in response to the search query, and including the search query and data from one or more of the search results in the trajectory stored in interaction buffer 702. Determining context input for a dialogue iteration, such as an initial or current dialogue iteration, may then include retrieving data for the context input, including the search query and data from one or more of the search results, from the stored trajectory.

[0151] As mentioned above, some implementations of the system / method can learn through self-play. This may include obtaining, for one or more rounds of agent-user interaction, natural language output statements for the agent in an agent interaction update iteration (i.e., where the output is for the agent's role), and obtaining natural language output statements for the user in a user interaction update iteration (i.e., where the output is for the user's role) that follows the agent interaction update iteration in response to the natural language output statements for the agent. An action selection policy neural network can then be trained based on the first reward and the second reward for the agent interaction update iteration, and based on the first reward but not the second reward for the user interaction update iteration.

[0152] Some implementations of the method / system use “red teaming” to improve the training process. Thus, in embodiments, determining context inputs for a dialogue iteration, particularly for an initial dialogue iteration, involves generating natural language requests using a fourth “red team” trained natural language generation neural network. Broadly, the “red team” natural language generation neural network is trained to generate language that prevents a dialogue system including the first, second, and third natural language generation neural networks from generating acceptable responses. More specifically, the fourth trained natural language generation neural network is trained, e.g., fine-tuned, to generate red team natural language statements, such as requests, that, when processed by the first or other language generation neural networks, particularly in combination with context inputs, cause the first or other language generation neural networks to generate natural language output statements that (in combination with the context inputs) violate one or more rules implemented by the rule violation detection neural network. One or more of the red team natural language statements may then be included in the context input for a dialogue iteration, such as an initial dialogue iteration.

[0153] FIG. 8 illustrates the performance of an exemplary implementation of the dialogue system 100. In FIG. 8, the y-axis illustrates the relative preference rate of natural language output from the dialogue system in a three-way comparison among other dialogue systems, and the x-axis illustrates the rule violation rate of the dialogue system under adversarial probing. Points 800 and 802 are for an implementation of the dialogue system described herein with eight samples and two samples, respectively, drawn from the language model neural network 110 (see the description of FIG. 4 ), while the other points are for other systems. It can be seen that an implementation of the dialogue system 100 can generate language responses that are not only preferred by human users but also have a low rule violation rate.

[0154] The first trained language generation neural network 110 may have been trained or fine-tuned with a corpus of language associated with the operation of a controller configured to control actions in a real-world environment to perform a task. For example, the first language generation neural network 110 may be initialized with an instance of the language generation neural network thus trained and then fine-tuned as described above. The controller may control the actions of a mechanical system, sometimes referred to as a mechanical agent, such as a robot or vehicle, or the manufacturing actions of a manufacturing plant, and the search system may be configured to return search results related to the operation of the controller. Implementations of the dialogue system 100 may be used, for example, to provide an intuitive user interface for examining the operation of the controller, for example, for fault finding or other purposes.

[0155] Generally, for example, in the examples below, one or more of the initial context inputs or natural language requests received by the dialogue system 100 from the user may include or consist of, for example, one or more sequences of letters or numbers that describe or encode observations of a real-world environment. Generally, for example, in the examples below, one or more of the natural language answers (and therefore also the natural language responses discussed above) provided to the user from the dialogue system 100 may include or consist of, for example, one or more sequences of letters or numbers that describe or encode actions to be performed in the real-world environment, such as, for example, structured natural language or computer code.

[0156] In some implementations, the context input, e.g., a natural language request, includes one or more natural language statements related to an environment, particularly a real-world environment, and includes a natural language request related to the environment. That is, the initial context input may include one or more natural language statements related to the environment and may therefore be updated to include a natural language request that may request information related to the environment. Similarly, the natural language answer or natural language output statement may also provide information related to the environment, e.g., in some implementations, related to or specifying an action to be taken in the environment. As an example, the natural language request may specify a goal to be achieved and, optionally, characteristics of the real-world environment, and the natural language answer may specify one or more actions to be taken to achieve the goal. The rules may be general, as in the example above, or may be specific to an environment or goal, e.g., specifying one or more constraints on actions to be taken to achieve the goal.

[0157] In some implementations, the environment is a real-world environment, and the method (or corresponding system) is used to diagnose a fault in a mechanical system operating in the real-world environment. Next, acquiring context input, e.g., initial context input, may include acquiring one or more observations of the mechanical system (here, including observations of the mechanical system's operation), e.g., from one or more sensors described below. These may be processed, e.g., as described below, to generate natural language expressions of the one or more observations used to provide one or more natural language statements of the context information. In these implementations, the natural language request may relate to the operation of the mechanical system, and the natural language answer or natural language output statement is used to identify a fault in the mechanical system. For example, the request may include a general question such as "Is the system operating correctly?" or "What's wrong with the system?" or a specific request such as "Is component X faulty?" The answer may provide a natural language response to the request. Optionally, the one or more rules may specify constraints on the mechanical system or possible answers, e.g., the first (trained) language generation neural network. The ability to interact with an agent including the first (trained) language generation neural network facilitates targeting a specific fault diagnosis. Diagnosis can use information stored within a trained language generation neural network and, if necessary, the ability to search external data stores to supplement the stored information with more comprehensive or more up-to-date information.

[0158] As another example, the environment may be a computer security monitoring environment, e.g., the system may be deployed as part of a system that monitors the security of one or more computers. For example, the environment may be a computer network security monitoring environment, e.g., the system may be deployed as part of a system that monitors the security of one or more computers on a computer network, e.g., a wireless network, a cellular network, a local area network, and / or the Internet. As another example, the environment may alternatively or additionally be a computer system security monitoring environment, e.g., the system may be deployed as part of a system that monitors a system for the presence of computer viruses and / or unresolved software vulnerabilities, e.g., zero-day attacks. Software vulnerabilities may be resolved by updating (e.g., applying patches) and / or removing (e.g., uninstalling) software from the computer system. In these examples, the natural language request may query whether a computer security incident has been resolved (e.g., "Is the incident resolved?"). The context input and / or natural language request may include relevant statements from system logs, i.e., statements potentially related to the event being queried. The initial context input may define characteristics of the computer system and / or software environment. A computer security incident may be, for example, a data breach, an unauthorized login or other access to a protected system, the detection of a computer virus, or the detection of a software vulnerability. An incident may be "resolved" when the underlying incident no longer poses a threat to the security of the computer system, such as when a computer virus is eliminated, access to a protected system is removed, a data breach is mitigated, or vulnerable software is updated or removed.The system can use the context input to generate a response to the request that includes a natural language statement indicating whether the incident has been resolved and, optionally, displaying the evidence used to determine this. The rule(s) may specify constraints on the computer system and / or software environment or constraints on the possible answers.

[0159] For example, context input, such as initial context input, may include one or more of: a code snippet from the software code, a system log, a program log, or other artifact that should be left on the computer by executing the program, or a validation rule that expresses requirements for the execution of the software program, or a natural language statement that describes the computer system on which the software executes. In general, context input may include relevant statements, i.e., statements that are potentially relevant to the event being queried.

[0160] In some implementations, obtaining context input, such as initial context input, may include obtaining one or more observations of a computer (including computers on the network) from a system log, data characterizing the computer network, or both, or from other data described above, and processing the one or more observations to generate natural language representations of the one or more observations. The natural language requests may relate to a computer security incident or the secure operation of the computer network. The process implemented by the dialogue system 100 may include using the natural language representations of the one or more observations to provide one or more natural language statements of the context information, and using the natural language answers or natural language output statements to identify the security status of the computer network or a computer network security flaw.

[0161] As another example, the environment may be a software testing or evaluation environment; for example, the system may be deployed as part of a system that tests software before deployment or that evaluates already deployed software to identify bugs. In these examples, when the system tests the software before deployment, the natural language request may ask whether the software performs as intended, and the context input, e.g., the initial context input, may include code snippets from the software code and, optionally, natural language statements describing the computer system on which the software executes. The system may then use the context input to generate an answer indicating whether the code executes as intended and, optionally, display evidence used to determine this. When the system monitors the execution of the code after deployment, the natural language request may ask whether the software program or portion of the software program executed as intended, and the context input may include one or more of code snippets from the software code, system logs, program logs, or other artifacts that should be left on the computer by executing the program, or validation rules that express requirements for the execution of the software program, or natural language statements describing the computer system on which the software executes. The system can then use the context input to generate an answer indicating whether the code executed as intended, and optionally display the evidence used to determine this. As a specific example, the software program may be part of the startup of a computer, and the system may generate an answer each time the computer boots up to verify whether the computer functions correctly after booting. Again, the rule(s) may specify constraints on the computer system and / or software environment, or constraints on the possible answers.

[0162] As another example, the environment may be an educational environment, e.g., the system may be deployed as part of an educational software program that assists a user in learning or practicing one or more corresponding skills. In these examples, the contextual input may include natural language statements that describe or refer to a scenario or scene in the real-world or virtual environment, and the request may be a question about the scenario or scene.

[0163] As another example, the environment may be an information retrieval environment; for example, the system may be deployed as part of a search engine or other software that allows a user to search for information within a corpus of documents, such as the Internet or another electronic document corpus. In these examples, the request may be a question in any suitable natural language, and the answer may optionally include evidence, such as including relevant statements from the corpus of documents, such as identified by searching the corpus using conventional information retrieval techniques.

[0164] In some further applications, the method or corresponding system is used for natural language control of tasks in a real-world environment, i.e., a natural language request may relate to a task, e.g., the natural language request may include a request to perform a task, and the answer, i.e., information provided by the method / system in the answer, may be used, e.g., to control a mechanical or computer system (sometimes called a mechanical agent) to perform the task.

[0165] As an example, a natural language request may include a high-level request, e.g., from a human, to perform a task, such as, "How do I put an empty bottle in the bin?", "Please bring me a glass of water," or "Can you put the vacuum cleaner in the cupboard?" The or each natural language answer may define one or more steps of the task, which may then be interpreted by a machine system, or more specifically, a control system of the machine system, to perform the steps of the task. For example, such a control system may translate the natural language answer into a series of primitive actions to be performed by the machine system to perform the steps of the task.

[0166] Thus, in some embodiments, the method or corresponding system is used to control a mechanical system or mechanical agent operating in a real-world environment to perform a task. The mechanical system may be, for example, a robot or an autonomous or semi-autonomous vehicle. For example, determining a context input, such as an initial context input, may include obtaining one or more observations of the real-world environment from one or more sensors and processing the one or more observations to generate natural language representations of the one or more observations. By way of example only, image capture models (including video caption models here) may be used to process images in this manner. Other models may be trained to perform corresponding tasks on types of sensor / sensor data other than images. The natural language representations of the one or more observations may be used to provide one or more natural language statements for the initial context input. The natural language request may be related to an action to be performed by the mechanical system. The natural language answer (or natural language output statement) may be used to control the mechanical system in the real-world environment. For example, the answer may define an action for controlling the movement or navigation of a robot or vehicle in the real-world environment. Such actions may be, for example, high-level actions or "skills" that can be translated into one or more lower-level or "basic" actions by a trained neural network.

[0167] In some embodiments, the mechanical system has a control system for controlling actions of the mechanical system. Receiving the natural language request may include receiving a control signal from the control system and generating the natural language request from the control signal. For example, there may be an interaction between an agent including a first (trained) language generation neural network and the mechanical system and / or a user (e.g., if the control system of the mechanical system has a human user interface). The aforementioned rules or preferences (preference scores) may impose constraints on actions to be taken, for example, for safety or other reasons.

[0168] A mechanical system, hereinafter also referred to as a mechanical agent, may include one or more sensors that capture observations of the environment, e.g., at specified time intervals, as the mechanical agent navigates through or attempts to perform tasks in the environment.

[0169] For example, the observations may include one or more of images, object position data, and sensor data to capture the observations as the mechanical agent interacts with the environment, such as, for example, sensor data from image sensors, distance sensors, or position sensors or from actuators. For example, in the case of a robot, the observations may include data characterizing the current state of the robot (e.g., one or more of joint positions, joint velocities, joint forces, torques, or accelerations, e.g., gravity-compensated torque feedback, and global or relative pose of an item held by the robot). In the case of a robot or other mechanical agent or vehicle, the observations may similarly include one or more of positions, linear or angular velocities, forces, torques, or accelerations, and global or relative pose of one or more parts of the mechanical agent. The observations may be defined in one, two, or three dimensions and may be absolute and / or relative observations. The observations may also include sensed electronic signals, such as motor current or temperature signals, and / or image data or video data, for example from a camera or LIDAR sensor, for example, data from a sensor on the mechanical agent or from a sensor located separately from the mechanical agent in the environment.

[0170] The mechanical agent may be associated with a control system that uses the observations generated by the sensors to generate control signals for controlling the mechanical agent. In particular, the control system may first determine an appropriate action for the mechanical agent to perform, e.g., as part of performing a specified task, such as navigating to a particular location, identifying a particular object, moving a particular object to a predetermined location, manipulating a particular object in some way, etc., and then generate a control signal that causes the mechanical agent to perform the action.

[0171] Such a control system may be deployed attached to the mechanical agent or may be deployed remotely from the mechanical agent and may transmit control signals to the mechanical agent via a data communications network.

[0172] The control signal may be a control input for controlling the mechanical agent. For example, if the mechanical agent is a robot, the control signal may be, for example, a torque for a joint of the robot or a higher-level control command. As another example, if the mechanical agent is an autonomous or semi-autonomous land, air, or sea vehicle, the control signal may include actions that control navigation, for example, steering, and movement of the vehicle, for example, braking and / or acceleration of the vehicle. For example, the control signal may be, for example, a torque to a control surface or other control element, for example, a steering control element of the vehicle, or a higher-level control command.

[0173] In other words, the control signals may include, for example, position, velocity, or force / torque / acceleration data of one or more joints of a robot or part of another mechanical agent (system).

[0174] In these examples, similar to the control system, software for implementing the methods described herein (“system software”) may be deployed attached to the mechanical agent or may be deployed remotely from the mechanical agent.

[0175] In these implementations, the system software can be used to provide an additional control layer above the control system, where the system software or other components can determine requests based on information received by the control system. For example, the requests can be determined by receiving control signals from a mechanical agent control system and then determining from the control signals, for example, one or more natural language requests associated with the mechanical agents in the environment. The responses can be used to control the mechanical system.

[0176] In some implementations, the mechanical agent control system is an autonomous or semi-autonomous control system that autonomously or semi-autonomously controls the navigation or other actions of a mechanical agent, such as a vehicle, etc. Additionally or alternatively, the mechanical agent control system may have an interface for receiving control commands, for example, from a human operator.

[0177] In these applications, the described system software can be used to provide an additional layer of control, for example, for safety purposes. For example, the described system software can be used to constrain control of a mechanical agent in a manner that would be dangerous or contrary to one or more rules (as defined by preference scores) or preferences. Rules related to control of a mechanical agent can be explicitly entered, for example, as natural language statements. As an example, such rules or preferences (preference scores) might include rules / preferences related to permitted movements of a vehicle, such as traffic rules, or rules / preferences related to permitted (or prohibited) or preferred rules related to safe movements or types of tasks. Such rules / preferences can include rules / preferences related to decisions made to ensure safe behavior of a mechanical agent, such as to constrain damage to the mechanical agent or to humans.

[0178] Thus, each request may relate to an action to be performed by the mechanical agent, e.g., an action being considered by a control system. For example, a request may define an action to be performed by the mechanical agent in the form of a question such as "Should I turn left?" or "Is it safe for the agent to turn left?" As another example, a request may ask what action to be performed by the mechanical agent, e.g., "In which direction should the mechanical agent turn?" In the case of a robot, a request may relate to a set of subtasks that should be performed to perform a task, e.g., "What to do next?" or "Should I pick up object X?" The subtasks themselves may include a set of elementary actions to move parts of the robot, e.g., to release a gripper.

[0179] In general, a request may include a natural language description that defines the information that a response from the system software should provide, i.e., the request may explicitly or implicitly determine what is needed from the response.

[0180] A response to a request may be used to control a mechanical agent in a real-world environment. More specifically, the response may be used to control an action performed by the mechanical agent. As one example, the response may suppress an action that would otherwise be performed. That is, the response may determine whether or not the action defined by the request is performed. As another example, the response may define an action to be performed, for example, if the request implicitly or explicitly requires that the action be determined.

[0181] In such implementations, obtaining contextual input may include obtaining one or more observations of a real-world environment, which includes the mechanical agent and therefore potentially includes one or more observations of the mechanical agent. The observations may be obtained from one or more sensors that may, but need not, be sensors of the mechanical agent. As noted above, the observations may include still or video images and / or other sensor data from one or more sensors that sense conditions of the environment or the mechanical agent. As used herein, "image" includes a LIDAR point cloud.

[0182] The one or more observations are processed, for example, by a first machine learning model to generate natural language representations of the one or more observations, i.e., to generate natural language text that describes the observations included in the context input.

[0183] There are many different types of machine learning models that can be used to achieve this. For example, so-called visual language models are typically configured to describe images or videos using natural language, such as to perform image or video capture tasks. More generally, such models can perform many different types of image processing tasks by formulating the task as a text generation problem, such as to detect or classify objects in an image or video. Correspondingly, other machine learning models can be trained to generate natural language text that describes data from other types of sensors, such as to represent physical positions or forces as natural language statements that describe a mechanical agent or environment or part thereof. A natural language representation of one or more observations is used to provide one or more of the natural language statements for the context input.

[0184] The rules or preferences (preference scores) may be related to the current position of the mechanical agent in the environment and may not be generated directly from observations generated by sensors.

[0185] By way of example only, rules or preferences (preference scores) may include common knowledge rules, driving rule manuals, or may be manually designed or obtained from a knowledge graph or the internet. Examples of such rules or preferences include "if the car's electrical system fails, the car is unsafe to drive," "the speed limit at the current location is 30 mph," "right turns are permitted after stopping at this red light," "turning across double yellow lines is prohibited," etc. System software may be used to constrain the outcomes of potential actions considered by the control system before they are sent as control signals to mechanical agents.

[0186] In some other embodiments, the environment is a real-world environment that includes a manufacturing plant, such as a manufacturing plant for producing a product, e.g., a chemical, biological, or mechanical product, or food. As used herein, "manufacturing" a product also includes purifying starting materials to create the product or processing starting materials, e.g., to remove contaminants and produce a cleaned or reclaimed product. A manufacturing plant may include multiple manufacturing units, such as vessels for chemicals or biological substances or machines for processing solids or other materials. The manufacturing units are configured such that intermediate versions or components of a product can be moved between the manufacturing units during the production of the product, e.g., via pipes or mechanical transport. In embodiments, a system is used to control one or more of the manufacturing units or to control the movement of intermediate versions or components of a product between the manufacturing units.

[0187] Thus, in these embodiments, obtaining the context input may then include obtaining one or more observations of the manufacturing unit or movement from one or more sensors. The sensors may include any type of sensor that monitors the manufacturing unit or movement, such as a sensor configured to sense, for example, mechanical movement or force; electrical conditions such as pressure, temperature, current, voltage, frequency, impedance, etc.; quantity, level, flow rate / movement rate, or flow path / movement path of one or more materials; physical or chemical conditions such as, for example, physical state, shape, or configuration, or chemical state such as pH; the mechanical configuration of the unit or the configuration of the unit such as the configuration of a valve; an image or video sensor for capturing image or video observations of the manufacturing unit or movement; or any other suitable type of sensor. In embodiments, the one or more observations are processed to generate natural language representations of the one or more observations, for example, as described above. The natural language representations of the one or more observations are used to provide one or more natural language statements of the context information.

[0188] The request may relate to an action that controls one or more operations or controls the movement of a production unit. A response to the request is used to control one or more operations or controls the movement of a production unit. For example, the response to the request may be used to control production, e.g., to minimize the use of energy or other resources or to obtain a desired quality or characteristic of a product. For example, the action may include an action that controls equipment in a plant, e.g., to adjust or turn on / off equipment or a production process, or an action that changes a setting that affects a production unit or the movement of a product or intermediate or component thereof.

[0189] In some embodiments, the manufacturing plant has a plant control system for controlling manufacturing units or for controlling movements. The request may be generated, for example, in response to receiving control signals from the plant control system and generating a natural language request from the control signals. In a manner similar to that described above, the plant control system may be autonomous, semi-autonomous, or human-controlled.

[0190] In a manner similar to that described above, the system may implement rules or preferences, for example, to control or limit the allocation of energy or other resources, or to ensure target quality or characteristics of a product, or to constrain the operation of a plant, such as, for example, a manufacturing unit, within safe limits.

[0191] In some implementations, the environment is a real-world environment of a service facility including multiple pieces of equipment, such as electrical components and electrical equipment, such as a server farm or data center, e.g., a telecommunications data center or a computer data center or service facility for storing or processing data. The service facility may also include environmental control equipment, such as temperature controls, e.g., chillers, or auxiliary controls that control the operating environment of the equipment, such as airflow controls or air conditioning equipment. Obtaining the context input may then include obtaining observations of the state of the environment, which may include any electronic signals representative of the function of the facility or the equipment within the facility. For example, the representation of the state of the environment may be derived from observations made by any sensors sensing the state of the facility's physical environment, or from observations made by any sensors sensing the state of one or more pieces of equipment or one or more auxiliary controls. These include sensors configured to sense electrical conditions, such as current, voltage, power, or energy; facility temperature; fluid flow, temperature, or pressure within the facility or within the facility's cooling system; or physical facility configuration, such as whether a ventilation system is open. These may be processed, for example, as described above, to generate natural language representations of one or more observations that are used to provide one or more natural language statements of the context information. The request may relate to the operation of a facility to minimize resource usage, such as adjusting the operation of one or more pieces of equipment (electrical components) to, for example, control power or water usage. For example, the request may ask which components to turn on to reduce resource usage, or whether it is safe to turn a given component on or off. The system may then determine how to operate the equipment based on the generated response, for example, by turning on or off one or more components indicated by the answer.

[0192] In some implementations, the environment is a real-world environment of a power generation facility, e.g., a renewable power generation facility such as a solar or wind farm, and the requirement may relate to, e.g., how to control the delivery of power to a power grid to control power generated by the facility, e.g., to meet demand or reduce the risk of mismatch between elements of the power grid, or to maximize the power generated by the facility.

[0193] In general, environmental state observations for context input may include any electronic signals representative of the electrical or mechanical function of power generation equipment within the power generation facility. For example, the representation of the environmental state may be derived from observations made by any sensors sensing the physical or electrical state of equipment within the power generation facility generating electrical power, or the physical environment of such equipment, or the state of auxiliary equipment supporting the power generation equipment. Such sensors may include sensors configured to sense the electrical state of the equipment, such as current, voltage, power, or energy; the temperature or cooling of the physical environment; a fluid, e.g., air or water, flow rate; or the physical configuration of the equipment; and observations of the electrical state of the power grid, e.g., from local or remote sensors. The environmental observations may also include one or more predictions regarding the future state of operation of the power generation equipment, such as a prediction of future wind levels or solar irradiance, or a prediction of future electrical conditions of the power grid.

[0194] The term "configured" is used herein in connection with systems and computer program components. To say that one or more computer systems are configured to perform a particular operation or action means that the system has installed thereon software, firmware, hardware, or a combination thereof that, when running, causes the system to perform the operation or action. To say that one or more computer programs are configured to perform a particular operation or action means that the one or more programs contain instructions that, when executed by a data processing apparatus, cause the apparatus to perform the operation or action. Embodiments of the subject matter and functional operations described herein can be implemented in digital electronic circuitry, in tangibly embodied computer software or firmware, in computer hardware containing the structures disclosed herein and structural equivalents thereof, or in one or more combinations thereof. Embodiments of the subject matter described herein can be implemented as one or more modules of computer program instructions, i.e., as one or more computer programs encoded on a tangible, non-transitory storage medium for execution by or controlling the operation of a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random-access or serial-access memory device, or one or more combinations thereof. Alternatively, or additionally, the program instructions may be encoded into an artificially generated propagated signal, such as a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to a receiving device suitable for execution by a data processing device.

[0195] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. An apparatus may also be or include special-purpose logic circuitry, such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, an apparatus may also optionally include code that creates an execution environment for a computer program, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0196] A computer program (which may also be called or described as a program, software, software application, app, module, software module, script, or code) can be written in any form of programming language, including compiled or interpreted, or declarative or procedural, and can be deployed in any form, such as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program, or in multiple cooperating files, e.g., files that store one or more modules, subprograms, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0197] As used herein, the term "database" is used broadly to refer to any collection of data. The data need not be structured in any particular way, or even at all, and may be stored on storage devices in one or more locations. Thus, for example, an index database may contain multiple collections of data, each of which may be organized and accessed in a different way.

[0198] Similarly, the term "engine" is used broadly herein to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine is implemented as one or more software modules or components and installed on one or more computers in one or more locations. In some cases, one or more computers are dedicated to a particular engine, while in other cases, multiple engines may be installed and run on the same computer or computers.

[0199] The processes and logic flows described herein may be implemented by one or more programmable computers executing one or more computer programs to perform functions by performing operations on input data and generating output. The processes and logic flows may also be performed by special purpose logic circuitry, such as an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0200] A computer suitable for executing a computer program can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from a read-only memory, a random-access memory, or both. The basic components of a computer are a central processing unit for executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by, or incorporated in, special-purpose logic circuitry. Generally, a computer also includes one or more mass storage devices, e.g., magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operatively coupled to receive data from or transmit data to them, or both. However, a computer need not have such devices. Furthermore, a computer can be incorporated in other devices, e.g., a mobile phone, a personal digital assistant (PDA), a portable audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name a few.

[0201] Computer-readable media suitable for storing computer program instructions and data include all types of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0202] To interact with a user, embodiments of the subject matter described herein can be implemented in a computer that has a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, e.g., a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user. For example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, spoken, or tactile input. Furthermore, a computer may interact with a user by sending documents to a device used by the user and receiving documents from a device used by the user (e.g., by sending a web page to a web browser on the user's device in response to a request received from the web browser). A computer may also interact with a user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and receiving reply messages from the user.

[0203] A data processing device for implementing machine learning models may also include, for example, dedicated hardware accelerator units for handling the general and numerically intensive portions (i.e., inference, workloads) of machine learning training or production.

[0204] The machine learning model can be implemented and deployed using a machine learning framework, such as the TensorFlow framework.

[0205] Embodiments of the subject matter described herein can be implemented in a computing system that includes a back-end component, e.g., a data server, or a middleware component, e.g., an application server, or a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an embodiment of the subject matter described herein, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communications network. Examples of communications networks include a local area network (LAN), a wide area network (WAN), e.g., the Internet.

[0206] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server sends data (e.g., HTML pages) to a user device, for example, for the purpose of displaying the data to and receiving user input from a user interacting with the device acting as a client. Data generated at the user device (e.g., results of user interaction) can be received from the device by the server.

[0207] While the specification contains many specific implementation details, these should not be construed as limiting the scope of any invention or the scope of patentable subject matter, but rather as descriptions of features that may be unique to particular embodiments of a particular invention. Certain features described herein as separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented in multiple embodiments, either separately or in any suitable subcombination. Furthermore, even if features may be described above as functioning in a particular combination and originally claimed as such, one or more features from a claimed combination may, in some cases, be deleted from the combination, and the claimed combination may be directed to subcombinations or variations of the subcombinations.

[0208] Similarly, while operations may be described in a particular order relative to the figures and in the claims, this should not be understood as requiring that such operations be performed in the particular order or sequential order shown, or that all of the operations shown be performed, to achieve desirable results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated into a single software product or packaged into multiple software products.

[0209] Specific embodiments of the present invention have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims may be performed in a different order and still produce desirable results. As an example, the processes corresponding to the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. 1. A method, implemented by one or more computers, for enabling a user to obtain information through an interaction, the interaction being between the user and an agent including a first trained language generation neural network, the first trained language generation neural network configured to process contextual input including one or more prompts, each including one or more natural language statements, to generate a natural language output, the method comprising: determining an initial context input and, in one or more of the plurality of dialogue update iterations; receiving a natural language request from the user; updating the context input to include the natural language request; and processing the contextual input using the first trained language generation neural network to generate one or more sample first natural language responses; generating one or more search queries from the natural language request; providing the one or more search queries to a search system interface; receiving one or more search results from the search system interface for each search query; determining, from the context input, for each of the search results, a corroborated context input comprising content from one of the search results; processing each corroborated context input using the first trained language generation neural network to generate one or more respective samples of a second natural language response; processing the one or more samples of the first natural language responses and the one or more samples of the second natural language responses using a trained response selection neural network to select a natural language answer from the one or more samples of the first natural language responses and the one or more samples of the second natural language responses; providing the natural language answer to the user in response to the natural language user request; updating the context input to include a representation of the natural language answer for a next dialogue update iteration; A method comprising:

2. 2. The method of claim 1, comprising: generating a plurality of samples of the first natural language response; generating a plurality of search queries from the natural language request; generating a plurality of search results for each search query; determining a respective corroborated context input for each search result; generating a respective sample of the second natural language response for each corroborated context input; and selecting the natural language answer from the plurality of samples of the first natural language response and the plurality of samples of the second natural language response.

3. In said next dialogue update iteration, receiving a follow-up natural language request from the user; updating the context input to include the follow-up natural language request; and using the first trained language generation neural network to process the contextual input to generate one or more samples of a third natural language response; generating one or more follow-up search queries from the follow-up natural language requests; providing the one or more follow-up search queries to the search system interface; receiving one or more follow-up search results from the search system interface for each follow-up search query; determining, from the context inputs, for each of the follow-up search results, a corroborated follow-up context input including content from the follow-up search results; using the first trained language generation neural network to process each corroborated follow-up context input to generate one or more respective samples of a fourth natural language response; processing the one or more samples of the third natural language responses and the one or more samples of the fourth natural language responses using the trained response selection neural network to select a follow-up natural language answer from the one or more samples of the third natural language responses and the one or more samples of the fourth natural language responses; providing the follow-up natural language answer to the user in response to the follow-up natural language request; updating the context input to include a representation of the follow-up natural language answer; 3. The method of claim 1 or 2, further comprising:

4. the trained response selection neural network comprises a second trained language model neural network, and processing the sample of natural language responses using the trained response selection neural network; For each sample of natural language responses, using the second trained language model neural network to process at least a portion of the contextual input and the sample of natural language responses to generate a preference score for the sample of natural language responses; selecting one of the samples of natural language responses based on the preference score for each sample of natural language responses to select the natural language answer; 4. The method of claim 1, 2, or 3, comprising:

5. For each sample of natural language responses, using a trained rule violation detection neural network to process at least in part the contextual input and the sample of natural language responses to determine, for each of a plurality of rules, a rule violation score that estimates a probability of violating the rule; further comprising selecting one of the samples of natural language responses to select the natural language answer is further based on the rule violation score of each of the rules for each sample. The method of claim 4.

6. selecting one of the samples of natural language responses to select the natural language answer; For each sample of natural language responses, determining a combined rule promotion score for the sample by combining the rule violation scores for each rule; combining the preference scores and the combined rule violation scores of the samples to determine a re-ranking score; selecting the natural language answer by selecting one of the samples of natural language responses based on the re-ranking score for each sample of natural language responses; The method of claim 5 , comprising:

7. 7. The method of claim 5 or 6, comprising, for each of the plurality of rules, using the trained rule violation detection neural network to process at least the portion of the contextual input, the sample of natural language responses, and the natural language expression of the rule to determine the rule violation score that estimates the probability of violating the rule.

8. the trained rule violation detection neural network comprises a third trained language generation neural network, and the method further comprises: for each of a plurality of rules, using the third trained language generation neural network to process at least the portion of the context input, the sample natural language responses, and the natural language expression of the rule to generate one or more natural language output tokens representing a decision on whether the rule has been violated; determining the rule violation score from one or more output layer values ​​corresponding to the one or more natural language output tokens; The method of claim 7, comprising:

9. for each of a plurality of rules, using the third trained language generation neural network to process at least the portion of the context input, the sample of natural language responses, and the natural language expression of the rule; processing at least the portion of the contextual input and the sample of natural language responses using the third trained language generation neural network to determine a shared intermediate state of the third trained language generation neural network; and then for each of a plurality of rules, processing the natural language expression of the rule using the third trained language generation neural network, starting from the shared intermediate state of the third trained language generation neural network; The method of claim 8, comprising:

10. 10. The method of claim 8, wherein the first trained language generation neural network, the second trained language model neural network, and the third trained language generation neural network each comprise a respective sequence-to-sequence neural network configured to receive an input sequence of tokens and process the input sequence of natural language tokens according to a respective set of neural network parameters to generate an output sequence of natural language tokens, and wherein the first trained language generation neural network, the second trained language model neural network, and the third trained language generation neural network include a set of shared input layers.

11. 11. The method of claim 1, wherein the first trained language generation neural network, the second trained language model neural network, and the third trained language generation neural network are stored on a user computing device, a search system is remote from the user, and the one or more search results are received via a wired or wireless communication link between the user computing device and the search system.

12. 1. A method for training a neural network system to enable use of an agent implemented by one or more computers, the agent comprising a first language generation neural network, to obtain information through an interaction between a user and the agent, the method comprising: determining a context input for a current dialogue iteration and, in one or more dialogue update iterations; obtaining a natural language output statement from an action selection policy neural network including the first language generation neural network by generating natural language tokens for the natural language output statement at each successive time step until an end of a statement generation episode, wherein generating the natural language tokens at a time step includes: using the action selection policy neural network to process the context input of the current dialogue iteration and previously generated tokens in the statement generation episode to select an action, wherein the action is to select a next token of the natural language output statement; processing at least a portion of the context input and the natural language output statement using a response selection neural network to determine a first reward for the natural language output statement; processing at least a portion of the context input and the natural language output statement using a rule violation detection neural network to determine a second reward for the natural language output statement; training the action selection policy neural network, which comprises the first language generation neural network, based on the first reward and the second reward using reinforcement learning techniques; said obtaining including: A method comprising:

13. adding a prompt to the context input of the current dialogue iteration to define a role of the action selection policy neural network when generating tokens of the natural language output statement, the role being one of a user role, an agent role, and a search query generation role; The method of claim 12 further comprising:

14. updating the context input to include a representation of the natural language output statement for a next dialogue update iteration; adding a prompt to the updated context input to define the role of the action selection policy neural network in the next dialogue update iteration, wherein the role of the agent in the next dialogue update iteration is different from the role of the agent in the current dialogue update iteration; 14. The method of claim 13, further comprising:

15. determining the first reward when the role is the user's role using a version of the response selection neural network trained to process contextual input without evidence supporting the natural language output statement; determining the first reward when the role is the search query generation role using a version of the response selection neural network trained to process contextual input with and without supporting evidence for the natural language output statement; determining the first reward when the role is the agent's role using the versions of the response selection neural network trained to process contextual inputs with and without supporting evidence for the natural language output statement, and the versions of the response selection neural network trained to process contextual inputs without supporting evidence for the natural language output statement; 15. The method of claim 13 or 14, further comprising:

16. 16. The method of claim 13, 14, or 15, further comprising training the action selection policy neural network without using the second reward when the role is one of the user role and the search query generation role.

17. storing a plurality of trajectories in a dialogue buffer, each trajectory including the context input for the current dialogue iteration, the natural language output statement, the first reward, and the second reward; training the action selection policy neural network with the stored trajectories using the reinforcement learning techniques; The method of any one of claims 12 to 16, further comprising:

18. storing one of the trajectories in the interaction buffer; determining a reward value from one or both of the first reward and the second reward within the trajectory; storing the trajectory if the reward value of the trajectory is greater than a minimum reward threshold; 20. The method of claim 17, further comprising:

19. For one or more of the trajectories, the natural language output statement is a search query statement comprising a search query for querying a search system, and the method comprises: providing the search query to a search system interface of the search system; receiving one or more search results from the search system interface in response to the search query; including data from one or more of the search query and the search results in the trajectory stored in the interaction buffer; further comprising determining the context input for a dialogue iteration includes retrieving data for the context input from the stored trajectory, the data including data from one or more of the search query and the search results; 19. The method of claim 17 or 18.

20. the response selection neural network comprises a second language model neural network configured to process at least a portion of the contextual input and the natural language output statement to generate a preference score defining the first reward, the method comprising: training the second language model neural network using training data items, each data item comprising a sample of interactions including a natural language request, a set of natural language responses generated by one or more training language generation neural networks, and preference data indicating relative preferences for the natural language responses; further comprising the set of natural language responses includes responses generated by processing a context input including samples of the interactions both with and without search results from a search query based on the natural language request using the one or more training language generation neural networks. The method according to any one of claims 12 to 19.

21. 21. The method of claim 20, comprising training the second language model neural network comprising backpropagating a gradient of a response selection objective function, the response selection objective function depending on an exponential function of the preference score of a relatively most preferred one of the natural language responses scaled by a sum of the exponential functions of each of the preference scores of the set of natural language responses, and an additional term representing that no option is preferred.

22. the rule violation detection neural network is a rule-conditioned classifier neural network, and the method comprises: processing at least a portion of the context input and the natural language output statements using the rule-conditioned classifier neural network, each conditioned on a respective one of a plurality of rules, to determine a plurality of rule violation scores, one for each of the rules, wherein the rule violation score for a rule represents a probability that the portion of the context input and the natural language output statements violate the rule; combining the rule violation scores to determine the second reward; The method according to any one of claims 12 to 21, comprising:

23. the rule violation detection neural network comprises a third language generation neural network, and the method further comprises: for each of the plurality of rules, processing at least a portion of the context input, the natural language output statement, and a natural language rule statement representing the rule to generate one or more natural language output tokens representing a determination of whether the rule has been violated; determining the rule violation scores from one or more output layer values ​​corresponding to the one or more natural language output tokens and used by the third language generation neural network to determine the one or more natural language output tokens; 23. The method of claim 22, comprising:

24. using a supervised learning algorithm to train the rule violation detection neural network on a training dataset including dialogue data items of the plurality of rules, each dialogue data item including a sequence of natural language statements representing a dialogue and a label indicating whether the dialogue complies with a particular rule; The method of any one of claims 12 to 23, further comprising:

25. obtaining the natural language output statements for the agent in an agent dialogue update iteration; obtaining the natural language output statement for the user in a user dialogue update iteration that follows the agent dialogue update iteration in response to the natural language output statement for the agent; training the action selection policy neural network based on the first reward and the second reward for the agent interaction update iteration; training the action selection policy neural network based on the first reward but not the second reward for the user interaction update iteration; The method of any one of claims 12 to 24, comprising:

26. determining the context input for an initial dialog iteration; generating natural language requests using a fourth trained natural language generation neural network, the fourth trained natural language generation neural network being trained to generate red team natural language statements that, when processed by the first or other language generation neural network in combination with context inputs, cause the first or other language generation neural network to generate natural language output statements that, in combination with the context inputs, violate one or more rules implemented by the rule violation detection neural network; including one or more of the red team natural language statements in the context input for the initial dialogue iteration; The method of any one of claims 12 to 25, comprising:

27. 27. The method of any one of claims 1 to 26, wherein the context input comprises one or more natural language statements related to the environment, including a natural language request related to the environment, and the natural language answer or natural language output statement is related to the environment.

28. the method is used to control a mechanical system operating in a real-world environment to perform a task; determining the initial context input; obtaining one or more observations of the real-world environment from one or more sensors; and processing the one or more observations to generate a natural language representation of the one or more observations; providing one or more natural language statements to the initial context input using the natural language representation of the one or more observations; Including, the natural language request relates to an action to be performed by the machine system; The method comprises: using the natural language answer or the natural language output statement to control the mechanical system in the real-world environment; The method of any one of claims 1 to 27, further comprising:

29. the machine system having a control system for controlling actions of the machine system, and receiving the natural language request; receiving a control signal from the control system; generating the natural language request from the control signal; 29. The method of claim 28, comprising:

30. 30. The method of claim 28 or 29, wherein the mechanical system comprises a robot or an autonomous or semi-autonomous vehicle, and the actions include actions for controlling movement or navigation of the robot or vehicle in the real-world environment.

31. the environment is a real-world environment; the contextual input is derived from at least one observation characterizing a current state of the real-world environment generated from measurements from one or more sensors configured to sense the real-world environment; the natural language request includes data characterizing the planned navigation of the robot or autonomous vehicle. the natural language answer or natural language output statement characterizes an action to be performed by the agent in response to the observation; 28. The method of claim 27.

32. controlling navigation of the agent based on the natural language answer or natural language output statement; 32. The method of claim 31 , further comprising:

33. the environment is a manufacturing plant for producing a product, the manufacturing plant comprising a plurality of manufacturing units configured such that intermediate versions or components of the product can be moved between the manufacturing units during the production of the product, and the method is used to control one or more of the manufacturing units and to control the movement of the intermediate versions or components of the product between the manufacturing units; obtaining the context input includes obtaining one or more observations of the manufacturing unit or the movement from one or more sensors, and processing the one or more observations to generate a natural language representation of the one or more observations; the natural language request relates to an action that controls one or more operations of the manufacturing unit or that controls the movement of the manufacturing unit; The method comprises: using the natural language representation of the one or more observations to provide one or more of the natural language statements of contextual information; using said natural language answers or natural language output statements to control one or more operations of said manufacturing unit or to control said movement; 28. The method of claim 27, further comprising:

34. The manufacturing plant has a plant control system for controlling the manufacturing unit or for controlling the movement, and receiving the natural language request comprises: receiving a control signal from the plant control system; generating the natural language request from the control signal; 34. The method of claim 33, comprising:

35. the environment is a real-world environment, and the method is used to control one or more devices in a facility having a plurality of devices; obtaining the contextual input includes obtaining one or more observations of the facility from one or more sensors; and processing the one or more observations to generate a natural language representation of the one or more observations; the natural language request relates to an operation of the facility; The method comprises: using the natural language representation of the one or more observations to provide one or more of the natural language statements of contextual information; using the natural language response or the natural language output statement to control the one or more devices in the facility; 28. The method of claim 27, further comprising:

36. 36. The method of claim 35, wherein the one or more devices control heating and / or cooling of the facility.

37. the environment is a real-world environment, and the method is used to diagnose faults in a mechanical system operating in the real-world environment; obtaining the contextual input includes obtaining one or more observations of the mechanical system from one or more sensors and processing the one or more observations to generate a natural language representation of the one or more observations; the natural language request relates to the operation of the mechanical system; The method comprises: using the natural language representation of the one or more observations to provide one or more of the natural language statements of contextual information; identifying faults in the mechanical system using the natural language answers or the natural language output statements; 28. The method of claim 27, further comprising:

38. 28. The method of claim 27, wherein the environment is a computer security system, the method is used to determine whether a computer security incident has been resolved on a computer network, and the contextual input includes data characterizing the computer security incident derived from a system log, data characterizing the computer network, or both.

39. obtaining the contextual input includes obtaining one or more observations of the computer network from the system log, the data characterizing the computer network, or both, and processing the one or more observations to generate a natural language representation of the one or more observations; the natural language request relates to the computer security incident or the secure operation of the computer network; The method comprises: using the natural language representation of the one or more observations to provide one or more of the natural language statements of contextual information; using the natural language answer or the natural language output statement to identify a security status of or a security flaw in the computer network; 39. The method of claim 38, further comprising:

40. 28. The method of claim 27, wherein the environment is a computer software evaluation system, the method is used to determine whether a portion of software code will or has executed as intended on a computer system, and the contextual input includes data characterizing one or more of the portion of software code, the execution of the portion of software code, the computer system on which the software code executes or has executed, an artifact of the execution of the software code, or one or more validation rules for the execution of the portion of software code.

41. obtaining the context input includes obtaining from the characterization data one or more observations of the execution of the portion of the software code, or the computer system on which the software code executes or has executed, or an artifact of the execution of the software code, or one or more validation rules for the execution of the portion of the software code, and processing the one or more observations to generate a natural language representation of the one or more observations; the natural language request relates to the execution of the portion of the software code; The method comprises: using the natural language representation of the one or more observations to provide one or more of the natural language statements of contextual information; using the natural language answer or the natural language output statement to determine whether the portion of software code is or has executed as intended on the computer system; 41. The method of claim 40, further comprising:

42. The method of any one of claims 1 to 41, wherein the natural language request comprises a natural language information request that defines information that the natural language answer provides.

43. One or more computer readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the respective method of any one of claims 1 to 42.

44. one or more computers; one or more storage devices communicatively coupled to the one or more computers, the one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the respective method of any one of claims 1 to 42; A system comprising:

Images