Generating suggested communications by simulating interactions using language model neural networks

Abstract

To provide methods, systems and recording mediums for generating suggested communications during a multi-agent interaction using a language model neural network.SOLUTION: A method comprises: identifying multiple first and second strategies for selecting between parameterizations for communications for first and second actors; parametrizing a set of context variables for the communications of the actors during an interaction; for each of multiple pairs including respective first and second strategies: generating simulations of the interaction using a language model neural network; determining, from the simulations, a score for the pair that indicates a degree to which the first and second actors satisfy objectives for the interaction; selecting, using the scores for the multiple pairs, a parametrization for the first actor; and generating, by processing an input that conditions the language model neural network on the selected parameterization, a suggested communication for the first actor.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] This specification relates to processing data using machine learning models. [Background technology]

[0002] As an example, a neural network is a machine learning model that uses one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as an input to another layer in the network, for example, the next hidden layer or the output layer. Each layer of the network generates an output from the received input according to the current values ​​of a respective set of weights. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, LA Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language models, arXiv preprint arXiv:2203.15556, 2022 [Non-Patent Document 2] JW Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, HF Song, J. Aslanides, S. Henderson, R. Ring, S. Young, E. Rutherford, T. Hennigan, J. Menick, A. Cassirer, R. Powell, G. van den Driessche, LA Hendricks, M. Rauh, P. Glase, P. Huang, J. G. Huang, R. Ring, S. Young, E. Welbl, S. Dathathri, S. Huang, J. Uesato, J. Mellor, I. Higgins, A. Creswell, N. McAleese, A.Wu, E. Elsen, SM Jayakumar, E. Buchatskaya, D. Budden, E. Sutherland, K. Simonyan, M. Paganini, L. Sifre, L. Marten, A. Li, A. Kunco, A. Kunco. Nematzadeh, E. Gribovskaya, D. Donato, A. Lazaridou, A. Mensch, J. Lespiau, M. Tsimpoukelli, N. Grigorev, D. Fritz, T. Sottiaux, M. Pajarskas, T. Pohlen, Z. Gong, D. Toyama, C. de Masson d'Autu, Y. Milik, T. V. Mili, V. Y. Babuschkin, A. Clark, D. de Las Casas, A. Guy, C. Jones, J. Bradbury, M. Johnson, BA Hechtman, L. Weidinger, I. Gabriel, WS Isaac, E. Lockhart, S. Osindero, L. Rimell, J. Dyer, O. Vinyals, K. Ayoub, J. L. Stanway, K. Bennett, D. K. Bennett, Hassac. Kavukcuoglu, and G. Irving.Scaling language models: Methods, analysis & insights from training gopher. CoRR, abs / 2112.11446, 2021. [Non-Patent Document 3] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019 [Non-Patent Document 4] Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. Towards a human-like open-domain chatbot. CoRR, abs / 2001.09977, 2020 [Non-Patent Document 5] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020 Summary of the Invention [Problem to be solved by the invention]

[0004] This specification describes a system, implemented as a computer program on one or more computers at one or more locations, that generates proposals for a given actor in a multi-actor interaction by simulating the interaction using a language model neural network. [Means for solving the problem]

[0005] Particular embodiments of the subject matter described in this specification can be implemented to realize one or more of the following advantages.

[0006] This specification describes techniques for generating suggested communications for actors in a multi-actor interaction. As an example, a multi-actor interaction can be a communication session in which a user exchanges natural language communications with one or more other actors, e.g., other users, software agents, or both. Thus, in this example, the system can generate suggested natural language communications, e.g., email messages, text messages, chat messages, etc., to be sent by the user to one or more other agents.

[0007] For example, some existing techniques, such as auto-completion or next turn prediction techniques, attempt to predict the next communication in an interaction by taking into account previous communications, and possibly previous interactions in which the user has been involved.

[0008] However, such approaches do not take into account that different interactions may have different objectives and that different suggestions may be more appropriate than others even if the previous communications are the same. Furthermore, these approaches generate short-term predictions that do not consider the long-term impact on a given interaction of the next communication in the interaction, e.g., the impact of a given communication on communications received from other agents.

[0009] In contrast, the present specification uses an entirely different paradigm to generate proposed communications. In particular, the present specification generates the proposals by using a language model neural network to evaluate multiple different parameterizations of a set of communication-related context variables for both the "first" actor for which the proposal is generated and other actors in the interaction, and then selecting the parameterization that results in the interaction objective being met. To do so, the system can generate simulations of the interactions using the language model neural network, and then score each of those simulations to determine the extent to which the simulated interaction met the interaction objective.

[0010] As a result, the system leverages the power of language model neural networks to effectively "model" the impact of different parameterizations of context variables on the remainder of the interaction, resulting in suggested communications that are more effective and more tailored to the current interaction, improving the user experience.

[0011] Furthermore, by performing simulations of the current interaction at different time steps within the current interaction, the system can effectively adjust its strategy for generating suggested communications as the current interaction progresses, further improving the relevance and usefulness of the generated suggested communications and further improving the user experience.

[0012] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description and drawings. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram of an example neural network system. [Diagram 2] FIG. 1 is a flow diagram of an example process for generating suggested communications using a language model neural network. [Diagram 3] FIG. 1 is a flow diagram of an example process for generating a simulation of a pair of strategies. [Figure 4] FIG. 1 is a flow diagram of an exemplary process for generating a score for a simulation. [Diagram 5] 1 is a flow diagram of an example process for generating suggested communications for a user during a communication session. [Figure 6] FIG. 1 illustrates an example of using a language model neural network to generate suggested communications. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Like reference numbers and designations in the various drawings indicate like elements.

[0015] 1 is a diagram of an exemplary neural network system 100. Neural network system 100 is one example of a system implemented as a computer program on one or more computers at one or more locations in which the systems, components, and techniques described below may be implemented.

[0016] The system 100 is a system for generating proposed communications 104 for a first actor 132 in a multi-actor interaction 120 by simulating the interaction using a language model neural network 110.

[0017] The language model neural network 110 is a neural network configured to process an input and generate an output that includes a probability distribution over a set of text tokens in a vocabulary of text tokens, where the probability of each token represents the likelihood that the text token immediately follows the input. The tokens in the vocabulary may include text tokens such as, for example, characters, subwords, word pieces, etc., and may optionally include tokens that represent other types of data, such as, for example, audio, images, video, or other sensor data.

[0018] For example, the language model neural network can be an autoregressive language model neural network.

[0019] A language model neural network is referred to as an autoregressive neural network because the neural network autoregressively generates an output sequence of tokens by generating each particular token in the output sequence conditional on the current input sequence, including any tokens that precede the particular text token in the output sequence, i.e., tokens already generated for any previous position in the output sequence that precedes the particular token's particular position, and on a context input ("context sequence") that provides a context for the output sequence.

[0020] For example, the current input sequence when generating a token at any given position in the output sequence may include a context sequence and any preceding tokens that precede the given position in the output sequence. As a particular example, the current input sequence may include a context sequence followed by any preceding tokens that precede the given position in the output sequence. Optionally, the context and the current output sequence may be delimited by one or more predefined tokens in the current input sequence.

[0021] More specifically, to generate a particular token at a particular position in a candidate output sequence, the neural network can process the current input sequence to generate a score distribution, e.g., a probability distribution, that assigns a respective score, e.g., a respective probability, to each text token in a vocabulary of text tokens. The neural network can then use the score distribution to select a text token from the vocabulary as the particular token. For example, the neural network can greedily select the highest scoring token or can sample a token from the distribution, e.g., using kernel sampling or another sampling technique.

[0022] As a particular example, the language model neural network may be an autoregressive transformer-based neural network that includes (i) multiple attention blocks, each of which applies a self-attention operation, and (ii) an output sub-network that processes the output of the last attention block to generate a score distribution.

[0023] The neural network can have any of a variety of transformer-based neural network architectures. Examples of such architectures include J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language models, arXiv preprint arXiv:2203.15556, 2022, J.W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, H. F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young, E. Rutherford, T. Hennigan, J. Menick, A. Cassirer, R. Powell, G. van den Driessche, L. A. Hendricks, M. Rauh, P. Huang, A. Glaese, J. Welbl, S. Dathathri, S. Huang, J. Uesato, J. Mellor, I. Higgins, A. Creswell, N. McAleese, A.Wu, E. Elsen, S. M. Jayakumar, E. Buchatskaya, D. Budden, E. Sutherland, K. Simonyan, M. Paganini, L. Sifre, L. Martens, X. L. Li, A. Kuncoro, A. Nematzadeh, E. Gribovskaya, D. Donato, A. Lazaridou, A. Mensch, J. Lespiau, M. Tsimpoukelli, N. Grigorev, D. Fritz, T. Sottiaux, M. Pajarskas, T. Pohlen, Z. Gong, D. Toyama, C. de Masson d'Autume, Y.Li, T. Terzi, V. Mikulik, I. Babuschkin, A. Clark, D. de Las Casas, A. Guy, C. Jones, J. Bradbury, M. Johnson, B. A. Hechtman, L. Weidinger, I. Gabriel, W. S. Isaac, E. Lockhart, S. Osindero, L. Rimell, C. Dyer, O. Vinyals, K. Ayoub, J. Stanway, L. Bennett, D. Hassabis, K. Kavukcuoglu, and G. Irving. Scaling language models: Methods, analysis & insights from training gopher. CoRR, abs / 2112.11446, 2021、Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019、Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. Towards a human-like open-domain chatbot. CoRR, abs / 2001.09977, 2020, and Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.

[0024] In general, however, a Transformer-based neural network includes a sequence of attention blocks, where during processing of a given input sequence, each attention block in the sequence receives a respective input hidden state for each input token of the given input sequence. The attention block then updates at least the hidden state of the last token in the given input sequence by at least partially applying self-attention to generate a respective output hidden state for the last token. The input hidden state of the first attention block is the embedding of the input token in the input sequence, and the input hidden state of each subsequent attention block is the output hidden state generated by the preceding attention block.

[0025] In this example, the output sub-network processes the output hidden state generated by the last attention block in the sequence for the last input token in the input sequence to generate a score distribution.

[0026] Typically, before using neural network 100, system 100 or another training system pre-trains language model neural network 110 on a language modeling task, such as, for example, a task requiring predicting the next token following a current sequence in training data, given a current sequence of text tokens. As a particular example, the language model neural network may be pre-trained on a maximum likelihood objective on a large dataset of text, such as, for example, publicly available text from the Internet or another text corpus. In some cases, system 100 or another training system then further trains language model neural network 110 on one or more objectives, including, for example, supervised fine-tuning, reinforcement learning, instruction tuning, etc.

[0027] In some implementations, the language model neural network 110 is fine-tuned to perform the tasks described below. In other implementations, the language model neural network 110 is not further trained and can be made to generate an appropriate output, for example, by including natural language instructions, few-shot prompts, or a combination of the two in any given input to the neural network 110.

[0028] As described above, the system 100 uses the language model neural network 110 to generate suggested communications for a first actor 132, which is one of multiple actors 130 in a multi-actor interaction 120.

[0029] A multi-actor interaction 120 may be any suitable interaction in which multiple actors 130 send communications 140 to one another.

[0030] For example, a multi-actor interaction 120 can be a communication session in which multiple actors 130 exchange electronic messages, e.g., emails, chats, SMS messages, MMS messages, etc. In this example, the communication 140 is generally a natural language communication that is all or part of the electronic message.

[0031] In this example, the "first actor" 132 needing a suggestion may be, for example, a user of a user device, such as a "first user," and the system 100 may provide the suggested communication for presentation to the first user in a user interface of the user device. Other actors may include other users, automated agents, or a combination of both.

[0032] In this example, the first user can specify one or more goals for the interaction. For example, the first user can submit a natural language input that specifies the goal of the interaction. As another example, the first user can submit an input that selects one or more predefined goals for the interaction from a set of possible goals, or can fill out a form that specifies the goal.

[0033] As another example, the system 100 can use the language model neural network 110 (or another model) to process one or more initial communications from an interaction to generate one or more candidate destinations for the interaction, and the user 132 can submit input selecting one or more of the candidate destinations as the final destination of the interaction.

[0034] As yet another example, the system 100 can automatically identify a purpose for an interaction based on the context of the interaction and, optionally, one or more initial communications from the interaction. For example, the system 100 can identify the purpose based on the communication for the interaction and the software application or website used to send the initial communication, e.g., when the interaction includes a website for making travel reservations and the initial communication identifies a date range, the system can identify that the purpose of the interaction is to make travel reservations within the specified date range.

[0035] The objectives can be based, for example, on the emotion of the communications of the first user, the other agent, or both, i.e., the one or more objectives can specify a target emotion of the communications of the first user, the other agent, or both during the interaction.

[0036] As another example, the objective may include one or more objectives based on a goal of the first user with respect to the interaction. For example, the objective may include solving a problem the first user has, taking a specified action, or achieving another type of goal of the first user with respect to the interaction.

[0037] The communications session may be any suitable communications session regarding any suitable topic, such as, for example, a customer support session, a negotiation, a contest, a scheduling or logistics session, a collaboration session, etc.

[0038] In another example, the multi-agent interaction 120 may be a communication session between mechanical, electronic, or electromechanical agents, such as, for example, robots, autonomous vehicles, or other agents. In this example, the communication may be natural language communication, computer code segments, or communication in a domain-specific language that the agents can parse. Examples of such communication may include communication for resource allocation or distribution, scheduling or logistics, etc.

[0039] In this example, the "first actor" 132 needing a proposal may be one of the agents.

[0040] In this example, a user of system 100 may specify one or more objectives for an interaction associated with a first user. For example, the objectives may be based on, for example, an amount of resources allocated to the first agent as a result of the interaction, a navigation time to a destination for the first agent as a result of the interaction, etc.

[0041] More specifically, the system 100 uses a language model neural network 110 to generate respective proposed communications for a first actor 132 at each of a number of time steps during an interaction 120 .

[0042] The generation of the proposed communication is described in more detail below.

[0043] When the first actor 132 is a user, the user can decide whether to submit the proposed communication as a communication to one or more of the other actors 130. When the first actor 132 is an agent, the system 100 can automatically submit the communication to one or more of the other actors 130 as an agent's communication, or a control system can decide whether to submit the proposed communication to one or more of the other actors 130.

[0044] 2 is a flow diagram of an exemplary process 200 for generating proposed communications at a given time step during an interaction between multiple actors. For convenience, process 200 is described as being performed by one or more computer systems located at one or more locations. For example, a neural network system suitably programmed in accordance with this specification, such as the neural network system 100 shown in FIG. 1, can perform process 200.

[0045] The system can perform process 200 at various points during an interaction. For example, the system can perform process 200 each time a first of a plurality of actors requests a suggestion from the system. For example, a user can select a user interface element when viewing recent communications in an interaction to request a suggested response communication.

[0046] The system identifies a first plurality of first strategies for selecting among a first parameterization of a first actor's communication (step 202).

[0047] In general, each first parameterization parameterizes a set of one or more context variables related to the communication of the first actor during the interaction.

[0048] That is, each first parameterization includes a respective value for a set of one or more context variables that each influence how the first actors communicate during the communication.

[0049] For example, one or more of the context variables in the set may define a communication style for the first actor's communication. Examples of such context variables include communication tone, emotion, argumentative style, writing style, use of literary techniques, etc.

[0050] As another example, one or more of the context variables in the set can define a set of information available as context to the first actor during the interaction, i.e., each of these context variables can determine how much external information (other than the communication exchanged during the interaction) is available to the first user during the interaction.

[0051] In some cases, the possible values ​​of each context variable are a discrete set of possible values, i.e., the context variable has a fixed, discrete set of possible values. For example, for a communication tone context variable, the possible values ​​may be a fixed set that includes some or all of: formal, informal, optimistic, cooperative, friendly, funny, satirical, assertive, cheerful, angry, etc.

[0052] In other cases, the possible values ​​of one or more of the context variables are continuous: for example, a tone variable in a communication context may have possible values ​​selected from a continuous range between 0 and 1, with 0 representing an informal tone and 1 representing a formal tone.

[0053] In some cases, all of the first strategies may assign the same parameterization to all communications during an interaction, i.e., each first strategy may assign the same values ​​to context variables of all communications of the first agent during the entire interaction.

[0054] In other cases, one or more of the first strategies assign different parameterizations to different communications, for example alternating between different parameterizations or cycling through a set of three or more parameterizations.

[0055] The system also identifies a second plurality of second strategies for selecting among a second parameterization regarding communication of a second actor of the plurality of actors for the interaction (step 204).

[0056] Each second parameterization parameterizes a set of context variables related to the communication of the second actor during the interaction.

[0057] The strategy, the set of context variables, and the parameterization may be the same or different for the first and second actor.

[0058] In general, a set of context variables, and possibly a parameterization of actors, can be received as input by the system.

[0059] In some implementations, the system can use a language model neural network, such as the language model neural network described above, or a different language model neural network, such as a smaller, more computationally efficient language model neural network, to generate or augment possible values ​​for a given one of the context variables.

[0060] For example, the system may receive initial input specifying a context variable and, optionally, one or more possible values ​​for the context variable, and may then use a language model neural network to process the input specifying the context variable and, if available, one or more possible values ​​to generate one or more additional possible values. That is, given an input identifying a context variable, the system may use a neural network to generate suggested possible values ​​for the context variable.

[0061] In some implementations, the system can generate a strategy for a given actor by selecting all possible parameterizations of the set of context variables, by randomly selecting a fixed number of parameterizations from the set of possible parameterizations, or by using some other selection scheme.

[0062] In some other implementations, in addition to specifying the context variables and parameterization, the user input may also identify a strategy for the first user, the second user, or both.

[0063] For each of a plurality of pairs, each including a respective first strategy and a respective second strategy, the system uses the language model neural network to generate one or more simulations of an interaction assuming that the first actor communicates according to the first strategy and the second actor communicates according to the second strategy (step 206).

[0064] For example, the system may select all possible pairs of the first and second strategies, or may randomly select a subset of the possible pairs of the first and second strategies, e.g., randomly select the number of pairs that can be simulated within the latency budget to generate the selected communication.

[0065] In some cases, the system generates a single simulation for each pair. In other cases, the system can generate multiple different simulations for the same pair because the language model neural network can generate different output sequences for the same input sequence, as described above.

[0066] The generation of a simulation using a language model neural network is described in more detail below with reference to FIG.

[0067] The system then determines pairwise scores from the one or more simulations that indicate the extent to which a first actor communicating according to a first strategy and a second actor communicating according to a second strategy satisfy one or more objectives of the interaction (step 208).

[0068] For example, as discussed above, an objective may be an objective received as input by the system and to be achieved by a first actor.

[0069] The determination of the score for a pair of strategies is described in more detail below.

[0070] When there is more than one simulation, the system may determine a respective simulation score for each simulation and then, for example, average or sum or otherwise combine the simulation scores to generate a final score for the pair.

[0071] The system then uses the scores of the multiple pairs to select a first parameterization for the first actor (step 210).

[0072] For example, the system may generate a representation of the interactions that includes pairwise scores and then apply a game theory solver to the representation to identify an optimal first parameterization of the first actor. Examples of such representations include payoff matrices, payoff tensors, game trees, etc.

[0073] Examples of game theory solvers that can be used include standard game (NFG) solvers, evolving game (EFG) solvers, and multi-agent reinforcement learning (MARL) solvers. An example of a MARL solver is the policy space response oracle (PSRO) solver.

[0074] Thus, by applying a game theoretic solver, a first parameterization is identified that, based on the results of the simulation, is most likely to result in the interaction objective being met when used to generate the communication of the first agent in the current step of the interaction.

[0075] The system then generates proposed communications for the first actor by processing inputs that condition the language model neural network on the selected first parameterization (step 212), i.e., the system generates proposed communications characterized by the first parameterization given the context of the interaction up to the given time step.

[0076] More specifically, the system can identify a selected parameterization, process input including the context of the interaction up to a given time step, and instructions to generate a proposed communication that is parameterized by the selected parameterization based on the given context using a language model neural network to generate the proposed communication.

[0077] The input may also optionally include additional information.

[0078] For example, the input may include information that characterizes the first agent or the first agent's objectives for the interaction. This information may include "private" information that is not available from messages of previous interactions. For example, when the first agent is attempting to arrange a meeting time, the first agent may not have communicated the first agent's preferred meeting time during the interaction. As another example, when the first agent is attempting to negotiate the price of a service or product, the first agent may not have communicated the first user's target price for the service for the product.

[0079] The system may then provide the proposed communication, for example, to the first actor's control system or for presentation to the first actor.

[0080] 3 is a flow diagram of an exemplary process 300 for generating a simulation of the interaction of a pair of strategies using a language model neural network. For convenience, the process 300 is described as being performed by one or more computer systems located at one or more locations. For example, a neural network system suitably programmed in accordance with this specification, such as the neural network system 100 shown in FIG. 1, can perform the process 300.

[0081] For example, given a pair including a respective first strategy and a respective second strategy, the system can execute process 300 to generate a simulation of an interaction assuming that the first actor communicates according to the first strategy and the second actor communicates according to the second strategy.

[0082] In particular, the system can execute process 300 in each of multiple iterations to generate a respective communication for each iteration. The communications across the iterations then act as a simulation of the interaction starting from the current point in the interaction.

[0083] The system identifies previous communications between the first and second actors during the simulation, i.e., communications generated in any previous iterations during the generation of the simulation (step 302). Optionally, when the simulation is being generated after an interaction has already begun, the system can also identify previous actual communications generated during the interaction before the generation of the simulation began.

[0084] The system selects either the first actor or the second actor as the current actor for the iteration, step 304. Generally, the system selects the first actor and the second actor according to a model of how the actors will interact during the interaction being simulated.

[0085] For example, the model may indicate that during an interaction, two actors communicate alternately. In this example, the system may select a first actor if the second actor generated the most recently identified communication, and may select a second actor if the second actor generated the most recently identified communication.

[0086] As another example, the model may indicate that each communication is sent by a first actor unless a recent communication by the first actor meets certain criteria, e.g., does not include a request for a response from the second actor. In this example, the system may select the second actor only if the recent communication was from the first actor and meets certain criteria.

[0087] As yet another example, the model may indicate that at any given iteration, a first actor generates a communication with probability p and a second actor generates a communication with probability 1-p. In this example, the system selects the first actor with probability p and the second actor with probability 1-p.

[0088] Once the first actor is selected as the current actor, the system selects a first parameterization for the iteration according to the first strategy (step 306), i.e., the system applies the first strategy to the identified communication to determine the first parameterization.

[0089] The system then uses the language model neural network to process (i) inputs that condition the language model neural network based on the selected first parameterization, and (ii) at least a portion of the previous communications during the simulation (step 308) to generate communications for the first actor. For example, the inputs may include instructions for generating communications parameterized by the selected first parameterization, given a description of the selected first parameterization and context provided by the previous communications. As discussed above, the inputs to the language model neural network may also optionally include actual previous communications from earlier in the actual interaction. In addition, as discussed above, the inputs may also include "private" information that characterizes the first actor or the state of the interaction.

[0090] Once the second actor is selected as the current actor, the system selects a second parameterization for the iteration according to a second strategy (step 310), i.e., the system applies the second strategy to the identified communications to determine the second parameterization.

[0091] The system then uses the language model neural network to process (i) inputs that condition the language model neural network based on the selected second parameterization, and (ii) at least a portion of the previous communications during the simulation (step 312) to generate communications for the second actor. For example, the inputs may include instructions for generating communications parameterized by the selected second parameterization, given a description of the selected second parameterization and context provided by the previous communications. As noted above, the inputs to the language model neural network may also optionally include actual previous communications from earlier in the actual interaction. The system may not have access to "private" information of the second actor. In this case, the system may omit this information from the input, may include an indication that the information is unavailable, or may randomly sample the information from a set of plausible characteristics.

[0092] Thus, by repeatedly executing iterations of process 300, the system generates a simulation that includes a respective communication from one of the actors in each of multiple iterations.

[0093] The system continues to execute iterations of process 300 until a termination criterion is met.

[0094] For example, the system may determine that a termination criterion has been met when a predetermined number of iterations have been performed.

[0095] As another example, the system may determine that a termination criterion is met when the communications generated during an iteration satisfy a specified pattern of communications that signifies the end of the interaction.

[0096] As yet another example, at each iteration, the system may use the language model neural network to process input that includes the communication generated during the iteration and a prompt to ask the language model neural network if the interaction is complete. The system may then determine that the termination criteria is met when the output of the language model neural network indicates the end of the interaction. For example, the input may also include data characterizing a goal of the interaction and may ask if the goal has been met. As another example, if the input also includes data characterizing the goal, the input may also ask if the goal is unachievable given the current state of the interaction.

[0097] 4 is a flow diagram of an exemplary process 400 for determining a score for a simulation of the interaction of a pair of strategies using a language model neural network. For convenience, the process 400 is described as being performed by one or more computer systems located at one or more locations. For example, a neural network system suitably programmed in accordance with this specification, such as the neural network system 100 shown in FIG. 1, can perform the process 400.

[0098] The system generates summary input from one or more of the communications generated during the simulation (step 402).

[0099] For example, the summary input may include all communications generated during a simulation. As another example, the summary input may include less than all communications generated during a simulation, such as, for example, every nth communication during a simulation, where n is an integer greater than 1, or the communications for the kth last iteration during a simulation, where k is an integer greater than or equal to 1.

[0100] As another example, the summary input may be an abstract summary of the communication. To generate the abstract summary, the system may process the communication generated during the simulation and appropriate prompts using a second language model neural network.

[0101] The "second" language model neural network may be the same as the language model neural network described above, or it may be a different, smaller language model neural network.

[0102] The system generates a criteria input that characterizes the purpose of the interaction, step 404. For example, the criteria input can be a natural language description of the purpose of the interaction.

[0103] The system processes the summary input and the criteria input using a second language model neural network to generate an output defining a simulation score for the simulation indicating the degree to which the simulation satisfies one or more criteria (step 406).

[0104] For example, the system may process the summary input, the criteria input, and instructions to score the interactions defined by the summary input given criteria described in the criteria input using a second language model neural network to cause the second language model neural network to generate a summary score.

[0105] As an example, the score may be a probability, log-probability, logit, or other likelihood assigned by the second language model neural network to a predetermined response phrase, e.g., the phrase "yes" or the phrase "no."

[0106] For example, in this example, the input to the second language model neural network may include a query asking the second language model neural network whether the interaction summarized by the summary input meets criteria. For example, the input may be in the form "Given this summary of the interaction [summary input], does the interaction meet these criteria: [criteria input]?" As another example, the input may include k examples of the form "Summary: [exemplary summary input]; Criteria: [exemplary criteria input]; Does summary meet criteria? [yes / no]", where k is 1 or more, followed by "Summary: [summary input]; Criteria: [criteria input]; Does summary meet criteria?"

[0107] As another example, the score may be directly regressed as one or more output tokens by the second language model neural network in response to processing the input, i.e., upon instruction, the language model neural network can process the input and directly output the score.

[0108] For example, in this example, the input to the second language model neural network may include a query requesting the second language model neural network to score the degree to which the summary input satisfies the criteria specified by the criteria input. For example, the input may be in the form "Given this summary of the interaction [summary input] and the following criteria: [criteria input], output a score from 0 to 1 that evaluates the degree to which the criteria is met." As another example, the input may include k examples of the form "summary: [exemplary summary input]; criteria: [exemplary criteria input]; criteria satisfaction score: [exemplary score]," where k is 1 or greater, followed by "summary: [summary input]; criteria: [criteria input]; criteria satisfaction score:."

[0109] Alternatively, instead of performing steps 404 and 406, the system can receive a summary input, e.g., an extracted summary generated by a language model neural network, or data specifying a set of heuristics to be applied to a combination of one or more communications. The system can then apply the set of heuristics to the summary input to generate a simulation score.

[0110] 5 is a flow diagram of an exemplary process 500 for generating suggested communications for a user during an interaction with another agent. For convenience, process 500 is described as being performed by one or more computer systems located at one or more locations. For example, a neural network system suitably programmed in accordance with this specification, such as the neural network system 100 shown in FIG. 1, may perform process 500.

[0111] For example, the system may perform process 500 during an interaction in which a user of a user device is participating in an interaction, e.g., a communication session, with another agent, e.g., another user of another user device or a software agent.

[0112] The communications session may be any suitable communications session regarding any suitable topic, such as, for example, a customer support session, a negotiation, a contest, a scheduling or logistics session, a collaboration session, etc.

[0113] The communications session may be conducted via any suitable software application, for example, as part of an email exchange, as part of an online chat, or as part of a text messaging exchange.

[0114] The system generates suggested communications for the first user, step 502. For example, the system can use a language model neural network, as described above, to generate the suggested communications for the first user.

[0115] As a particular example, the system may receive a request for a proposed communication from a user taking into account existing communications within a session and the context of the session, which generally specifies one or more purposes of the session and, optionally, additional information about the session, e.g., information about other agents in the session.

[0116] For example, consider an email conversation between Bob (a tenant) and Alice (a landlord). After an initial email is sent from Alice informing Bob that she needs to raise the rent, the system can generate the following suggested correspondence: Email from Alice to Bob: ############################ Hello Bob, How are you. I hate to raise the rent because you've been renting from me for a long time, but as you know inflation recently went up by 6 percent. To stay solvent, I need to raise your rent by 6 percent as well. I hope you understand my point of view. What do you think about this? Would you like to continue renting? thank you. Alice

[0117] As part of generating the proposed communications, the system may, for example, iterate through the following email communication as part of one simulation: Email from Bob to Alice: ############################ Hello Alice, Thank you for your email. I appreciate you being upfront with me and I understand your position. It's a tough situation because I'm on a fixed income and this is a lot of money for me. I understand you have business expenses as well. I don't want to put you in a tough position either. Is there anything we can do to find a compromise? thank you. Bob ############################ Email from Alice to Bob: ############################ Hello Bob, Thanks for sharing your thoughts. How about an extra $300? thank you. Alice

[0118] The system can also repeat the following email communication as part of another simulation: Email from Bob to Alice: ############################ Hello Alice, Thank you for your kind words. I have lived here for a long time and have been a good tenant. Would it be possible to continue the lease at the current rent? If so, I would be happy to sign a 5 year lease. thank you. Bob ############################ Email from Alice to Bob: ############################ Hello Bob, Thank you for sharing your thoughts. That's great. I will send you the rental agreement documents for signing. thank you. Alice

[0119] After generating each simulation, the system can score the simulation as described above. For example, if Bob's objective is to minimize rent increases and maintain a friendly relationship with Alice, the system may measure the rent increases, if any, in each simulation, perform a sentiment analysis on Alice's measurements to generate a sentiment score and a rent increase score, and then combine the scores to generate a final simulation score for the simulations.

[0120] The system can then apply a game theory solver to the simulation scores of the pairs of strategies to determine a final strategy and generate communications selected according to the final strategy.

[0121] The system provides the suggested communication for presentation to the first user in a user interface of the user device (step 504).

[0122] The system receives user input selecting the proposed natural language communication, step 506. For example, the user may have selected a user interface element in the user interface that indicates that the proposed communication is acceptable.

[0123] In response, the system transmits the proposed natural language communication to the second actor (step 508), for example, over a data communications network.

[0124] 6 illustrates an example 600 of generating suggested communications using a language model neural network (“LLM”) 110. In the example 600, the interaction is a fruit trade negotiation between “Sarah” and “Sophia” via email communication.

[0125] In example 600, the set of context variables includes a single context variable "tone" that specifies the tone of the natural language communication. Possible values ​​for tone are "assertive" and "calm."

[0126] In the example 600, the strategy samples 602 the value “assertive” with probability p and the value “calm” with probability 1−p. For example, these probabilities may be fixed or may be predicted by a learned model based on the current state 604 of the interaction at the time this proposed communication is being generated. That is, the learned model may process the state 604 of the interaction, e.g., communications already generated in the interaction, or a summary of the interaction as described above, using the learned model to generate the probabilities p and 1−p. For example, the learned model may have been trained using reinforcement learning on a set of training interactions, or using supervised learning using a set of training interactions where each communication achieved a goal labeled with the corresponding values ​​of the context variables used to generate the communication.

[0127] In the example 600, the system selects the value "mild" 606 for the context variable.

[0128] To introduce probability into the generation, the system also uses what are referred to in the example 600 as "chance nodes" 608 to sample seeds for the generation of proposed communications.

[0129] The system then processes the prompt 610, which identifies the context of the proposed communication and the selected value of the seed, using the language model neural network 110 to generate a suggested communication 612 for Sophia that has a “calm” tone but is responsive to previous communications in the interaction.

[0130] In example 600, the prompt also includes "private" information that Sophia has not communicated to Sarah but that is necessary to achieve the interaction objective of maximizing the value of the fruit that Sophia owns at the end of the negotiation. That is, the prompt identifies what fruits Sophia has and how much value she places on each type of fruit.

[0131] This specification uses the term "configured" with respect to systems and computer program components. A system of one or more computers configured to perform a particular operation or action means that the system has installed thereon software, firmware, hardware, or a combination thereof that, during operation, causes the system to perform the operation or action. A computer program or programs configured to perform a particular operation or action means that one or more programs contain instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0132] The subject matter and embodiments of the functional operations described herein can be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware including the structures disclosed herein and their structural equivalents, or one or more combinations thereof. The subject matter described herein can be implemented as one or more computer programs, e.g., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by or control of the operation of a data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or one or more combinations thereof. Alternatively, or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal generated to encode information for transmission to an appropriate receiver device for execution by a data processing device.

[0133] The term "data processing apparatus" refers to data processing hardware and encompasses any kind of apparatus, device, and machine for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. An apparatus may be or further include special purpose logic circuitry, such as, for example, an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). An apparatus may optionally include, in addition to hardware, code that creates an execution environment for a computer program, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0134] A computer program, which may also be referred to or written as a program, software, software application, app, module, software module, script, or code, may be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in a single file dedicated to the program, or in multiple coordinated files, e.g., files storing one or more modules, subprograms, or portions of code, or in part of a file holding other programs or data, e.g., one or more scripts stored in a markup language document. A computer program may be deployed to be executed on one computer, or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0135] As used herein, the term "database" is used broadly to refer to any collection of data, which need not be structured in any particular way, or at all, and which may be stored on storage devices in one or more locations. Thus, for example, an index database may contain multiple collections of data, each of which may be organized and accessed differently.

[0136] Similarly, the term "engine" is used broadly herein to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine is implemented as one or more software modules or components installed on one or more computers at one or more locations. In some cases, one or more computers are dedicated to a particular engine, and in other cases, multiple engines may be installed and executed on the same computer or computers.

[0137] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by special purpose logic circuitry, such as, for example, an FPGA or ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0138] A computer suitable for executing a computer program can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. In general, the central processing unit receives instructions and data from a read-only memory or a random access memory, or both. The essential elements of a computer are a central processing unit for implementing or executing instructions, and one or more memory devices for storing instructions and data. The central processing unit and memory can be supplemented by or incorporated in special-purpose logic circuitry. In general, a computer also includes one or more mass storage devices for storing data, such as, for example, magnetic, magneto-optical, or optical disks, or is operatively coupled to receive data from, transfer data to, or both, the one or more mass storage devices. However, a computer need not have such devices. Furthermore, a computer can be incorporated in another device, such as, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as, for example, a universal serial bus (USB) flash drive, to name just a few examples.

[0139] Computer readable media suitable for storing computer program instructions and data include, by way of example only, all forms of non-volatile memory, media, and memory devices including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD ROM and DVD-ROM disks.

[0140] To provide for user interaction, embodiments of the subject matter described herein can be implemented on a computer having a display device, such as, for example, a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, as well as a keyboard and a pointing device, such as, for example, a mouse or trackball, by which the user can provide input to the computer. Other types of devices can also be used to provide user interaction, for example, feedback provided to the user can be any form of sensory feedback, such as, for example, visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, the computer can interact with the user by sending and receiving documents to and from a device used by the user, such as, for example, by sending a web page to a web browser on the user's device in response to a request received from the web browser. The computer can also interact with the user by sending text messages or other forms of messages to a personal device, such as, for example, a smartphone running a messaging application, and receiving a response message from the user in return.

[0141] A data processing apparatus for implementing machine learning models may also include dedicated hardware accelerator units for handling common and computationally intensive parts of, e.g., machine learning training or production, e.g., inference, workloads.

[0142] The machine learning model can be implemented and deployed using a machine learning framework, for example, the TensorFlow framework or the Jax framework.

[0143] An embodiment of the subject matter described herein can be implemented in a computing system that includes a back-end component, e.g., as a data server, or includes a middleware component, e.g., an application server, or includes a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app, through which a user can interact with an implementation of the subject matter described herein, or includes any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), e.g., the Internet.

[0144] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communications network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server sends data, e.g., HTML pages, to a user device, e.g., to display data to and receive user input from a user interacting with the device acting as a client. Data generated at the user device, e.g., a result of a user interaction, can be received at the server from the device.

[0145] Although the present specification contains many specific implementation details, these are not to be construed as limitations on the scope of any invention or scope that may be claimed, but rather as descriptions of features that may be specific to certain embodiments of a particular invention. Some features described in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described above as acting in some combinations and may initially be claimed as such, in some cases one or more features from the claimed combination may be deleted from the combination, and the claimed combination may be directed to a subcombination, or a variation of the subcombination.

[0146] Similarly, although operations are illustrated in the figures and described in the claims in a particular order, this should not be understood as requiring such operations to be performed in the particular order shown, or in sequential order, or that all of the illustrated operations be performed to achieve a desired result. In some situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be incorporated together in a single software product or packaged in multiple software products.

[0147] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. [Explanation of symbols]

[0148] 100 Neural Network System 110 Language Model Neural Networks 104 Recommended Communication 120 Multi-Actor Interaction 132 First Actor 140 Communications 200 processes 300 processes 400 processes 500 processes 600 Examples 604 Status 606 Value: "Calm" 608 Chance Nodes 610 Prompt 612 Proposed Correspondence

Claims

1. A method implemented by one or more computers, comprising: During interactions between multiple actors, identifying a first plurality of first strategies for selecting between first parameterizations relating to a communication of a first actor of the plurality of actors for the interaction, each first parameterization parameterizing a set of context variables relating to the communication of the first actor during the interaction; identifying a second plurality of second strategies for selecting between second parameterizations relating to a communication of a second actor of the plurality of actors for the interaction, each second parameterization parameterizing a set of context variables relating to the communication of the second actor during the interaction; For each of a plurality of pairs each including a respective first strategy and a respective second strategy, generating one or more simulations of the interaction assuming that the first actor communicates according to the first strategy and the second actor communicates according to the second strategy using a language model neural network; determining a pair score from the one or more simulations indicative of the extent to which the first actor communicating according to the first strategy and the second actor communicating according to the second strategy fulfill an interaction objective; selecting a first parameterization for the first actor using the scores of the plurality of pairs; generating a proposed communication for the first actor by processing inputs that condition the language model neural network on the selected first parameterization; A method comprising:

2. the first actor being a user of a user device, and the method further comprising: providing the proposed communication for presentation to the first user in a user interface of the user device. The method of claim 1, further comprising:

3. during the interaction, the communication generated by the first actor is a natural language communication; The method of claim 1 or 2, wherein the suggested communication is a suggested natural language communication.

4. the first actor being a user of a user device, and during the interaction the first user receiving a communication from the second actor and sending a communication to the second actor via a user interface of the user device, the method comprising: The method of claim 3 , further comprising providing the suggested natural language communication for presentation to the first user in the user interface of the user device.

5. receiving a user input selecting the proposed natural language communication; in response, transmitting the proposed natural language communication to the second actor; 5. The method of claim 4, further comprising:

6. selecting a first parameterization using the scores of the plurality of pairs, 6. The method of claim 1, further comprising applying a game theory solver to the representation of the interaction including the scores of the plurality of pairs to identify a first parameterization that is optimal for the first user.

7. generating one or more simulations of the interaction using a language model neural network, the step including, for each simulation and in each of a plurality of iterations: identifying prior communications between the first actor and the second actor during the simulation; selecting the first actor or the second actor as a current actor for the iteration; When the first actor is selected as the current actor, selecting a first parameterization for said iteration according to said first strategy; using the language model neural network to process (i) inputs that condition the language model neural network based on the selected first parameterization, and (ii) at least a portion of the previous communications during the simulation, to generate communications of the first actor; When the second actor is selected as the current actor, selecting a second parameterization for the iteration according to the second strategy; using the language model neural network to process (i) inputs that condition the language model neural network based on the selected second parameterization, and (ii) at least a portion of the previous communications during the simulation, to generate communications of the second actor; 7. The method of any one of claims 1 to 6, comprising:

8. The method according to claim 1 , wherein the set of context variables comprises one or more context variables defining a communication style of the communication.

9. 9. The method of claim 1, wherein the set of context variables comprises one or more context variables that define a set of information available as context to the first actor or the second actor.

10. 10. The method of claim 1, wherein one or more of the first strategies select the same first parameterization for each communication during the interaction.

11. 11. The method of claim 1, wherein one or more of the first strategies select different first parameterizations for different communications during the interaction.

12. receiving user input specifying a context variable; processing inputs defining the context variables using the language model neural network to generate one or more possible values ​​for the context variables that can be used to parameterize the context variables; 12. The method of claim 1, further comprising:

13. determining, from the one or more simulations, a pair score indicative of the extent to which the first actor communicating according to the first strategy and the second actor communicating according to the second strategy fulfill the interaction objective, comprising, for each of the one or more simulations: generating a summary input from one or more of the communications generated during the simulation; generating a criteria input characterizing the purpose of the interaction; processing the summary input and the criteria input using a second language model neural network to generate an output defining a simulation score for the simulation indicative of the extent to which the simulation satisfies the one or more criteria; 13. The method of any one of claims 1 to 12, comprising:

14. determining, from the one or more simulations, a pair score indicative of the extent to which the first actor communicating according to the first strategy and the second actor communicating according to the second strategy fulfill the interaction objective, comprising: for each of the one or more simulations: processing input including one or more of the communications generated during the simulation using a second language model neural network to generate a summary of the simulation; generating a simulation score for the simulation from the summary that indicates the extent to which the simulation meets the one or more criteria; 14. The method of any one of claims 1 to 13, comprising:

15. generating a simulation score for the simulation from the summary indicative of the extent to which the simulation satisfies the one or more criteria; The method of claim 14 comprising applying one or more heuristics to the summary to generate the simulation score.

16. generating a simulation score for the simulation from the summary indicative of the extent to which the simulation satisfies the one or more criteria; generating a criteria input characterizing the purpose of the interaction; processing the summary input and the criteria input using the second language model neural network to generate an output defining the simulation score; 16. The method of claim 14 or 15, comprising:

17. 17. The method of claim 13, wherein the second language model neural network is the language model neural network.

18. One or more computers; one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform each of the operations of the method according to any one of claims 1 to 17; A system comprising:

19. One or more computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform each of the operations of the method of any one of claims 1 to 17.

Images