Generate suggested communications by simulating interactions using language model neural networks

Abstract

To provide methods, systems and recording mediums for generating suggested communications during a multi-agent interaction using a language model neural network.SOLUTION: A method comprises: identifying multiple first and second strategies for selecting between parameterizations for communications for first and second actors; parametrizing a set of context variables for the communications of the actors during an interaction; for each of multiple pairs including respective first and second strategies: generating simulations of the interaction using a language model neural network; determining, from the simulations, a score for the pair that indicates a degree to which the first and second actors satisfy objectives for the interaction; selecting, using the scores for the multiple pairs, a parametrization for the first actor; and generating, by processing an input that conditions the language model neural network on the selected parameterization, a suggested communication for the first actor.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] This specification relates to processing data using machine learning models. [Background technology]

[0002] As an example, a neural network is a machine learning model that uses one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as an input to another layer in the network, e.g., the next hidden layer or the output layer. Each layer of the network generates an output from the received input according to the current values ​​of its respective set of weights. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, LA Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language models, arXiv preprint arXiv:2203.15556, 2022 [Non-patent document 2] JW Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, HF Song, J. Aslanides, S. Henderson, R. Ring, S. Young, E. Rutherford, T. Hennigan, J. Menick, A. Cassirer, R. Powell, G. van den Driessche, LA Hendricks, M. Rauh, P. Glag, J. P. Huang, J. Huang, R. Ring, S. Young, E. Welbl, S. Dathathri, S. Huang, J. Uesato, J. Mellor, I. Higgins, A. Creswell, N. McAleese, A.Wu, E. Elsen, SM Jayakumar, E. Buchatskaya, D. Budden, E. Sutherland, K. Simonyan, M. Paganini, L. Sifre, L. Marten, A. Li, A. Kunco, A. Kunco. Nematzadeh, E. Gribovskaya, D. Donato, A. Lazaridou, A. Mensch, J. Lespiau, M. Tsimpoukelli, N. Grigorev, D. Fritz, T. Sottiaux, M. Pajarskas, T. Pohlen, Z. Gong, D. Toyama, C. de Masson d'Autu, Y. Milik, T. V. Mili, V. Y. Babuschkin, A. Clark, D. de Las Casas, A. Guy, C. Jones, J. Bradbury, M. Johnson, BA Hechtman, L. Weidinger, I. Gabriel, WS Isaac, E. Lockhart, S. Osindero, L. Rimell, J. Dyer, O. Vinyals, K. Ayoub, J. L. Stanway, K. Bennett, D. K. Bennett, Hassac. Kavukcuoglu, and G. Irving.Scaling language models: Methods, analysis & insights from training gopher. CoRR, abs / 2112.11446, 2021. [Non-patent document 3] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019 [Non-patent document 4] Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. Towards a human-like open-domain chatbot. CoRR, abs / 2001.09977, 2020 [Non-Patent Document 5] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020 Summary of the Invention [Problem to be solved by the invention]

[0004] This specification describes a system, implemented as a computer program on one or more computers at one or more locations, that generates proposals for a given actor in a multi-actor interaction by simulating the interaction using a language model neural network. [Means for solving the problem]

[0005] Particular embodiments of the subject matter described in this specification can be implemented to realize one or more of the following advantages.

[0006] This specification describes techniques for generating suggested communications for actors in a multi-actor interaction. As an example, a multi-actor interaction can be a communication session in which a user exchanges natural language communications with one or more other actors, e.g., other users, software agents, or both. Thus, in this example, a system can generate suggested natural language communications, e.g., email messages, text messages, chat messages, etc., to be sent by the user to one or more other agents.

[0007] For example, some existing techniques, such as auto-completion or next turn prediction techniques, attempt to predict the next communication in an interaction by taking into account previous communications, and possibly previous interactions in which the user has been involved.

[0008] However, such approaches do not take into account that different interactions may have different objectives, and that different suggestions may be more appropriate than others, even if previous communications are the same. Furthermore, these approaches generate short-term predictions that do not consider the long-term impact on a given interaction of subsequent communications in the interaction, e.g., the impact of a given communication on communications received from other agents.

[0009] In contrast, the present specification generates proposed communications using an entirely different paradigm. In particular, the present specification generates proposals by using a language model neural network to evaluate multiple different parameterizations of a set of communication context variables for both the “first” actor for whom the proposal is being generated and other actors in the interaction, and then selecting the parameterization that results in the objective of the interaction being met. To do so, the system can generate simulations of the interactions using the language model neural network and then score each of those simulations to determine the extent to which the simulated interaction met the objective of the interaction.

[0010] As a result, the system leverages the power of language model neural networks to effectively "model" the impact of different parameterizations of context variables on the remainder of the interaction, resulting in suggested communications that are more effective and tailored to the current interaction, improving the user experience.

[0011] Furthermore, by running simulations of the current interaction at different time steps within the current interaction, the system can effectively adjust its strategy for generating suggested communications as the current interaction progresses, further improving the relevance and usefulness of the generated suggested communications and further enhancing the user experience.

[0012] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description and drawings. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram of an exemplary neural network system. [Figure 2] FIG. 1 is a flow diagram of an exemplary process for generating suggested communications using a language model neural network. [Figure 3] FIG. 1 is a flow diagram of an exemplary process for generating a simulation of a pair of strategies. [Figure 4] FIG. 1 is a flow diagram of an exemplary process for generating a score for a simulation. [Figure 5] FIG. 1 is a flow diagram of an example process for generating suggested communications for a user during a communication session. [Figure 6] FIG. 1 illustrates an example of using a language model neural network to generate suggested communications. DETAILED DESCRIPTION OF THE INVENTION

[0014] Like reference numbers and designations in the various drawings indicate like elements.

[0015] 1 is a diagram of an exemplary neural network system 100. Neural network system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations in which the systems, components, and techniques described below may be implemented.

[0016] The system 100 generates proposed communications 104 for a first actor 132 in a multi-actor interaction 120 by simulating the interaction using a language model neural network 110 .

[0017] The language model neural network 110 is a neural network configured to process an input and generate an output that includes a probability distribution over a set of text tokens in a vocabulary of text tokens, where the probability of each token represents the likelihood that the text token immediately follows the input. The tokens in the vocabulary may include text tokens such as, for example, characters, subwords, word pieces, etc., and may optionally include tokens that represent other types of data, such as, for example, audio, images, video, or other sensor data.

[0018] For example, the language model neural network may be an autoregressive language model neural network.

[0019] A language model neural network is called an autoregressive neural network because the neural network autoregressively generates an output sequence of tokens by generating each particular token in the output sequence conditional on the current input sequence, including any tokens that precede the particular text token in the output sequence, i.e., tokens already generated for any previous position in the output sequence that precedes the particular position of the particular token, and on a context input ("context sequence") that provides the context for the output sequence.

[0020] For example, the current input sequence when generating a token at any given position in the output sequence may include the context sequence and any preceding tokens that precede the given position in the output sequence. As a particular example, the current input sequence may include the context sequence followed by any preceding tokens that precede the given position in the output sequence. Optionally, the context and the current output sequence may be separated by one or more predetermined tokens in the current input sequence.

[0021] More specifically, to generate a particular token at a particular position in a candidate output sequence, the neural network can process the current input sequence to generate a score distribution, e.g., a probability distribution, that assigns a respective score, e.g., a respective probability, to each text token in a vocabulary of text tokens. The neural network can then use the score distribution to select a text token from the vocabulary as the particular token. For example, the neural network can greedily select the token that receives the highest score, or it can sample a token from the distribution, e.g., using kernel sampling or another sampling technique.

[0022] As a particular example, the language model neural network may be an autoregressive transformer-based neural network that includes (i) multiple attention blocks, each applying a self-attention operation, and (ii) an output sub-network that processes the output of the last attention block to generate a score distribution.

[0023] The neural network can have any of a variety of transformer-based neural network architectures. Examples of such architectures include J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language models, arXiv preprint arXiv:2203.15556, 2022, J.W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, H. F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young, E. Rutherford, T. Hennigan, J. Menick, A. Cassirer, R. Powell, G. van den Driessche, L. A. Hendricks, M. Rauh, P. Huang, A. Glaese, J. Welbl, S. Dathathri, S. Huang, J. Uesato, J. Mellor, I. Higgins, A. Creswell, N. McAleese, A.Wu, E. Elsen, S. M. Jayakumar, E. Buchatskaya, D. Budden, E. Sutherland, K. Simonyan, M. Paganini, L. Sifre, L. Martens, X. L. Li, A. Kuncoro, A. Nematzadeh, E. Gribovskaya, D. Donato, A. Lazaridou, A. Mensch, J. Lespiau, M. Tsimpoukelli, N. Grigorev, D. Fritz, T. Sottiaux, M. Pajarskas, T. Pohlen, Z. Gong, D. Toyama, C. de Masson d'Autume, Y.Li, T. Terzi, V. Mikulik, I. Babuschkin, A. Clark, D. de Las Casas, A. Guy, C. Jones, J. Bradbury, M. Johnson, B. A. Hechtman, L. Weidinger, I. Gabriel, W. S. Isaac, E. Lockhart, S. Osindero, L. Rimell, C. Dyer, O. Vinyals, K. Ayoub, J. Stanway, L. Bennett, D. Hassabis, K. Kavukcuoglu, and G. Irving. Scaling language models: Methods, analysis & insights from training gopher. CoRR, abs / 2112.11446, 2021、Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019、Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. Towards a human-like open-domain chatbot. CoRR, abs / 2001.09977, 2020, and Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Examples include those described in arXiv preprint arXiv:2005.14165, 2020.

[0024] In general, however, a Transformer-based neural network includes a sequence of attention blocks, where, during processing of a given input sequence, each attention block in the sequence receives a respective input hidden state for each input token in the given input sequence. The attention block then updates at least the hidden state of the last token in the given input sequence by, at least in part, applying self-attention to generate a respective output hidden state for the last token. The input hidden state of the first attention block is the embedding of the input token in the input sequence, and the input hidden state of each subsequent attention block is the output hidden state generated by the preceding attention block.

[0025] In this example, the output sub-network processes the output hidden state produced by the last attention block in the sequence for the last input token in the input sequence to produce a score distribution.

[0026] Typically, before using neural network 100, system 100 or another training system pre-trains language model neural network 110 on a language modeling task, such as, for example, a task requiring predicting the next token following a current sequence in training data, given a current sequence of text tokens. As a particular example, the language model neural network may be pre-trained on a maximum likelihood objective on a large dataset of text, such as publicly available text from the Internet or another text corpus. In some cases, system 100 or another training system then further trains language model neural network 110 on one or more objectives, including, for example, supervised fine-tuning, reinforcement learning, instruction tuning, etc.

[0027] In some implementations, the language model neural network 110 is fine-tuned to perform the tasks described below. In other implementations, the language model neural network 110 is not further trained; for example, any given input to the neural network 110 can be made to generate an appropriate output by including natural language instructions, few-shot prompts, or a combination of the two.

[0028] As described above, the system 100 uses a language model neural network 110 to generate suggested communications for a first actor 132, which is one of multiple actors 130 in a multi-actor interaction 120.

[0029] A multi-actor interaction 120 may be any suitable interaction in which multiple actors 130 send communications 140 to one another.

[0030] For example, multi-actor interaction 120 may be a communication session in which multiple actors 130 exchange electronic messages, e.g., emails, chats, SMS messages, MMS messages, etc. In this example, communication 140 is generally natural language communication that is all or part of the electronic message.

[0031] In this example, the "first actor" 132 wanting the suggestion may be, for example, a user of a user device, such as a "first user," and the system 100 may provide the suggested communication for presentation to the first user in a user interface of the user device. Other actors may include other users, automated agents, or a combination of both.

[0032] In this example, the first user can specify one or more goals for the interaction. For example, the first user can submit natural language input specifying the goal of the interaction. As another example, the first user can submit input selecting one or more predefined goals for the interaction from a set of possible goals, or can fill out a form specifying the goal.

[0033] As another example, the system 100 may use the language model neural network 110 (or another model) to process one or more initial communications from an interaction to generate one or more candidate goals for the interaction, and the user 132 may submit input selecting one or more of the candidate goals as the final goal of the interaction.

[0034] As yet another example, system 100 can automatically identify a purpose for an interaction based on the context of the interaction and, optionally, one or more initial communications from the interaction. For example, system 100 can identify the purpose based on the communication for the interaction and the software application or website used to send the initial communication; for example, when the interaction includes a website for making travel reservations and the initial communication identifies a date range, the system can identify that the purpose of the interaction is to make travel reservations within the specified date range.

[0035] The objectives can be based, for example, on the emotions of the communications of the first user, the other agent, or both, i.e., one or more objectives can specify target emotions of the communications of the first user, the other agent, or both, during the interaction.

[0036] As another example, the objective may include one or more objectives based on a goal of the first user regarding the interaction. For example, the objective may include solving a problem the first user has, taking a specified action, or achieving another type of goal of the first user regarding the interaction.

[0037] The communication session may be any suitable communication session regarding any suitable topic, such as, for example, a customer support session, a negotiation, a contest, a scheduling or logistics session, a collaboration session, etc.

[0038] In another example, multi-agent interaction 120 may be a communication session between mechanical, electronic, or electromechanical agents, such as, for example, robots, autonomous vehicles, or other agents. In this example, the communication may be natural language communication, computer code segments, or communication in a domain-specific language that the agents can parse. Examples of such communication may include communication for resource allocation or distribution, scheduling or logistics, etc.

[0039] In this example, the "first actor" 132 needing a proposal may be one of the agents.

[0040] In this example, a user of system 100 may specify one or more objectives for an interaction associated with a first user. For example, the objectives may be based on, for example, the amount of resources allocated to the first agent as a result of the interaction, the navigation time to a destination for the first agent as a result of the interaction, etc.

[0041] More specifically, the system 100 uses a language model neural network 110 to generate a respective proposed communication for the first actor 132 at each of a plurality of time steps during the interaction 120 .

[0042] The generation of the proposed communication is described in more detail below.

[0043] When the first actor 132 is a user, the user can decide whether to submit the proposed communication as a communication to one or more of the other actors 130. When the first actor 132 is an agent, the system 100 can automatically submit the communication to one or more of the other actors 130 as an agent's communication, or a control system can decide whether to submit the proposed communication to one or more of the other actors 130.

[0044] 2 is a flow diagram of an exemplary process 200 for generating proposed communications at a given time step during an interaction between multiple actors. For convenience, process 200 is described as being performed by one or more computer systems located at one or more locations. For example, a neural network system suitably programmed in accordance with this specification, such as neural network system 100 shown in FIG. 1, can perform process 200.

[0045] The system may perform process 200 at various points during an interaction. For example, the system may perform process 200 each time a first actor of multiple actors requests a suggestion from the system. For example, a user may select a user interface element when viewing recent communications in an interaction to request a suggested response communication.

[0046] The system identifies a first plurality of first strategies for selecting among a first parameterization of a first actor's communications (step 202).

[0047] Generally, each first parameterization parameterizes a set of one or more context variables related to the communication of the first actor during the interaction.

[0048] That is, each first parameterization includes a respective value for a set of one or more context variables that each affect how the first actors communicate during the communication.

[0049] For example, one or more of the context variables in the set may define a communication style for the first actor's communication. Examples of such context variables include the tone of the communication, the emotion, the argumentative style, the writing style, the use of literary techniques, etc.

[0050] As another example, one or more of the context variables in the set can define the set of information available as context to the first actor during the interaction, i.e., each of these context variables can determine how much external information (other than the communications exchanged during the interaction) is available to the first user during the communication.

[0051] In some cases, the possible values ​​of each context variable are a discrete set of possible values, i.e., the context variable has a fixed, discrete set of possible values. For example, for a communication tone context variable, the possible values ​​may be a fixed set that includes some or all of: formal, informal, optimistic, cooperative, friendly, funny, satirical, assertive, cheerful, angry, etc.

[0052] In other cases, the possible values ​​of one or more of the context variables are continuous: for example, a tone variable in a communication context may have possible values ​​selected from a continuous range between 0 and 1, with 0 representing an informal tone and 1 representing a formal tone.

[0053] In some cases, all of the first strategies may assign the same parameterization to all communications during an interaction, i.e., each first strategy may assign the same value to the context variables of all communications of the first agent during the entire interaction.

[0054] In other cases, one or more of the first strategies assign different parameterizations to different communications, for example, alternating between different parameterizations or cycling through a set of three or more parameterizations.

[0055] The system also identifies a second plurality of second strategies for selecting among a second parameterization regarding communication of a second actor of the plurality of actors for the interaction (step 204).

[0056] Each second parameterization parameterizes a set of context variables related to the communication of the second actor during the interaction.

[0057] The strategy, set of context variables, and parameterization may be the same or different for the first and second actors.

[0058] In general, a set of context variables, and possibly a parameterization of actors, can be received as input by the system.

[0059] In some implementations, the system can use a language model neural network, such as the language model neural network described above, or a different language model neural network, such as a smaller, more computationally efficient language model neural network, to generate or augment the possible values ​​for a given one of the context variables.

[0060] For example, the system may receive initial input specifying a context variable and, optionally, one or more possible values ​​for the context variable, and may then use a language model neural network to process the input specifying the context variable and, if available, one or more possible values ​​to generate one or more additional possible values. That is, given an input identifying a context variable, the system may use a neural network to generate suggested possible values ​​for the context variable.

[0061] In some implementations, the system can generate a strategy for a given actor by selecting all possible parameterizations of a set of context variables, by randomly selecting a fixed number of parameterizations from the set of possible parameterizations, or by using some other selection scheme.

[0062] In some other implementations, in addition to specifying context variables and parameterization, the user input may also identify a strategy for the first user, the second user, or both.

[0063] For each of a plurality of pairs, each including a respective first strategy and a respective second strategy, the system uses the language model neural network to generate one or more simulations of an interaction assuming that the first actor communicates according to the first strategy and the second actor communicates according to the second strategy (step 206).

[0064] For example, the system may select all possible pairs of the first strategy and the second strategy, or may randomly select a subset of the possible pairs of the first strategy and the second strategy, e.g., randomly select the number of pairs that can be simulated within the latency budget to generate the selected communication.

[0065] In some cases, the system generates a single simulation for each pair. In other cases, the system can generate multiple different simulations for the same pair because the language model neural network can generate different output sequences for the same input sequence, as described above.

[0066] The generation of simulations using language model neural networks is described in more detail below with reference to FIG.

[0067] The system then determines pairwise scores from the one or more simulations that indicate the extent to which a first actor communicating according to a first strategy and a second actor communicating according to a second strategy fulfill one or more objectives of the interaction (step 208).

[0068] For example, as mentioned above, the objective may be an objective received as input by the system and to be achieved by a first actor.

[0069] The determination of the score for a pair of strategies is described in more detail below.

[0070] When there is more than one simulation, the system may determine a respective simulation score for each simulation and then, for example, average, sum, or otherwise combine the simulation scores to generate a final score for the pair.

[0071] The system then uses the scores of the multiple pairs to select a first parameterization for the first actor (step 210).

[0072] For example, the system may generate a representation of the interaction that includes pairwise scores and then apply a game theory solver to the representation to identify an optimal first parameterization for the first actor. Examples of such representations include payoff matrices, payoff tensors, game trees, etc.

[0073] Examples of game theory solvers that can be used include the standard game (NFG) solver, the evolving game (EFG) solver, and the multi-agent reinforcement learning (MARL) solver. An example of a MARL solver is the policy space response oracle (PSRO) solver.

[0074] Thus, by applying a game theory solver, a first parameterization is identified that, based on the results of the simulation, is most likely to satisfy the interaction objectives when used to generate the first agent's communications in the current step of the interaction.

[0075] The system then generates proposed communications for the first actor by processing inputs that condition the language model neural network on the selected first parameterization (step 212). That is, the system generates proposed communications characterized by the first parameterization given the context of the interaction up to the given time step.

[0076] More specifically, the system can identify a selected parameterization, process input including the context of the interaction up to a given time step, and instructions to generate a proposed communication parameterized by the selected parameterization based on the given context using a language model neural network, to generate the proposed communication.

[0077] The input may also optionally include additional information.

[0078] For example, the input may include information characterizing the first agent or the first agent's objectives for the interaction. This information may include "private" information not available from messages of previous interactions. For example, when the first agent is attempting to arrange a meeting, the first agent may not have communicated the first agent's preferred meeting time during the interaction. As another example, when the first agent is attempting to negotiate the price of a service or product, the first agent may not have communicated the first user's target price for the service or product.

[0079] The system may then provide the proposed communication, for example, to the first actor's control system or for presentation to the first actor.

[0080] 3 is a flow diagram of an exemplary process 300 for generating a simulation of the interaction of a pair of strategies using a language model neural network. For convenience, process 300 is described as being performed by one or more computer systems located at one or more locations. For example, a neural network system suitably programmed in accordance with this specification, such as neural network system 100 shown in FIG. 1, can perform process 300.

[0081] For example, given a pair including a respective first strategy and a respective second strategy, the system can execute process 300 to generate a simulation of an interaction assuming that the first actor communicates according to the first strategy and the second actor communicates according to the second strategy.

[0082] In particular, the system can execute process 300 in each of multiple iterations to generate a respective communication for each iteration. The communications across the iterations then act as a simulation of the interaction starting from the current point in the interaction.

[0083] The system identifies previous communications between the first and second actors during the simulation, i.e., communications generated in any previous iterations during the generation of the simulation (step 302). Optionally, when the simulation is being generated after the interaction has already begun, the system can also identify previous actual communications generated during the interaction before the generation of the simulation began.

[0084] The system selects either the first actor or the second actor as the current actor for the iteration, step 304. Generally, the system selects the first actor and the second actor according to a model of how the actors will interact during the interaction being simulated.

[0085] For example, the model may indicate that two actors communicate alternately during an interaction. In this example, the system may select a first actor if the second actor generated the most recently identified communication, or may select a second actor if the second actor generated the most recently identified communication.

[0086] As another example, the model may indicate that each communication is sent by a first actor unless a recent communication by the first actor meets certain criteria, e.g., does not include a request for a response from the second actor. In this example, the system may select a second actor only if the recent communication was from the first actor and met certain criteria.

[0087] As yet another example, the model may indicate that in any given iteration, a first actor generates a communication with probability p and a second actor generates a communication with probability 1-p. In this example, the system selects the first actor with probability p and the second actor with probability 1-p.

[0088] Once the first actor is selected as the current actor, the system selects a first parameterization for the iteration according to a first strategy (step 306), i.e., the system applies the first strategy to the identified communication to determine the first parameterization.

[0089] The system then processes (i) inputs that condition the language model neural network based on the selected first parameterization and (ii) at least a portion of previous communications during the simulation using the language model neural network to generate communications for the first actor (step 308). For example, the inputs may include instructions for generating communications parameterized by the selected first parameterization, taking into account a description of the selected first parameterization and context provided by the previous communications. As noted above, the inputs to the language model neural network may also optionally include actual, previous communications from earlier in the actual interaction. Additionally, as noted above, the inputs may also include "private" information characterizing the first actor or the state of the interaction.

[0090] Once the second actor is selected as the current actor, the system selects a second parameterization for the iteration according to a second strategy (step 310), i.e., the system applies the second strategy to the identified communications to determine the second parameterization.

[0091] The system then uses the language model neural network to process (i) inputs that condition the language model neural network based on the selected second parameterization and (ii) at least a portion of previous communications during the simulation to generate communications for the second actor (step 312). For example, the inputs may include instructions for generating communications parameterized by the selected second parameterization given a description of the selected second parameterization and context provided by the previous communications. As noted above, the inputs to the language model neural network may also optionally include actual previous communications from earlier in the actual interaction. The system may not have access to "private" information for the second actor. In this case, the system may omit this information from the input, include an indication that the information is unavailable, or randomly sample the information from a set of plausible characteristics.

[0092] Thus, by repeatedly executing iterations of process 300, the system generates a simulation that includes a respective communication from one of the actors in each of multiple iterations.

[0093] The system continues to execute iterations of process 300 until a termination criterion is met.

[0094] For example, the system may determine that the termination criteria have been met when a predetermined number of iterations have been performed.

[0095] As another example, the system may determine that the termination criteria have been met when the communications generated during an iteration satisfy a specified pattern of communications that signifies the end of the interaction.

[0096] As yet another example, in each iteration, the system may use a language model neural network to process input including communications generated during the iteration and a prompt asking the language model neural network whether the interaction has ended. The system may then determine that the termination criteria have been met when the output of the language model neural network indicates the end of the interaction. For example, the input may also include data characterizing the goal of the interaction and may ask whether the goal has been met. As another example, if the input also includes data characterizing the goal, the input may also ask whether the goal cannot be achieved given the current state of the interaction.

[0097] 4 is a flow diagram of an exemplary process 400 for determining a score for a simulation of the interaction of a pair of strategies using a language model neural network. For convenience, process 400 is described as being performed by one or more computer systems located at one or more locations. For example, a neural network system suitably programmed in accordance with this specification, such as neural network system 100 shown in FIG. 1, can perform process 400.

[0098] The system generates summary input from one or more of the communications generated during the simulation (step 402).

[0099] For example, the summary input may include all communications generated during the simulation. As another example, the summary input may include less than all communications generated during the simulation, such as, for example, every nth communication during the simulation, where n is an integer greater than 1, or the communications for the kth last iteration during the simulation, where k is an integer greater than or equal to 1.

[0100] As another example, the summary input may be an abstract summary of the communication. To generate the abstract summary, the system may process the communication generated during the simulation and appropriate prompts using a second language model neural network.

[0101] The "second" language model neural network may be the same as the language model neural network described above, or may be a different, smaller language model neural network.

[0102] The system generates a reference input that characterizes the purpose of the interaction, step 404. For example, the reference input can be a natural language description of the purpose of the interaction.

[0103] The system processes the summary input and the criteria input using a second language model neural network to generate an output defining a simulation score for the simulation indicating the degree to which the simulation meets one or more criteria (step 406).

[0104] For example, the system may process a summary input, a criteria input, and instructions to score the interaction defined by the summary input given criteria described in the criteria input using a second language model neural network to cause the second language model neural network to generate a summary score.

[0105] As an example, the score may be a probability, log-probability, logit, or other likelihood assigned by the second language model neural network to a predetermined response phrase, e.g., the phrase "yes" or the phrase "no."

[0106] For example, in this example, the input to the second language model neural network may include a query asking the second language model neural network whether the interaction summarized by the summary input meets criteria. For example, the input may be in the form "Given this summary of the interaction [summary input], does the interaction meet these criteria: [criteria input]?" As another example, the input may include k examples of the form "Summary: [example summary input]; Criteria: [example criteria input]; Does summary meet criteria? [yes / no]," where k is 1 or greater, followed by "Summary: [summary input]; Criteria: [criteria input]; Does summary meet criteria?"

[0107] As another example, the score may be directly regressed as one or more output tokens by the second language model neural network in response to processing the input, i.e., upon instruction, the language model neural network can process the input and directly output the score.

[0108] For example, in this example, the input to the second language model neural network may include a query requesting the second language model neural network to score the degree to which a summary input satisfies a criteria specified by a criteria input. For example, the input may be in the form "Given this summary of the interaction [summary input] and the following criteria: [criteria input], output a score from 0 to 1 that assesses the degree to which the criteria is met." As another example, the input may include k examples of the form "Summary: [example summary input]; Criteria: [example criteria input]; Criteria satisfaction score: [example score]," where k is 1 or greater, followed by "Summary: [summary input]; Criteria: [criteria input]; Criteria satisfaction score:."

[0109] Alternatively, instead of performing steps 404 and 406, the system can receive a summary input, e.g., an extracted summary generated by a language model neural network, or data specifying a set of heuristics to be applied to a combination of one or more communications. The system can then apply the set of heuristics to the summary input to generate a simulation score.

[0110] 5 is a flow diagram of an exemplary process 500 for generating suggested communications for a user during an interaction with another agent. For convenience, process 500 is described as being performed by one or more computer systems located at one or more locations. For example, a neural network system suitably programmed in accordance with this specification, such as neural network system 100 shown in FIG. 1, can perform process 500.

[0111] For example, the system may perform process 500 during an interaction in which a user of a user device is participating in an interaction, e.g., a communication session, with another agent, e.g., another user of another user device or a software agent.

[0112] The communication session may be any suitable communication session regarding any suitable topic, such as, for example, a customer support session, a negotiation, a contest, a scheduling or logistics session, a collaboration session, etc.

[0113] The communication session may be carried out via any suitable software application, for example, as part of an email exchange, as part of an online chat, or as part of a text messaging exchange.

[0114] The system generates suggested communications for the first user (step 502). For example, the system can use a language model neural network, as described above, to generate suggested communications for the first user.

[0115] As a particular example, the system may receive a request for a proposed communication from a user taking into account existing communications within a session and the context of the session, which generally specifies one or more purposes of the session and, optionally, additional information about the session, e.g., information about other agents in the session.

[0116] For example, consider an email conversation between Bob (a tenant) and Alice (a landlord). After an initial email is sent from Alice informing Bob that the rent must be increased, the system can generate the following suggested correspondence: Email from Alice to Bob: ############################ Hi Bob, How are you. I hate to raise the rent because you've been renting my house for a long time, but as you know, inflation recently rose by 6 percent. To stay solvent, I'll need to raise your rent by 6 percent as well. I hope you understand my point of view. What do you think about this? Would you like to continue renting? thank you. Alice

[0117] As part of generating the proposed communications, the system may, for example, iterate the following email communication as part of one simulation: Email from Bob to Alice: ############################ Hello Alice, Thank you for your email. I appreciate your candor and understand your position. I'm on a fixed income and this is a lot of money for me, so it's a tough situation. I understand you have business expenses as well. I don't want to put you in a tough position either. Is there anything we can do to find a compromise? thank you. Bob ############################ Email from Alice to Bob: ############################ Hi Bob, Thanks for sharing your thoughts. How about an extra $300? thank you. Alice

[0118] The system can also repeat the following email communication as part of another simulation: Email from Bob to Alice: ############################ Hello Alice, Thank you for your kind words. I've lived here for a long time and have been a good tenant. Would it be possible for me to continue the lease at the current rent? If so, I'd be happy to sign a five-year lease. thank you. Bob ############################ Email from Alice to Bob: ############################ Hi Bob, Thank you for sharing your thoughts. That's great. I'll send you the lease documents to sign. thank you. Alice

[0119] After generating each simulation, the system can score the simulation as described above. For example, if Bob's objective is to minimize rent increases and maintain a friendly relationship with Alice, the system can measure the rent increases, if any, in each simulation, perform a sentiment analysis on Alice's measurements, generate a sentiment score and a rent increase score, and then combine the scores to generate a final simulation score for the simulation.

[0120] The system can then apply a game theory solver to the simulation scores of the pairs of strategies to determine a final strategy and generate communications selected according to the final strategy.

[0121] The system provides the suggested communication for presentation to the first user in a user interface of the user device (step 504).

[0122] The system receives user input selecting the proposed natural language communication, step 506. For example, the user may have selected a user interface element in the user interface that indicates that the proposed communication is acceptable.

[0123] In response, the system transmits the proposed natural language communication to the second actor, for example, over a data communications network (step 508).

[0124] 6 illustrates an example 600 of generating suggested communications using a language model neural network (“LLM”) 110. In example 600, the interaction is a fruit trade negotiation between “Sarah” and “Sophia” via email communication.

[0125] In example 600, the set of context variables includes a single context variable "tone" that specifies the tone of the natural language communication. Possible values ​​for tone are "assertive" and "gentle."

[0126] In example 600, the strategy samples 602 the value “assertive” with probability p and the value “calm” with probability 1−p. For example, these probabilities may be fixed or may be predicted by a trained model based on the current state 604 of the interaction at the time the proposed communication is being generated. That is, the trained model may process the state 604 of the interaction, such as communications already generated in the interaction or a summary of the interaction as described above, to generate the probabilities p and 1−p. For example, the trained model may have been trained using reinforcement learning on a set of training interactions, or using supervised learning using a set of training interactions where each communication achieved a goal and was labeled with the corresponding values ​​of the context variables used to generate the communication.

[0127] In example 600, the system selects the value "gentle" 606 for the context variable.

[0128] To introduce probability into the generation, the system also uses what are referred to in the example 600 as "chance nodes" 608 to sample seeds for the generation of proposed communications.

[0129] The system then uses the language model neural network 110 to process the prompt 610, which identifies the context of the proposed communication and the selected value of the seed, to generate a suggested communication 612 for Sophia that has a "calm" tone but is responsive to previous communications in the interaction.

[0130] In example 600, the prompt also includes "private" information that Sophia has not communicated to Sarah but that is necessary to achieve the interaction goal of maximizing the value of the fruit that Sophia owns at the end of the negotiation. That is, the prompt identifies the fruits that Sophia has and how much value she places on each type of fruit.

[0131] This specification uses the term "configured" with respect to systems and computer program components. When one or more computer systems are configured to perform particular operations or actions, it means that the system has installed thereon software, firmware, hardware, or a combination thereof that, when in operation, causes the system to perform the operation or action. When one or more computer programs are configured to perform particular operations or actions, it means that one or more programs contain instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0132] Embodiments of the subject matter and functional operations described herein can be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware including the structures disclosed herein and their structural equivalents, or one or more combinations thereof. Embodiments of the subject matter described herein can be implemented as one or more computer programs, e.g., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by or controlling the operation of a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or one or more combinations thereof. Alternatively, or additionally, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal generated to encode information for transmission to an appropriate receiver apparatus for execution by the data processing apparatus.

[0133] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. An apparatus may also be or include special-purpose logic circuitry, such as, for example, an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). An apparatus may optionally include, in addition to hardware, code that creates an execution environment for a computer program, for example, code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0134] A computer program, also referred to or written as a program, software, software application, app, module, software module, script, or code, can be written in any form of programming language, including compiled or interpreted, declarative, or procedural, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program can, but does not necessarily, correspond to a file in a file system. A program can be stored in a single file dedicated to the program, or in multiple coordinated files, e.g., files storing one or more modules, subprograms, or portions of code, or in files holding other programs or data, e.g., one or more scripts stored in a markup language document. A computer program can be deployed to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0135] As used herein, the term "database" is used broadly to refer to any collection of data, which need not be structured in any particular way, or at all, and which may be stored on storage devices in one or more locations. Thus, for example, an index database may contain multiple collections of data, each of which may be organized and accessed differently.

[0136] Similarly, the term "engine" is used broadly herein to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine is implemented as one or more software modules or components installed on one or more computers at one or more locations. In some cases, one or more computers are dedicated to a particular engine, and in other cases, multiple engines can be installed and executed on the same one or more computers.

[0137] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by special purpose logic circuitry such as, for example, an FPGA or ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0138] A computer suitable for running a computer program can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from a read-only memory or a random-access memory, or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by, or incorporated in, special-purpose logic circuitry. Generally, a computer also includes one or more mass storage devices for storing data, such as, for example, magnetic, magneto-optical, or optical disks, or is operatively coupled to receive data from, transfer data to, or both of, the one or more mass storage devices. However, a computer need not have such devices. Furthermore, a computer can be incorporated in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as, for example, a universal serial bus (USB) flash drive, to name just a few.

[0139] Computer-readable media suitable for storing computer program instructions and data include, by way of example, all forms of non-volatile memory, media, and memory devices including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0140] To provide for user interaction, embodiments of the subject matter described herein can be implemented on a computer having a display device, such as, for example, a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, as well as a keyboard and pointing device, such as, for example, a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide for user interaction; for example, feedback provided to the user can be any form of sensory feedback, such as, for example, visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input. Additionally, a computer can interact with a user by sending and receiving documents from a device used by the user, such as, for example, by sending a web page to a web browser on the user's device in response to a request received from the web browser. A computer can also interact with a user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and receiving a reply message from the user in return.

[0141] A data processing apparatus for implementing machine learning models may also include dedicated hardware accelerator units for handling common and computationally intensive portions of the workload, e.g., machine learning training or production, e.g., inference.

[0142] The machine learning model can be implemented and deployed using a machine learning framework, for example, the TensorFlow framework or the Jax framework.

[0143] Embodiments of the subject matter described herein can be implemented in a computing system that includes a back-end component, e.g., as a data server, or includes a middleware component, e.g., an application server, or includes a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described herein, or includes any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), e.g., the Internet.

[0144] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server sends data, e.g., HTML pages, to a user device, e.g., to display data to and receive user input from a user interacting with the device acting as a client. Data generated at the user device, e.g., the results of user interactions, can be received at the server from the device.

[0145] While the specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Some features that are described herein in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, while features may be described above as acting in some combinations and may initially be claimed as such, in some cases one or more features from a claimed combination can be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of a subcombination.

[0146] Similarly, while operations are illustrated in the figures and described in the claims in a particular order, this should not be understood as requiring such operations to be performed in the particular order shown, or in sequential order, or that all of the illustrated operations be performed to achieve desirable results. In some situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be incorporated together in a single software product or packaged in multiple software products.

[0147] Specific embodiments of the subject matter are described. Other embodiments are within the scope of the claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown or sequential order to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. [Explanation of symbols]

[0148] 100 Neural Network Systems 110 Language Model Neural Networks 104 Recommended Communications 120 Multi-Actor Interaction 132 First Actor 140 Communications 200 processes 300 processes 400 processes 500 processes 600 examples 604 Status 606 Value "Calm" 608 Chance Nodes 610 prompt 612 Proposed Communication

Claims

1. A method implemented by one or more computers, comprising: During interactions between multiple actors, identifying a first plurality of first strategies for selecting between first parameterizations relating to communications of a first actor of the plurality of actors for the interaction, each first parameterization parameterizing a set of context variables relating to the communications of the first actor during the interaction; identifying a second plurality of second strategies for selecting between second parameterizations relating to communications of a second actor of the plurality of actors for the interaction, each second parameterization parameterizing a set of context variables relating to the communications of the second actor during the interaction; For each of a plurality of pairs each including a respective first strategy and a respective second strategy, using a language model neural network to generate one or more simulations of the interaction assuming that the first actor communicates according to the first strategy and the second actor communicates according to the second strategy; determining a pairwise score from the one or more simulations that indicates the extent to which the first actor communicating according to the first strategy and the second actor communicating according to the second strategy fulfill an objective of the interaction; selecting a first parameterization for the first actor using the scores of the plurality of pairs; generating a proposed communication for the first actor by processing inputs that condition the language model neural network on the selected first parameterization; A method comprising:

2. the first actor is a user of a user device, and the method further comprises: providing the suggested communication for presentation to the user in a user interface of the user device. The method of claim 1 further comprising:

3. during the interaction, the communication produced by the first actor is a natural language communication; The method of claim 2 , wherein the suggested communication is a suggested natural language communication.

4. the first actor is a user of a user device, and during the interaction, the user receives a communication from the second actor and sends a communication to the second actor via a user interface of the user device, and the method comprises: The method of claim 3 , further comprising providing the suggested natural language communication for presentation to the user in the user interface of the user device.

5. receiving a user input selecting the proposed natural language communication; in response, transmitting the proposed natural language communication to the second actor; 5. The method of claim 4, further comprising:

6. selecting a first parameterization using the scores of the plurality of pairs, 3. The method of claim 2, comprising applying a game theory solver to the representation of the interaction including the scores of the plurality of pairs to identify a first parameterization that is optimal for the user.

7. generating one or more simulations of the interaction using a language model neural network, said step including, for each simulation and in each of a plurality of iterations: identifying prior communications between the first actor and the second actor during the simulation; selecting the first actor or the second actor as a current actor for the iteration; When the first actor is selected as the current actor, selecting a first parameterization for the iteration according to the first strategy; using the language model neural network to process (i) inputs that condition the language model neural network based on the selected first parameterization, and (ii) at least a portion of the previous communications during the simulation, to generate communications of the first actor; When the second actor is selected as the current actor, selecting a second parameterization for the iteration according to the second strategy; using the language model neural network to process (i) inputs that condition the language model neural network based on the selected second parameterization, and (ii) at least a portion of the previous communications during the simulation, to generate communications of the second actor; 2. The method of claim 1, comprising:

8. The method of claim 1 , wherein the set of context variables includes one or more context variables that define a communication style of the communication.

9. The method of claim 1 , wherein the set of context variables includes one or more context variables that define a set of information available as context to the first actor or the second actor.

10. The method of claim 1 , wherein one or more of the first strategies select the same first parameterization for each communication during the interaction.

11. The method of claim 1 , wherein one or more of the first strategies select different first parameterizations for different communications during the interaction.

12. receiving user input specifying a context variable; processing inputs defining the context variables using the language model neural network to generate one or more possible values ​​for the context variables that can be used to parameterize the context variables; The method of claim 1 further comprising:

13. determining, from the one or more simulations, a pair score indicative of the extent to which the first actor communicating according to the first strategy and the second actor communicating according to the second strategy fulfill a purpose of the interaction, the step of: generating a summary input from one or more of the communications generated during the simulation; generating a reference input characterizing the purpose of the interaction; processing the summary input and the criteria input using a second language model neural network to generate an output defining a simulation score for the simulation indicating the extent to which the simulation satisfies the one or more criteria; 2. The method of claim 1, comprising:

14. determining, from the one or more simulations, a pair score indicative of the extent to which the first actor communicating according to the first strategy and the second actor communicating according to the second strategy fulfill a purpose of the interaction, the step of: processing input including one or more of the communications generated during the simulation using a second language model neural network to generate a summary of the simulation; generating a simulation score for the simulation from the summary that indicates the extent to which the simulation meets the one or more criteria; 14. The method of claim 13, comprising:

15. generating a simulation score for the simulation from the summary indicating the extent to which the simulation meets the one or more criteria; The method of claim 14 , comprising applying one or more heuristics to the summary to generate the simulation score.

16. generating a simulation score for the simulation from the summary indicating the extent to which the simulation meets the one or more criteria; generating a reference input characterizing the purpose of the interaction; processing the summary input and the reference input using the second language model neural network to generate an output that defines the simulation score; 15. The method of claim 14, comprising:

17. The method of claim 13 , wherein the second language model neural network is the language model neural network.

18. one or more computers; one or more storage devices storing instructions that, when executed by said one or more computers, cause said one or more computers to perform each of the operations of the method of any one of claims 1 to 17; A system comprising:

19. One or more computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform each of the operations of the method of any one of claims 1 to 17.

Images