Multi-agent debate

Abstract

A method for multi-agent debate is presented. In the method, a set of responses to a task expressed in a natural language is generated by a plurality of agents in at least one round of debate for the task. The set of responses is updated by selecting a response from the set of responses and / or adjusting at least one response in the set of responses. The updated response set is provided to the plurality of agents as input for debate for subsequent rounds to generate a result for the task.

Description

Technical Field

[0001] The present disclosure relates generally to the field of computers, and more particularly, to methods, devices, and computer program products for multi-agent debate. Background Art

[0002] Conversational platforms allow users to give their tasks and then provide responses to the tasks. Tasks are usually expressed in natural language. For example, users usually enter natural language questions and then the conversational platform provides answers to the questions. Today, conversational platforms can rely on models such as machine learning models to perform reasoning on tasks and provide responses in natural language. There are various works to enhance the reasoning capabilities of models to improve the accuracy of generated responses. Summary of the invention

[0003] In a first aspect of the present disclosure, a method for multi-agent debate is provided. The method includes: obtaining a set of responses to a task expressed in natural language, the set of responses being generated by multiple agents in at least one round of debate for the task; updating the set of responses by at least one of: selecting a response from the set of responses, or adjusting at least one response in the set of responses; and generating a result for the task by providing the updated set of responses to multiple agents as input for subsequent rounds of debate.

[0004] In a second aspect of the present disclosure, an electronic device is provided. The electronic device comprises: a computer processor coupled to a computer readable memory unit, the memory unit comprising instructions for implementing the method according to the first aspect of the present disclosure when executed by the computer processor.

[0005] In a third aspect of the present disclosure, a computer program product is provided, which includes a computer-readable storage medium having program instructions contained therein, and the program instructions can be executed by an electronic device to enable the electronic device to perform the method according to the first aspect of the present disclosure.

[0006] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The above and other objects, features and advantages of the present disclosure will become more apparent by describing some embodiments of the present disclosure in more detail in the accompanying drawings, wherein like reference numerals generally refer to like components in the embodiments of the present disclosure.

[0008] Figure 1An example environment is shown in which example embodiments of the present disclosure may be implemented;

[0009] Figure 2 A diagram showing a debate process utilizing a hidden Markov model with observable and unobservable variables according to some embodiments of the present disclosure;

[0010] Figure 3 An example of a misconception between models and a refutation of the misconception is shown according to some embodiments of the present disclosure;

[0011] Figure 4 An example process for intervening in a previous response generated by an agent according to some embodiments of the present disclosure is shown;

[0012] Figure 5 An example flowchart showing a method of multi-agent debate according to some embodiments of the present disclosure; and

[0013] Figure 6 A block diagram of an electronic device is shown in which various embodiments of the present invention may be implemented. DETAILED DESCRIPTION

[0014] The principle of the present disclosure will now be described with reference to some embodiments. It should be understood that these embodiments are described only for illustrative purposes, and help those skilled in the art to understand and implement the present disclosure, without implying any limitation to the scope of the present disclosure. The disclosure described herein can be realized in various ways except the way described below.

[0015] In the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0016] References in this disclosure to "one embodiment," "an embodiment," "an example embodiment," etc. indicate that the described embodiment may include a particular feature, structure, or characteristic, but not every embodiment necessarily includes the particular feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same embodiment. Furthermore, it is considered to be within the knowledge of those skilled in the art to affect such features, structures, or characteristics in conjunction with other embodiments, whether or not explicitly described.

[0017] It should be understood that although the terms "first" and "second" etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element without departing from the scope of the exemplary embodiments. As used herein, the term "and / or" includes any and all combinations of one or more listed terms.

[0018] The terms used herein are used only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments. As used herein, the singular forms "a", "an", and "the" are also intended to include the plural forms unless the context clearly indicates otherwise. It will be further understood that the terms "comprises", "including", "having", "having", "including", and / or "comprising" when used herein specify the presence of stated features, elements, and / or components, etc., but do not exclude the presence or addition of one or more other features, elements, components, and / or combinations thereof.

[0019] The principle of the present disclosure will now be described with reference to some embodiments. It should be understood that these embodiments are described only for illustrative purposes, and help those skilled in the art to understand and implement the present disclosure, without implying any limitation to the scope of the present disclosure. The disclosure described herein can be implemented in a variety of ways except for the manner described below. In the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meanings as those of ordinary skill in the art to which the present disclosure belongs.

[0020] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of relevant laws and regulations and relevant rules.

[0021] It is understandable that before using the technical solutions disclosed in the embodiments of the present invention, the user should be informed of the type, scope of use and usage scenarios of the information involved in the present invention in an appropriate manner in accordance with relevant laws and regulations, and the user's authorization should be obtained.

[0022] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly inform the user that the requested operation will require obtaining and using the user's information. Therefore, the user can independently choose whether to provide the information to the electronic device, application, server, storage medium, or other software or hardware that executes the operation of the technical solution of the present application based on the prompt message.

[0023] As an optional but non-limiting implementation, in response to receiving an active request from the user, the method of sending the prompt information to the user may include, for example, a pop-up window, and the prompt information may be presented in the pop-up window in the form of text. In addition, the pop-up window may also carry a selection control for the user to select "agree" or "disagree" to provide information to the electronic device.

[0024] It is understandable that the above-mentioned process of notifying and obtaining user authorization is merely exemplary and does not limit the implementation of the present disclosure. Other methods that meet relevant laws and regulations are also applicable to the implementation of the present disclosure.

[0025] As used herein, the term "model" is referred to as the association between input and output learned from training data, and therefore a corresponding output can be generated for a given input after training. The generation of the model can be based on machine learning techniques. Typically, a machine learning model can be constructed that receives input information and makes predictions based on the input information. For example, a classification model can predict the category of input information in a predetermined set of categories. As used herein, a "model" may also be referred to as a "machine learning model," "learning model," "machine learning network," or "learning network," which are used interchangeably herein. Example Environment

[0026] Figure 1 A block diagram of an example environment 100 is shown in which various embodiments of the present disclosure may be implemented. Figure 1 In the environment 100 of the present invention, the conversation platform 110 may receive a task 135 from a user 130. The task may be provided or expressed in a natural language (e.g., in English, Chinese, or other natural languages). The conversation platform 110 includes one or more agents 120-1, 120-2, ..., 120-N (collectively or individually referred to as agents 120, where N indicates an integer greater than or equal to 1) to process tasks from users 130 and provide a final result 140 for the task 135. In an example, the conversation platform 110 may receive a natural language question from the user 130 and provide a final answer to the question. In the following, for the purpose of explanation without any limitations, some embodiments are described by taking questions as examples of tasks 135 and taking answers as examples of results 140.

[0027] Agents 120-1, 120-2, ..., 120-N may be configured to process natural language tasks using machine learning models 125-1, 125-2, ..., 125-N (collectively or individually referred to as machine learning models 125 or simply models 125), respectively. Alternatively, agents 120-1, 120-2, 120-N may be implemented by machine learning models 125-1, 125-2, 125-N, respectively. In the following, the terms "model" or "agent" may be used interchangeably.

[0028] In some embodiments, model 125 may be a language model (LM) that is trained on a large-scale corpus to learn the ability to understand one or more types of natural language. In some embodiments, model 125 may be a generative model for content generation. Model 125 may be constructed based on various machine learning techniques.

[0029] LMs trained on large-scale corpora have demonstrated a remarkable ability to perform unseen tasks with high efficiency. This behavior, often referred to as emergent, allows LMs to serve as a general tool for a wide range of language-based functions. One such behavior of particular interest is the ability of LMs to acquire and process opinions from other models (or humans). This ability allows LMs to collaboratively solve tasks by participating in debates. For a given task, multi-agent debates operate by eliciting responses from each model, distributing these responses between models, and then eliciting updated responses from each model.

[0030] Multi-agent debate focuses on iterative collaboration between agents in order to make decisions. Work on multi-agent debate is generally focused on multi-agent debate in the context of question-answering tasks, and aims to provide higher quality answers (compared to the answers of a single model) by involving multiple models in the discussion. The basic debate format supports debate by first asking each model the same question, and then iteratively re-asking the agents the same question contextualized by the responses of all models in the previous round. Different solutions to this process have also been proposed. For example, when each model in the debate is assigned a different role, one solution focuses on debate, and another solution focuses on round-robin debate. Some other solutions for iteratively improving the quality of answers have also been proposed.

[0031] Despite the above schemes, current work on multi-agent debate lacks theoretical research, and the debate effectiveness needs to be improved.

[0032] Embodiments of the present disclosure propose a solution for multi-agent debate. According to embodiments of the present disclosure, a response set to a task expressed in a natural language is generated by multiple agents in at least one round of debate for the task. The response set is updated by selecting a response from the response set and / or adjusting at least one response in the response set. The updated response set is provided to multiple agents as input for subsequent rounds of debate to generate a result for the task.

[0033] In an embodiment of the present disclosure, during a multi-agent debate process, previous responses from agents are not directed for subsequent rounds of debate. Instead, these previous responses are intervened, for example, by selecting a subset of responses and / or by correcting some of them. These intervened responses are used for observation by agents that generate responses in the next round of debate. In this way, the accuracy of the final result and the efficacy of the debate can be improved.

[0034] The theoretical basis and embodiments of the present invention are described in detail below with reference to the accompanying drawings. Theoretical formulation of multi-agent debate

[0035] To better understand the embodiments of the present disclosure, a theoretical formulation of multi-agent debate that extracts connections from contextual learning and Bayesian reasoning is first provided. Let x be a given question with an associated answer z, e.g., x = "What color is the sky?" and z = "blue". Task x and its associated answer z are obtained by first selecting a latent concept θ (0) ∈Θ, and then sample (x,y)~D(θ (0) ). For example, in the above task, it may be the case that θ represents concepts such as “nature”, “common sense”, and “refraction”. For a given task x, a collection of n LMs (or agents) collaborate to reason about the correct answer z by iteratively participating in T rounds of discussion. For example, at round t = 0, each agent i observes x and then provides a response At round t>0, each agent i observes x and That is, the output of n agents at the previous time step, and output the response If t = T or if the agents reach a consensus, the process ends.

[0036] To measure whether consensus is reached, function a extracts the answer from a given response y. Assume y = "During the day, the sky is blue", then a(y) = "blue". In round t, agent i provides the response The probability of is given by: where φ i Captures model hyperparameters (such as training data, architecture, etc.). Input (Y (t) , x) and the hyperparameter φ i All of these will ultimately affect the output Note that at each round, all agents observe the same input, i.e. (Y (t) , x). Therefore, the agent outputs The difference is due to 1) the random nature of the output generation and 2) the unique parameter φ of each model i to be sure.

[0037] According to some embodiments of the present disclosure, a theoretical formulation of multi-agent debate can provide key insights into the inner workings of the debate process, which can then be used to develop more effective debates. Agent debate can be viewed as a multivariate hidden Markov model with observed variables Y, x, and φ and hidden variables θ corresponding to latent concepts, such as Figure 2 Assumptions are shown in Table 1. Table 1

[0038] The assumptions in Table 1 can be interpreted as saying that the model generates y i By the model parameter φ i and the concept θ uniquely determined by the model identity. In the case of encoder-decoder based models, the union between φ and θ can be conceptualized as corresponding to the embedding produced by the encoder. With this embedding in hand, the original input no longer affects the output of the model, but rather the embedding and the model parameters will uniquely determine the output of the model.

[0039] We now give some descriptions to derive the lemmas shown in Table 2, which will help to examine how the model response evolves over time. Table 2

[0040] The importance of this lemma lies in that, using other responses Y (t) Generates a given response The probability of can be generated based on the absence of other responses and the skewness caused by these responses. Note that in can be considered as A weighted version of , where the weights are determined by the skew term Given.

[0041] As a proof, this result holds via marginalization of the posterior predictive distribution over the latent concept Θ, namely:

[0042] Multi-agent debate can be viewed as a special type of in-context learning, where each model responds to is an example (also called a piece of context). The most important difference between learning in context and debate is that each example From the perspective of contextual learning, each example corresponds to independent and identical distributions (iid) drawn from a fixed distribution D(θ), i.e., each example is intended to convey a similar idea faithfully. However, in the case of debate, each Relative to Y (t) , x and more importantly relative to φ i Distribution; y (t) For the latter it is no longer iid unless all models have the same parameters φ. In this way, they can be competing factors within the set of responses Y(t), which will not exist in contextual learning.

[0043] There is a natural connection between learning in context and Bayesian inference. Specifically, when seeing a prompt and n pieces of context, the probability that the model provides a given response is When the context consists of an increasing number of examples originating from the same concept θ, the model places a higher weight on responses associated with that latent concept. That is, for The concept is that the model is more likely to generate a response corresponding to θ, i.e., adding more examples increases the density placed on θ during generation. This phenomenon also extends to debate, where the responses Y of other agents (t) Now substitute the context. As mentioned before, each response can have different distributions on the latent concepts, which ultimately affects the response of each model at time t+1, as shown in the lemma in Table 2, which is caused by the skew term This increased influence can be both a benefit and a threat to debate. Some embodiments described below may be centered around utilizing this skew term so that it improves debate effectiveness.

[0044] Through this perspective of debate, a more efficient debate process can be designed by exploiting the concept space Θ. To this end, multi-agent debate can be formulated as an optimization problem, where the skew terms described in the lemma shown in Table 2 correspond to the optimization variables. For a given task x with a correct answer z*, each round of debate can be formulated as the following optimization problem. That is, time t is designed to produce response Y (t) , so that they maximize the probability that n models choose the correct answer at the next time step. Next, this objective can be expanded on the latent concept space Θ, which becomes

[0045] There are three key challenges in directly optimizing this objective. First, the underlying concept Θ and the true concept θ* from which x and z* are derived are unknown. Second, the relationship between a given response and the underlying concept is also unknown. Third, the optimization variable (Y (t)The responses in ( ) are natural language and therefore difficult to optimize directly. However, this latent concept-based goal extension allows the design of several methods that work within the concept space to better optimize the debate. To motivate these methods, the present disclosure makes several observations about the debate process as a whole. Debate Principles

[0046] The first thing to examine is how model capability and the diversity of model opinions affect the collaborative process. As can be seen from the following description, with response Y (t) Both diversity of information in the model and different opinions and capabilities across the model ensemble are necessary.

[0047] Assume that the collaborative process is conducted using only one type of model (actually n copies of the same model). That is, for all i∈[n], φi≡φ. Then, as the number of agents increases, the debate process is more greatly affected by the echo chamber effect, i.e., the probability that a round of debate leads to a change in the most likely concept perceived by the agent approaches 0. That is, a larger number of similar agents leads to static debate dynamics, essentially undermining the purpose of debate. Table 3

[0048] The theorem in Table 3 implies that when debate is conducted using multiple copies of the same model (or very similar models), increasing the number of models leads to a debate centered around a single (unchanged) concept θ′(t), rather than a balanced distribution over multiple concepts.

[0049] As a proof sketch, by the lemma shown in Table 2, this term It can be expressed as The term is a constant relative to the number of agents n. According to x, Y (t-1) , and φ are iid, and except for θ′ (t-1) In addition, for all θ, the term will disappear. Therefore, as θ=θ′ (t-1) , Each term in the expanded form of will similarly vanish. When →∞, all models predict the same concept, i.e., with probability 1 at time step t.

[0050] Next, we examine how similar responses affect the collaborative process. At time t, assume that there are m repeated responses and k non-repeated responses, i.e. As m increases, the answer of model i at time t will be biased towards concepts related to y'. Table 4 shows such a theorem. Table 4

[0051] As a proof sketch, by the lemma shown in Table 2, model i observes the response Y (t) When the response is given at time t The conditional probability can be written as:

[0052] Now consider generating any two responses The ratio between the likelihoods is as follows:

[0053] Using the decomposition in Equation 6, for each of these terms, let θ′ be the concept that model i believes is most likely to produce response y′, that is,

[0054] For every θ≠θ′, Eq. Applicable. Therefore Therefore, when examining the limits of the ratio in Equation 7 with respect to the number m of repetitive signals, the following equation is obtained:

[0055] Therefore, the relationship between any two generated probabilities can be uniquely defined by θ′. Since the final responses of the model are sampled according to these ratios, the final response will be uniquely determined by the concept θ′.

[0056] The theorem in Table 4 indicates the sensitivity of multi-LM debates to majority control. If a large number of models provide the same answer to task x, then those repeated answers will drown out the individual responses provided by other models as well as task x itself.

[0057] Next, we describe a specific type of homogeneity in model capabilities and responses, namely shared misconceptions. When common misconceptions are shared between models, debate is less effective and may converge to the misconception associated with the shared misconception. Given this, a more carefully designed debate process is needed.

[0058] exist Figure 3 , provide examples to illustrate what it means to share a false idea. Figure 3 A collection of model responses to a question 301 about the animal "red panda" is shown, with the correct answer 302 being "no". The models incorrectly equate red pandas with pandas, which ultimately results in each model providing incorrect answers 311, 312, 313. In situations such as Figure 3 In the case shown, since each model shares the same false belief about the problem, the debate will converge to an incorrect answer.

[0059] Next, the concept of wrong idea is formalized as follows. For a given concept θ* with parameter φ i A model of θ is said to have a wrong idea about θ* if there exists another concept θ′ that satisfies θ. That is, for tasks generated from concept θ*, the model believes that the false concept θ′ explains more than half of the tasks better than the true concept θ*.

[0060] When m agents among the agents have wrong ideas and share the same wrong concept θ′, these m agents can be considered to have a shared wrong idea. When models share common wrong ideas, the responses generated by those models are biased towards the wrong concept θ′. Table 5 shows such a theorem. Table 5

[0061] As a proof sketch, for a task x~D(θ*), and an agent i with a wrong idea,

[0062] The opposite is true for agents that do not share the false idea. Thus, any agent with a shared false idea is more likely to provide a θ′. Thus, when seeing x, agents with a shared false idea are more likely to provide a response that conveys the false concept θ′. Agents see x in every round of the debate; this deteriorating effect causes the debate to center around θ′ rather than the true concept θ*.

[0063] It should be noted that the phenomenon of convergence to misconceptions cannot be mitigated by adding more models. When a model forms a misconception through training data, because of the high correlation in the training data between models, it is possible that other models may have the same misconception unless they are specifically trained to avoid such errors. Interference in the debate process

[0064] In the above, a theoretical framework for multi-LLM debates that extract connections from context learning and Bayesian reasoning is proposed, and theoretical insights into several key principles of multi-LLM debates are provided. Given the theoretical framework and principles, several modifications to the debate process, referred to as interventions, are proposed in some embodiments of the present disclosure.

[0065] Reference now Figure 4, which shows an exemplary debate process 400 with intervention. The debate process 400 is used to generate a result of a task expressed in natural language, for example, to generate an answer to a natural language question. In the current round t of the debate on the task, multiple agents 120 generate corresponding responses to the task. After the t round is completed, a response set 401 is obtained. The response set 401 may include responses generated by multiple agents 120 in at least one round of debate. For example, Figure 4 As shown, the response generated by the agent 120 at round t is determined as at least a part of the response set.

[0066] In some embodiments, the previous response set 401 may consist of responses generated by multiple agents in a previous round, for example, responses generated by each agent in the current round t.

[0067] In some embodiments, previous response set 401 may include responses generated over multiple rounds of debate. In other words, in such embodiments, previous response set 401 may include previous responses generated from multiple historical rounds.

[0068] In some embodiments, the previous response set 401 may include all previous responses generated up to the current round t. For example, the previous response set 401 may be represented as Y (0) ∪...∪ (t) . By using all previous responses, better performance can be achieved.

[0069] Continuing with process 400, at block 410, one or more interventions may be applied to previous set of responses 401, thereby producing updated set of responses 402. Interventions may include two categories: pruning interventions and modifying interventions.

[0070] The pruning intervention can be used to select a subset of responses from the previous response set 401. In other words, the pruning intervention focuses on selecting which responses to keep (t) Y (t) The selection may be based on the mutual information of the two responses and / or the relevance of the responses to the task. Thus, the pruning intervention may include diversity pruning 411 and quality pruning 412, which will be described in detail below.

[0071] The modification intervention may be used to adjust at least one response in the previous response set 401. In other words, the modification may focus on changing or editing the response Y. (t) A correction intervention may be used to correct a misconception in a response. A response may be corrected by a different agent than the agent that generated the response. A correction intervention may include a misconception rebuttal 413.

[0072] In some embodiments, at block 410, two or more interventions may be combined to update the set of prior responses 401. Any suitable way to combine different interventions is possible.

[0073] In some embodiments, pruning interventions may be combined with modification interventions. For example, pruning interventions (diversity pruning and / or quality pruning) may be first applied to select a subset of responses from the previous response set 401. Then, modification interventions may be applied to adjust the responses in the subset of responses.

[0074] In some embodiments, diversity pruning 411, quality pruning 412, and false idea rebuttal 410 may be combined. For example, quality pruning 412 is first applied to select a first subset of responses from the previous response set 401 based on the relevance of each response to the task. Then, diversity pruning 412 may be applied to select a second subset of responses from the first subset of responses based on the mutual information of each pair of responses in the first subset of responses. Finally, false idea rebuttal 410 may be applied to correct false ideas in the responses in the second subset of responses.

[0075] By applying one or more interventions, an updated response set 402 is obtained. Figure 4 As shown, the updated response set 402 can be provided to the plurality of agents 120 as input for a subsequent round t+1. For example, the updated response set 402 is included in a prompt to each agent 120. Each of the plurality of agents 120 observes the updated response set 402 and the task x, and generates a response in a subsequent round t+1. The process 400 can be iteratively performed until the total number of rounds reaches a threshold or a consensus is reached among the plurality of agents 120.

[0076] Some more examples regarding the three interventions are now described.

[0077] According to the quality intervention, responses can be selected based on the relevance of each response to the task. In some embodiments, a quality metric for each response in the first plurality of responses can be determined based on the task and the response. The quality metric for the response can represent or characterize the relevance of the response to the task. Then, a first number of responses can be selected from the first plurality of responses based on the corresponding quality metrics for the first plurality of responses.

[0078] In such an embodiment, the first plurality of responses is included in the previous set of responses 401. For example, if the quality intervention is applied as a first intervention (specifically, before the diversity intervention), the first plurality of responses may be the previous set of responses 401. If the quality intervention is applied, such as after the diversity intervention, the first plurality of responses may include responses selected according to the diversity intervention.

[0079] The quality metric may be any suitable metric suitable for reflecting the relevance of the task to the response. To this end, in some embodiments, an embedding model for natural language may be used, such as a sentence embedding model. For example, the embedding model may be used to generate a first representation for the task and a second representation for the response. The similarity between the first representation and the second representation may be determined as a quality metric for the response.

[0080] As an example, quality pruning 412 selects k responses out of n responses at time t. Quality pruning 412 aims to select the k highest question responses, rather than selecting for diversity. This is done by selecting the k responses that maximize the following equation: That is, quality pruning aims to select Y (t) The k responses from the distribution in that are conceptually most similar to the original task x.

[0081] In fact, every distribution D(θ|y i ), D(θ|x) cannot be accessed directly, but can be approximated by a sentence embedding model. The embedding space of such a model, such as g, is used as a proxy for the latent concept space. Therefore, in practice, quality pruning can be operated as: where d(g(y i ), g(y j )) is the representation similarity used as a quality measure.

[0082] Table 6 shows an example theorem for quality pruning. Table 6

[0083] According to the diversity intervention, a response can be selected based on the mutual information of each pair of responses in the previous response set 401. In some embodiments, a diversity metric for each pair of responses in the second plurality of responses can be determined based on the pair of responses. The diversity metric for each pair of responses can represent or characterize the mutual information for the pair of responses. Then, a second number of responses can be selected from the second plurality of responses based on the diversity metric for each pair of responses.

[0084] In such an embodiment, the second plurality of responses is included in the previous response set 401. For example, if the diversity intervention is applied as a first intervention (specifically, before the quality intervention), the second plurality of responses may be the previous response set 401 of responses. If the diversity intervention is applied, such as after the quality intervention, the second plurality of responses may include responses selected according to the quality intervention.

[0085] The diversity metric may be any suitable metric suitable for reflecting mutual information. To this end, in some embodiments, similar to the quality metric, an embedding model for natural language may be used, such as a sentence embedding model. For example, the embedding model may be used to generate a pair of representations of a pair of responses. The diversity metric for the pair of responses may be determined based on the similarity between the pair of representations, for example, as 1 minus the similarity.

[0086] As an example, the diversity pruning intervention selects Y (t) The k responses that maximize mutual information among the n responses in . That is, Y′ (t) At time t,

[0087] In practice, the distribution D(θ|y) is unknown but can be approximated by a sentence embedding model. Similar to quality pruning, the embedding space of such a model (e.g., g) is used as a proxy for the latent concept space. Let d(u,v)≡1-cosine of similarity(u,v), then in practice Y ′(t) It can be calculated as:

[0088] Next, we show that when the models have shared false ideas, diversity pruning reduces the likelihood that the debate process will converge to the false concept corresponding to the shared false idea. Table 6 shows an example theorem for diversity pruning. Table 7

[0089] As shown in the theorems in Tables 6 and 7, diversity pruning reduces the probability that the debate converges to an incorrect answer derived from a specific concept, while quality pruning increases the probability that the debate converges to a correct answer derived from a true concept. In some embodiments, both interventions can be used simultaneously to more effectively guide the debate process away from incorrect answers and towards correct answers.

[0090] The effectiveness of diversity pruning is highly dependent on the quality of the models used in the debate, while quality pruning is highly dependent on the richness of the task (a very detailed task is likely to convey more concepts relevant to the answer than a concise task lacking such details). Thus, quality pruning can be prioritized for detailed tasks, and diversity pruning can be prioritized if all models in the debate are believed to be of very high quality.

[0091] According to the false idea rebuttal, a given response in the previous response set 401 can be adjusted by the target agent to correct one or more false ideas in the given response. The given response may not be generated by the target agent. The response can be adjusted by prompting the target agent in the plurality of agents to correct the response. The adjusted response generated by the target agent can be maintained in the updated set 402 for the subsequent round t+1.

[0092] In some embodiments, a target agent among multiple agents may be assigned to a given response. Prompt information indicating correction of the given response (e.g., correction of a wrong idea) may be generated and provided to the target agent. Then, an adjusted response for the given response may be derived from the output of the target agent.

[0093] As an example, in addition to choosing to use Y in the next round of debates (t) In addition to the responses in (t) The misconception refutation aims to solve the problem by having the model first identify the (t) ∈Y (t) The operation is performed precisely by finding the wrong ideas in the debate and then correcting those wrong ideas. In each round t of the debate process, each model i is randomly assigned a response. First, model i is prompted to identify the response Given a list of wrong ideas, the model is then asked to refute and Corrected versions of both. Figure 3 An exemplary rebuttal 320 is shown for illustration purposes only.

[0094] Compared with providing answers directly, the LM-based agent is more skilled in evaluating answers. Therefore, by using the agent to correct the response, a more accurate response can be achieved. Table 8 shows an example theorem for wrong idea refutation. Table 8

[0095] As can be seen from the above description, the results from the theoretical framework are used to improve the efficacy of the debate procedure. In particular, three interventions (modifications to the debate process) are proposed. Namely, diversity pruning aims to maximize the mutual information in the responses shared between rounds. This intervention has the additional benefit of reducing the severity of the echo chamber effect. In addition, quality pruning is proposed, which aims to maximize the relevance of each shared response. This intervention improves the probability that the debate process converges to the correct answer.

[0096] Finally, misconception refutation directly identifies and attempts to refute misconceptions in the model’s responses. This intervention draws inspiration from the fact that LMs are generally more skilled at evaluating answers than directly providing them.

[0097] For each intervention, theoretical results are described above to outline how each intervention improves debate. Experiments are conducted on four common benchmarks to show that these interventions improve debate efficacy in practice. Example Procedure and Equipment

[0098] Figure 5 A flowchart of a method 500 for multi-agent debate according to some examples of the present disclosure is shown. The method 500 may be implemented in Figure 1 The embodiment of the present invention is implemented at the conversation platform 110 shown in FIG. At block 510, a set of responses to the task expressed in natural language is obtained. The set of responses is generated by multiple agents in at least one round of debate for the task. At block 520, the set of responses is updated by at least one of the following: selecting a response from the set of responses or adjusting at least one response in the set of responses. At block 530, a result for the task is generated by providing the updated set of responses to multiple agents as input for subsequent rounds of debate.

[0099] In some embodiments, selecting a response from a response set includes: determining a quality metric for the response based on the task and each response in a first set of multiple responses, the first set of multiple responses being included in the response set, the quality metric representing a relevance of the response to the task; and selecting a first number of responses from the first set of multiple responses based on the quality metric for each response in the first set of multiple responses.

[0100] In some embodiments, determining a quality metric for a response based on the task and each response in a first set of multiple responses includes: generating a first representation for the task and a second representation for the response through an embedding model for natural language; and determining a similarity between the first representation and the second representation as a quality metric for the response.

[0101] In some embodiments, selecting responses from the response set includes: determining a diversity metric for each pair of responses in a second set of multiple responses, the second set of multiple responses included in the response set, the diversity metric representing mutual information for the pair of responses; and selecting a second number of responses from the second set of multiple responses based on the diversity metric for each pair of responses.

[0102] In some embodiments, determining a diversity measure for each pair of responses in the second set of multiple responses includes: generating a pair of representations for the pair of responses through an embedding model for natural language; and determining a diversity measure for the pair of responses based on similarities between the pair of representations.

[0103] In some embodiments, adjusting at least one response in a response set includes: assigning a target agent among multiple agents to a given response in the response set, the given response not being generated by the target agent; providing prompt information to the target agent indicating correction of the given response; and deriving an adjusted response for the given response from the output of the target agent.

[0104] In some embodiments, updating a response set includes: selecting a first subset of responses from the response set based on a relevance of each response in the response set to the task; selecting a second subset of responses from the first subset of responses based on mutual information of each pair of responses in the first subset of responses; and determining an updated response set based on the second subset of responses.

[0105] In some embodiments, determining an updated response set based on the second response subset includes: adjusting a response in the second response subset by prompting a target agent among multiple agents to correct the response, the response not generated by the target agent; and determining the adjusted response as a response in the updated response set.

[0106] In some embodiments, the set of responses includes responses generated by multiple agents in multiple previous rounds of the debate.

[0107] In some implementations of the present disclosure, an electronic device is provided, comprising a computer processor coupled to a computer-readable memory unit, the memory unit comprising instructions, which when executed by the computer processor implement a method for multi-agent debate. The method comprises: obtaining a set of responses to a task expressed in a natural language, the set of responses being generated by multiple agents in at least one round of debate for the task; updating the set of responses by at least one of: selecting a response from the set of responses or adjusting at least one response in the set of responses; and generating a result for the task by providing the updated set of responses to multiple agents as input for subsequent rounds of debate.

[0108] In some embodiments, selecting a response from a response set includes: determining a quality metric for the response based on the task and each response in a first set of multiple responses, the first set of multiple responses being included in the response set, the quality metric representing a relevance of the response to the task; and selecting a first number of responses from the first set of multiple responses based on the quality metric for each response in the first set of multiple responses.

[0109] In some embodiments, determining a quality metric for the response based on the task and each response in the first set of multiple responses includes: generating a first representation for the task and a second representation for the response through an embedding model for natural language; and determining a similarity between the first representation and the second representation as a quality metric for the response.

[0110] In some embodiments, selecting responses from the response set includes: determining a diversity metric for each pair of responses in a second set of multiple responses, the second set of multiple responses included in the response set, the diversity metric representing mutual information of the pair of responses; and selecting a second number of responses from the second set of multiple responses based on the diversity metric for each pair of responses.

[0111] In some embodiments, determining a diversity measure for each pair of responses in the second set of multiple responses includes: generating a pair of representations for the pair of responses through an embedding model for natural language; and determining a diversity measure for the pair of responses based on similarities between the pair of representations.

[0112] In some embodiments, adjusting at least one response in a response set includes: assigning a target agent among multiple agents to a given response in the response set, the given response not being generated by the target agent; providing prompt information to the target agent indicating correction of the given response; and deriving an adjusted response for the given response from the output of the target agent.

[0113] In some embodiments, updating a response set includes: selecting a first subset of responses from the response set based on a relevance of each response in the response set to the task; selecting a second subset of responses from the first subset of responses based on mutual information of each pair of responses in the first subset of responses; and determining an updated response set based on the second subset of responses.

[0114] In some embodiments, determining an updated response set based on the second response subset includes: adjusting a response in the second response subset by prompting a target agent among multiple agents to correct the response, the response not generated by the target agent; and determining the adjusted response as a response in the updated response set.

[0115] In some embodiments, the set of responses includes responses generated by multiple agents in multiple previous rounds of the debate.

[0116] In some embodiments of the present disclosure, a non-transitory computer program product is provided, the non-transitory computer program product comprising a computer-readable storage medium having program instructions embodied therein, the program instructions being executable by an electronic device to cause the electronic device to perform a method of multi-agent debate. The method comprises: obtaining a set of responses to a task expressed in a natural language, the response set being generated by multiple agents in at least one round of debate for the task; updating the response set by at least one of the following: selecting a response from the response set or adjusting at least one response in the response set; and generating a result for the task by providing the updated response set to multiple agents as input for subsequent rounds of debate. In some embodiments of the present invention, the method further comprises the other steps described in the present disclosure.

[0117] Figure 6 A block diagram of an electronic device 600 is shown in which various embodiments of the present disclosure may be implemented. It should be understood that Figure 6 The electronic device 600 shown is for illustration purposes only and does not imply any limitation on the functionality and scope of the present disclosure in any way. The electronic device 600 can be used to implement the above method 600. Figure 6 As shown, the electronic device 600 may be a general electronic device and may include at least one or more processors or processing units 610 , a memory 620 , a storage unit 630 , one or more communication units 640 , one or more input devices 650 , and one or more output devices 660 .

[0118] Processing unit 610 may be a physical or virtual processor and may implement various processes based on program 625 stored in memory 620. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to increase the parallel processing capabilities of electronic device 600. Processing unit 610 may also be referred to as a central processing unit (CPU), a microprocessor, a controller, or a microcontroller.

[0119] The electronic device 600 typically includes various computer storage media. Such media can be any media accessible to the electronic device 600, including but not limited to volatile and non-volatile media, or removable and non-removable media. The memory 620 can be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (such as a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM) or flash memory) or any combination thereof. The storage unit 630 can be any removable or non-removable medium, and can include a machine-readable medium, such as a memory, a flash drive, a disk or other medium, which can be used to store information and / or data and can be accessed in the electronic device 600.

[0120] The electronic device 600 may further include additional removable / non-removable volatile / non-volatile memory media. Figure 6 , but a magnetic disk drive for reading from and / or writing to a removable non-volatile magnetic disk and an optical disk drive for reading from and / or writing to a removable non-volatile optical disk may be provided. In this case, each drive may be connected to a bus (not shown) via one or more data medium interfaces.

[0121] The communication unit 640 communicates with another electronic device via a communication medium. In addition, the functions of the components in the electronic device 600 can be implemented by a single computing cluster or multiple computing machines that can communicate via a communication connection. Therefore, the electronic device 600 can operate in a networked environment using logical connections to one or more other servers, networked personal computers (PCs), or other general network nodes.

[0122] The input device 650 may be one or more of various input devices, such as a mouse, keyboard, trackball, voice input device, etc. The output device 660 may be one or more of various output devices, such as a display, a speaker, a printer, etc. With the aid of the communication unit 640, the electronic device 600 may also communicate with one or more external devices (not shown) (such as a storage device and a display device), wherein one or more devices enable a user to interact with the electronic device 600, or if necessary, any device (such as a network card, a modem, etc.) enables the electronic device 600 to communicate with one or more other electronic devices. Such communication may be performed via an input / output (I / O) interface (not shown).

[0123] In some embodiments, instead of being integrated in a single device, some or all components of the electronic device 600 may also be arranged in a cloud computing architecture. In a cloud computing architecture, components may be provided remotely and work together to implement the functions described in the present disclosure. In some embodiments, cloud computing provides computing, software, data access and storage services, which will not require the end user to know the physical location or configuration of the system or hardware that provides these services. In various embodiments, cloud computing provides services via a wide area network (such as the Internet) using a suitable protocol. For example, a cloud computing provider provides an application via a wide area network, which can be accessed by a web browser or any other computing component. The software or components of the cloud computing architecture and the corresponding data may be stored on a server at a remote location. The computing resources in a cloud computing environment may be merged or distributed at a location in a remote data center. Cloud computing infrastructure may provide services through a shared data center, although they act as a single access point for users. Therefore, cloud computing architecture may be used to provide components and functions described herein from a service provider at a remote location. Alternatively, they may be provided from a conventional server or directly installed or otherwise installed on a client device.

[0124] The functions described herein may be performed at least in part by one or more hardware logic components. For example, but not limited to, illustrative types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), etc.

[0125] The program code for executing the method of the subject matter described herein can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device so that the program code enables the functions / operations specified in the flow chart and / or block diagram to be implemented when executed by the processor or controller. The program code can be executed in whole or in part on the machine, in part on the machine as a stand-alone software package, in part on a remote machine, or in whole on a remote machine or server.

[0126] In the context of the present disclosure, a machine-readable medium may be any tangible medium that may contain or store a program used by or in conjunction with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0127] In addition, although operations are shown in a particular order, this should not be understood as requiring such operations to be performed in the particular order shown or in sequence, or performing all operations shown to achieve desired results. In some cases, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be interpreted as limitations on the scope of the subject matter described herein, but should be interpreted as descriptions of features that may be specific to a particular embodiment. Certain features described in the context of a separate embodiment may also be implemented in combination in a single implementation. On the contrary, the various features described in a single implementation may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0128] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter specified in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

[0129] From the foregoing, it should be understood that specific embodiments of the presently disclosed technology have been described herein for illustrative purposes, but that various modifications may be made without departing from the scope of the present disclosure. Accordingly, the technology of the present disclosure is not to be limited, except as in the appended claims.

[0130] The embodiments of the subject matter and functional operations described in the present disclosure can be implemented in various systems, digital electronic circuits or computer software, firmware or hardware (including the structures disclosed in this specification and their structural equivalents) or a combination of one or more of them. The embodiments of the subject matter described in this specification can be implemented as one or more computer program products, that is, one or more modules of computer program instructions encoded on a tangible and non-transitory computer-readable medium, for being executed by a data processing device or controlling the operation of the data processing device. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a material combination that affects a machine-readable propagation signal, or a combination of one or more of them. The term "data processing unit" or "data processing device" covers all devices, equipment and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. In addition to hardware, the device can also include code that creates an execution environment for the computer program in question, for example, code that constitutes a processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0131] A computer program (also referred to as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or portions of code). A computer program may be deployed to execute on one computer or multiple computers located at one site or distributed over multiple sites and interconnected by a communications network.

[0132] For example, processors suitable for executing computer programs include general and special purpose microprocessors, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include or be operably coupled to receive data from or transmit data to or both of one or more large-capacity storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data. However, a computer need not have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices. The processor and memory can be supplemented by or incorporated into a dedicated logic circuit.

[0133] The specification, together with the drawings, are intended to be considered exemplary only, wherein exemplary means example.As used herein, the use of "or" is intended to include "and / or" unless the context clearly dictates otherwise.

[0134] Although the present disclosure contains many details, these details should not be interpreted as limitations on the scope of any disclosure or what may be claimed, but rather as descriptions of features that may be specific to a particular disclosed embodiment. Certain features described in the present disclosure in the context of separate embodiments may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple embodiments. In addition, although features may be described above as functioning in certain combinations and even initially claimed as such, in some cases, one or more features from the claimed combination may be removed from the combination, and the claimed combination may involve a sub-combination or a variation of the sub-combination.

[0135] Similarly, although operations are shown in a particular order in the accompanying drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in sequence, or that all of the illustrated operations be performed, to achieve the desired results. In addition, the separation of various system components in the embodiments described in this disclosure should not be understood as requiring such separation in all embodiments. Only a few embodiments and examples are described, and other embodiments, enhancements, and variations can be made based on what is described and illustrated in this disclosure.

Claims

1. A multi-agent debate method, comprising: Obtain a set of responses to the task expressed in natural language, the set of responses being generated by a plurality of agents in at least one round of debate on the task; updating the response set by at least one of: selecting a response from the response set, or adjusting at least one response in the response set; as well as A result for the task is generated by providing the updated set of responses to the plurality of agents as input for subsequent rounds of the debate.

2. The method of claim 1 , wherein selecting a response from the set of responses comprises: determining a quality metric for each response in a first plurality of responses based on the task and the response, the first plurality of responses being included in the set of responses, the quality metric representing a relevance of the response to the task; as well as A first number of responses is selected from the first plurality of responses based on the quality metric for each response in the first plurality of responses.

3. The method of claim 2, wherein determining a quality metric for each response in the first plurality of responses based on the task and the response comprises: Generate a first representation for the task and a second representation for the response by using an embedding model for the natural language; as well as A similarity between the first representation and the second representation is determined as the quality metric for the response.

4. The method of claim 1 , wherein selecting a response from the set of responses comprises: determining a diversity metric for each pair of responses in a second plurality of responses included in the set of responses, the diversity metric representing mutual information for the pair of responses; as well as A second number of responses is selected from the second plurality of responses based on the diversity metric for each pair of responses.

5. The method of claim 4, wherein determining a diversity metric for each pair of responses in the second plurality of responses comprises: generating a pair of representations for the pair of responses by using an embedding model for the natural language; as well as The diversity measure for the pair of responses is determined based on a similarity between the pair of representations.

6. The method of claim 1 , wherein adjusting at least one response in the set of responses comprises: assigning a target agent from the plurality of agents to a given response from the set of responses, the given response not being generated by the target agent; Providing prompt information to the target intelligent agent to correct the given response; as well as An adjusted response for the given response is derived from the output of the target agent.

7. The method of claim 1 , wherein updating the response set comprises: selecting a first subset of responses from the set of responses based on a relevance of each response in the set of responses to the task; selecting a second subset of responses from the first subset of responses based on mutual information of each pair of responses in the first subset of responses; as well as An updated set of responses is determined based on the second subset of responses.

8. The method of claim 7, wherein determining the updated set of responses based on the second subset of responses comprises: For responses in the second response subset, adjusting the response by prompting a target agent among the plurality of agents to correct the response, the response not being generated by the target agent; as well as The adjusted response is determined as a response in an updated set of responses.

9. The method of claim 1, wherein the set of responses comprises responses generated by the plurality of agents in a plurality of previous rounds of the debate.

10. An electronic device comprising a computer processor coupled to a computer readable memory unit, the memory unit comprising instructions which, when executed by the computer processor, implement a method of multi-agent debate, the method comprising: Obtain a set of responses to the task expressed in natural language, the set of responses being generated by a plurality of agents in at least one round of debate on the task; updating the response set by at least one of: selecting a response from the response set, or adjusting at least one response in the response set; as well as A result for the task is generated by providing the updated set of responses to the plurality of agents as input for subsequent rounds of the debate.

11. The apparatus of claim 10, wherein selecting a response from the set of responses comprises: determining a quality metric for each response in a first plurality of responses based on the task and the response, the first plurality of responses being included in the set of responses, the quality metric representing a relevance of the response to the task; as well as A first number of responses is selected from the first plurality of responses based on the quality metric for each response in the first plurality of responses.

12. The apparatus of claim 11, wherein determining a quality metric for each response in the first plurality of responses based on the task and the response comprises: Generate a first representation for the task and a second representation for the response by using an embedding model for the natural language; as well as A similarity between the first representation and the second representation is determined as the quality metric for the response.

13. The apparatus of claim 10, wherein selecting a response from the set of responses comprises: determining a diversity metric for each pair of responses in a second plurality of responses included in the set of responses, the diversity metric representing mutual information for the pair of responses; as well as A second number of responses is selected from the second plurality of responses based on the diversity metric for each pair of responses.

14. The apparatus of claim 13, wherein determining a diversity metric for each pair of responses in the second plurality of responses comprises: generating a pair of representations for the pair of responses by using an embedding model for the natural language; as well as The diversity measure for the pair of responses is determined based on a similarity between the pair of representations.

15. The apparatus of claim 10, wherein adjusting at least one response in the set of responses comprises: assigning a target agent from the plurality of agents to a given response from the set of responses, the given response not being generated by the target agent; Providing prompt information to the target intelligent agent to correct the given response; as well as An adjusted response for the given response is derived from the output of the target agent.

16. The apparatus of claim 10, wherein updating the response set comprises: selecting a first subset of responses from the set of responses based on a relevance of each response in the set of responses to the task; selecting a second subset of responses from the first subset of responses based on mutual information of each pair of responses in the first subset of responses; as well as An updated set of responses is determined based on the second subset of responses.

17. The apparatus of claim 16, wherein determining the updated set of responses based on the second subset of responses comprises: For responses in the second response subset, adjusting the response by prompting a target agent among the plurality of agents to correct the response, the response not being generated by the target agent; as well as The adjusted response is determined as a response in the updated set of responses.

18. The apparatus of claim 10, wherein the set of responses comprises responses generated by the plurality of agents in a plurality of previous rounds of the debate.

19. A computer program product, the computer program product comprising a non-transitory computer-readable storage medium having program instructions implemented thereon, the program instructions being executable by an electronic device to cause the electronic device to perform a multi-agent debate method, the method comprising: Obtain a set of responses to the task expressed in natural language, the set of responses being generated by a plurality of agents in at least one round of debate on the task; updating the response set by at least one of: selecting a response from the response set, or adjusting at least one response in the response set; as well as A result for the task is generated by providing the updated set of responses to the plurality of agents as input for subsequent rounds of the debate.

20. The computer program product of claim 19, wherein selecting a response from the set of responses comprises: determining a quality metric for each response in a first plurality of responses based on the task and the response, the first plurality of responses being included in the set of responses, the quality metric representing a relevance of the response to the task; as well as A first number of responses is selected from the first plurality of responses based on the quality metric for each response in the first plurality of responses.

Images