Method, apparatus, medium and program product for multi-agent debate

Abstract

The embodiment of the invention relates to a method, equipment, medium and program product for multi-agent debate. In a multi-agent debate scheme, an embedded communication protocol is introduced in a multi-agent debate process, and the problem of potential suboptimum caused by information loss when a natural language token is sampled is solved. Specifically, the token sampling process is eliminated from a multi-agent debate, and the embedded expectation is output through the original transformer such that agents communicate their trust across the entire vocabulary.

Description

Technical Field

[0001] Example embodiments of the present disclosure relate generally to computer science techniques, and particularly to multi-agent debate. Background Art

[0002] Conversational platforms allow users to ask questions and then provide answers to the user's questions. Users typically enter questions in natural language. Today, conversational platforms rely on models such as language models to perform reasoning tasks on user questions and provide natural language responses. There are various works to strengthen the reasoning capabilities of the models in order to improve the accuracy of the generated responses. Summary of the invention

[0003] According to the implementation of the technical solution described in this article, a solution for multi-agent debate is provided. In this solution, it is proposed to introduce an embedding communication protocol in the multi-agent debate process, which solves the potential suboptimality caused by information loss when sampling natural language tokens. Specifically, the token sampling process is eliminated from the multi-agent debate, and the expectations of the embedding are output by the original transformer, so that the agents communicate their trust across the entire vocabulary.

[0004] This section is provided to introduce some concepts in a simplified form, which are further described in the following detailed description. This section is neither intended to identify the key features or essential features of the technical solutions described herein, nor is it intended to be used to limit the scope of the technical solutions described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The above and other objects, features and advantages of the exemplary embodiments disclosed herein will become more understandable through the following detailed description with reference to the accompanying drawings, in which several exemplary embodiments disclosed herein are illustrated by way of example and not limitation, wherein:

[0006] Figure 1 A block diagram illustrating an example environment in which various embodiments of the present disclosure may be implemented;

[0007] Figures 2A-2D An example of a traditional model debate is shown;

[0008] Figure 3A An example of multi-agent debate via embedding is shown according to some example embodiments of the present disclosure;

[0009] Figure 3B A block diagram illustrating an architecture for multi-agent debate via embedding according to some example embodiments of the present disclosure;

[0010] Figure 4 An example algorithm for multi-agent debate via embedding is shown according to some example embodiments of the present disclosure;

[0011] Figures 5A-5C Experimental results on the proposed method according to some example embodiments of the present disclosure are shown;

[0012] Figure 6 A flowchart showing a process for model training according to some example embodiments of the present disclosure; and

[0013] Figure 7 A block diagram of a computing device / server is shown in which one or more embodiments of the present disclosure may be implemented. DETAILED DESCRIPTION

[0014] The principle of the present disclosure will now be described with reference to some example embodiments. It should be understood that these embodiments are described only for illustrative purposes to help those skilled in the art understand and implement the present disclosure without limiting the scope of the present disclosure. The disclosure described herein can be implemented in various ways except for the way described below.

[0015] In the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0016] References in this disclosure to "one embodiment," "an embodiment," "an example embodiment," etc. indicate that the described embodiment may include a particular feature, structure, or characteristic, but not every embodiment necessarily includes the particular feature, structure, or characteristic. Moreover, such phrases do not necessarily refer to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in conjunction with an example embodiment, whether or not explicitly described, it is considered within the knowledge of those skilled in the art to affect embodiments related to such features, structures, or characteristics in other embodiments.

[0017] It should be understood that although the terms "first" and "second" etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element without departing from the scope of the exemplary embodiments. As used herein, the term "and / or" includes any and all combinations of one or more of the listed terms.

[0018] The terms used herein are only used for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments. As used herein, the singular forms "a", "an", and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the terms "comprises", "comprising", "having", when used herein, specify the existence of stated features, elements and / or components, etc., but do not exclude the existence or addition of one or more other features, elements, components and / or combinations thereof.

[0019] As used herein, the term "model" refers to the association between input and output learned from training data, and therefore the corresponding output can be generated for a given input after training. The generation of the model can be based on machine learning technology. Machine learning technology can also be referred to as artificial intelligence (AI) technology. In general, a machine learning model can be established to receive input information and make predictions based on the input information. For example, a classification model can predict the category of input information in a predetermined set of categories. As used herein, "model" may also be referred to as "machine learning model", "learning model", "machine learning network" or "learning network", which are used interchangeably herein.

[0020] In general, machine learning can usually involve three stages, namely the training stage, the verification stage, and the application stage (also called the reasoning stage). In the training stage, a given machine learning model can be iteratively trained (or optimized) using a large amount of training data until the model can obtain consistent reasoning from the training data similar to that of human intelligence. During training, a set of parameter values ​​of the model are iteratively updated until the training goal is achieved. Through the training process, the machine learning model can be considered to be able to learn the association between input and output (also called input-output mapping) from the training data. In the verification stage, the verification input is applied to the trained machine learning model to test whether the model can provide the correct output, thereby determining the performance of the model. In general, the verification stage can be considered as a step in the training process, or sometimes it can be omitted. In the application stage, the resulting machine learning model can be used to process real-world model inputs based on a set of parameter values ​​obtained from the training process and determine the corresponding model outputs.

[0021] Figure 1 A block diagram of an example environment 100 is shown in which various embodiments of the present disclosure may be implemented. Figure 1In the environment of FIG. 1 , the dialogue platform 110 may receive questions from the user 130. The question may be provided in natural language, for example, “If Ann is 9 years old, and her brother is twice as old as her, how old will her brother be in 3 years?” The dialogue platform 110 includes one or more agents 120-1, 120-2, ..., 120-N (collectively or individually referred to as agents 120, where N is an integer greater than or equal to 1) to process questions from the user 130 and provide a final answer to the question, for example, “Answer: 21”.

[0022] Agents 120-1, 120-2, ..., 120-N are configured to process natural language problems using machine learning models 125-1, 125-2, ..., 125-N (collectively or individually referred to as machine learning models 125, or simply referred to as models 125). In some embodiments, model 125 can be a language model that is trained on a large-scale corpus to learn the ability to understand one or more types of natural language. In some embodiments, model 125 can be a generative model for content generation. Model 125 can be constructed based on various machine learning techniques.

[0023] Various approaches have been proposed to improve the reasoning capabilities of machine learning models, especially language models.

[0024] One way is to improve the reasoning ability in the language model via hints. There are various works that use contextual learning to strengthen the reasoning ability of the model. In the solution, it is proposed to use a series of intermediate reasoning steps to enhance the ability of large language models for complex reasoning tasks. Another solution generalizes this idea to use forward prediction and backtracking when necessary to explore multiple different intermediate steps for the best reasoning path. These techniques, along with rare examples, enable language models to improve their performance on a variety of tasks.

[0025] Another way is self-improvement of language models via feedback. From the observation that humans repeatedly improve their initial answers to have better versions, there is a wide range of work trying to apply this concept to language models. In the solution, it is proposed to adopt a language model to play the role of generator and critic. First, the model generates an initial output, and then the same model provides feedback to itself to have an improved version. It shows that with specific feedback, a strong model can improve its performance. However, this strategy does not work for models with less ability because it is found that it is difficult to continuously generate feedback in the required format. In the same line of work, another solution uses verbal reinforcement as a means to instruct the agent to learn from its previous mistakes. The strategy is tested in a trading game setting, where one language model acts as a balloon seller and the other plays the role of the buyer. The goal is to get the best deal after negotiation. One of the agents will receive feedback after each round to improve. The inventors of the present disclosure found that the buyer role is more difficult to improve than the seller role, and only strong and well-aligned models can be continuously improved from iterative model feedback. In other works, these solutions focus on improving the encoding ability of language models by using natural language to interpret the generated code and the execution results from unit tests to make the language model self-debug. They reveal that a good setting is to use a different model for each role, specifically, a critic who is better than the generator provides feedback to a less powerful model, or a human gives feedback to a well-trained model. In addition, self-healing of weaker models does not achieve improvements compared to the majority voting mechanism.

[0026] Recently, a new line of research, multi-agent debate, has attracted significant attention, which is to improve the initial output from language models through interaction. It involves language models providing initial direct responses, followed by the opportunity for these models to inspect and combine the answers of other models to improve their confidence. The method encourages divergent thinking in the language model and removes bottlenecks when feedback is given. Prior work has shown that this method outperforms self-improvement on various datasets. However, these works adopt strong agents for debate purposes. Therefore, it is unclear whether less strong open source models can also benefit from participating in debates.

[0027] Discussion and debate between models have seen a resurgence of interest recently. However, despite the presence of inter-model interactions, communication in existing research is all conducted in natural language, raising interest in the necessity and efficiency of asking questions. The recent surge in research on discussion and debate between models demonstrates the great potential of language models to participate in collaborative communication. Similar to the human decision-making process when opinions diverge, language models can also use discussion and debate to improve their responses. A recent series of studies have explored in depth the adaptation of this form of social behavior to language models. In particular, it has been shown that debate can significantly benefit the quality of the generated responses, as long as at least one of the participating models is a strong language model. Notably, it was also found that the effectiveness of self-repair was uniquely seen in language models in code generation tasks. Similarly, the capabilities of language models in trading game settings have been tested, and it was found that only strong and well-aligned models can continuously improve their responses through feedback. These findings collectively suggest that current communication protocols may not be sufficient for less powerful models to effectively benefit from feedback. Therefore, the benefits of discussion and debate between models of other scales and architectures remain to be fully revealed.

[0028] One potential factor that can limit the effectiveness of debate among language models may be the reliance on natural language for communication. While this is somewhat unavoidable, since language models are trained to understand and generate natural language, it can also be suboptimal. Specifically, the natural language generation process requires sampling tokens from a distribution over the entire vocabulary, which can pose a risk of information loss. The inventors have discovered such information loss from empirical studies. This can be illustrated using the following examples.

[0029] In the example, in the inference task, the most trusted token selected by the model may be wrong. Figure 2A In the example of traditional model debate shown in FIG, when performing reasoning on arithmetic problem 201, the most plausible token (“6”) is wrong, while the correct token (“5”) has the second highest probability. Therefore, it is beneficial to consider the first k tokens with k ≥ 2. Figure 2B In the example of a traditional model debate shown, the agent 202 provides various candidate answers 210, 212, 214 to a question 205, which is achieved by utilizing various token sequences. Response correction is performed based on majority voting to determine the final answer 216. This indicates that the greedy approach to token selection may not be optimal.

[0030] Previous work on multi-agent debate has only used natural language for inter-model communication. Figure 2CIn the illustrated example, regarding natural language question 207, in a first round of debate, agent 202 generates an initial answer 220 to question 207 in natural language, and agent 204 generates an initial answer 230 to question 207 in natural language. In a second round of debate, agent 202 generates an updated answer 222 in natural language after considering initial answer 230 from agent 204. Similarly, in a second round of debate, agent 204 generates an updated answer 232 in natural language after considering initial answer 220 from agent 202. The agents present and debate their individual responses and reasoning processes in multiple rounds to arrive at a common final answer to question 207.

[0031] Figure 2D A schematic architecture 250 of a conventional model debate for natural language is shown. For input in natural language, a tokenizer 252 is used to convert prompt words into natural language to obtain embeddings. Initially, the initial input includes the prompt word "prompt" based on the answer from the user. Then, after debate with other agents, the input is updated, for example, by considering the answers from other agents (vocab (1) ,…vocab (l-1) ). The embedding outputs from the tokenizer 252 include emb(prompt), emb(vocab (1) )、…、emb(vocab (l-1) ), which is input to agent 254 as model input. Agent 254 provides the probabilities of various embeddings (V embeddings), which are converted into corresponding natural language vocabulary vocab1, vocab2…, vocab V Then, we sample natural language vocabulary vocab from V natural language vocabularies. l , and concatenate it with the previous natural language input for tokenization. This process iterates until the final answer is obtained.

[0032] As discussed above, there is a need to improve the output of natural language responses from the multi-agent debate process. Since the sampling process for natural language response generation naturally raises the question of whether the language model communicates more effectively if it is not restricted to natural language, information loss can be observed.

[0033] In an embodiment of the present disclosure, it is proposed to introduce an embedding communication protocol in the multi-agent debate process instead of using natural language for inter-model communication, which solves the potential suboptimality caused by information loss when sampling natural language tokens. Specifically, the token sampling process is eliminated from the multi-agent debate, and the agents are made to communicate their trust in the entire vocabulary through the expectation of the original transformer output embedding.

[0034] Significantly, even without any modifications to the model weights, the content of communication between models deviates from natural language, allowing for more robust inter-model trust information transfer, especially when the models are uncertain about their sampled outputs. Extensive experiments have shown that discussions and debates in open source models consistently benefit from this novel change in communication protocol compared to their natural language counterparts in a variety of reasoning tasks.

[0035] Some example embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0036] Figure 3A An illustrative example of multi-agent debate via embedding according to some example embodiments of the present disclosure is shown.

[0037] In the embodiments of the present disclosure, in the same spirit, a multi-agent debate setting is adopted. However, in contrast to previous work, a novel form of communication is introduced in which agents can interact without using natural language. Models (or agents) are enabled to directly output embedding vectors, and also use embedding vectors from other models as input during debates. Although designed for inter-model communication, it is possible to convert them back to natural language through nearest neighbor search.

[0038] like Figure 3A As shown, with respect to a question 301 in natural language, agent 120-1 generates an initial response embedding 310, which can be converted into a natural language response 330, and agent 120-2 also generates an initial response embedding 320, which can be converted into a natural language response 340. The response embeddings 310 and 320 are in the form of raw vectors to avoid information loss from the token sampling process. Agent 120-1 also obtains the initial response embedding 320 from agent 120-2 as its input, as well as other response embeddings (if available) from other agents involved in the multi-agent debate process. Similarly, agent 120-2 also takes the initial response embedding 310 from agent 120-1 and other response embeddings from other agents (if available) as input to agent 120-2. In this round of debate, agent 120-1 can generate a new response embedding 312 to question 301, which can be converted into a natural language response 332. Agent 120-2 can also generate a new response embedding 314 to question 301, which can be converted into a natural language response 342. More than one round may be performed until final response embedding is achieved among the agents.It should be understood that although two agents are shown, more than two agents may be involved in the multi-agent debate process, and the number of agents may be set according to actual applications.

[0039] In embodiments of the present disclosure, it is certain that models or agents can communicate more effectively if they are not limited to natural language but can communicate within the embedding space of tokenizers in a multi-agent debate setting. This novel communication mechanism bypasses the sampling process from the vocabulary set, thereby retaining rich information from the lower credibility vocabulary. Although this form of communication deviates from natural language, it is still interpretable to humans via nearest neighbor search in the vocabulary.

[0040] Through the embodiments of the present disclosure, models or agents can communicate effectively without relying on natural language, and even achieve excellent results. The inventors have demonstrated this by evaluating the methods proposed in this article on different datasets across multiple domains, and showed significant performance improvements over natural language counterparts.

[0041] On the one hand, the experimental results align with previous observations that weaker models do not benefit from debate or self-improvement when communicating via natural language compared to majority voting methods. On the other hand, the experiments show that communication in embeddings can promote more effective information transfer between models, unlocking the benefits of debate even for less powerful models. This suggests that debate is still an effective form of communication to improve model performance, even for smaller open source models.

[0042] In some embodiments of the present disclosure, a novel communication protocol is introduced for models that share the same tokenizer, regardless of whether they use the same or different embedding-to-vocabulary mappings.

[0043] In some embodiments of the present disclosure, comprehensive experiments involving multiple open source models across five different datasets have been performed to verify the effectiveness of the communication protocol proposed in this paper. The experimental results show that even less powerful models can still benefit from debate.

[0044] In some embodiments of the present disclosure, extensive ablation studies are conducted to elucidate the mechanisms that make communication in embeddings more effective for debate between models.

[0045] In the following, we first introduce the embedded communication protocol for debating between models. Figure 3B A block diagram of an architecture 350 for multi-agent debate via embedding is shown, according to some example embodiments of the present disclosure. The architecture 350 shows an iterative multi-agent debate process at an agent among multiple agents.

[0046] When generating a response, the agent 120 or the model 125 used (e.g., a causal language model) automatically generates tokens one at a time based on the words that appear before each token. With its corresponding embedding set Where [V] = {1, ..., V} is an index set. Let "prompt" and response (l-1) are the prompt word and the first (l-1) generated response tokens, respectively. Here, prompt contains at least the question provided by the user. In some embodiments, prompt also contains instructions to be input to the model, and (possibly) responses collected in previous debate rounds. In the last layer of model 125 (e.g., causal language model), the vector of logistic regression is calculated To generate the next token, vocabulary collection is performed according to the following distribution Sampling on: where T>0 is the temperature. It should be noted that when T→0, sampling degenerates to the selection mechanism token generation. The distribution p (l) can be viewed as the model's trust about which token is the most appropriate one at the current position. However, when sampling a realization value from the distribution as output, all other information contained in the model's trust is discarded. At the cost of this information loss, accurately presenting an acquired vocabulary at each position is necessary for humans to understand the output of the language model. However, in the scenario of model debate, it is not always necessary to understand the inter-model message. As discussed above, rich information is contained in the model's trust. Therefore, the token sampling process at the cost of information readability becomes a point that needs to be paid attention to in inter-model communication.

[0047] In an embodiment of the present disclosure, in order to exploit the rich information encoded in the trust of the model, it is proposed to allow the model 125 to communicate in an embedded form during debate.

[0048] In general, it is desirable to encode as much trust information as possible during inter-model communication. However, given that models 125 are designed to understand natural language sequences, they may not be able to comprehend vectors that reside outside the convex hull of the tokenizer’s embedding space. It is proposed to use a weighted average of embeddings to replace tokens in the autoregressive response generation process, rather than targeting the distribution p. (l) The concatenation of the word embeddings of prompt is represented by emb(prompt). Initially, the embedding generation process is defined autoregressively as Can be a vocabulary The embedding generation process of model 125 continues until (i) the maximum sequence length is reached, or (ii) the EOS token embedding becomes the nearest neighbor of the newly generated embedding. Figure 3B A complete diagram of the response generation process is shown in .

[0049] The response embeddings are fed into the debate process using embedded representations of responses (referred to as “response embeddings”) from the collected multiple models 125 (also referred to as debaters). To start the debate, a tokenizer is used to convert questions and (possibly) instructions into embedded emb(prompt) (referred to as “query embeddings”). Then, for a debate turn, multiple models 125 (i.e., model debaters) import embedded questions, instructions, and (potentially) responses from previous turns to generate their embedded responses according to Equation 2. Figure 4 An example algorithm 400 for multi-agent debate via embedding is shown, according to some example embodiments of the present disclosure.

[0050] In reinforcement learning, the original SARSA uses the next sampled state-action pair to update the Q-value. Its variant replaces the sampled Q-value with the expected value over all next actions, leading to significant advantages over the original SARSA. For causal language models, the autoregressive token generation process can be viewed as a Markov decision process, where at time step l, the state is the previously generated response response (l) , the action space is the vocabulary set Strategy p defined in Equation 1 (l) , and the reward is associated with the accuracy of the response. To this end, the weighted average of the embeddings shares the same spirit as expected SARSA, computing the expectation of possible tokens. Meanwhile, the natural language response generation process that probabilistically samples tokens is aligned with the original SARSA.

[0051] To illustrate the effectiveness of multi-agent debate on embeddings as discussed above, various open source models are tested across multiple reasoning tasks.

[0052] In some embodiments of the present disclosure, the method proposed herein (denoted as "CIPHER") is benchmarked against the following three baselines: (i) Single Answer: a language model gives a single direct response to a question in natural language, (ii) Self-Consistency: a language model independently generates multiple responses to each question, and then majority voting is applied to determine the final answer, and (iii) Natural Language Questions: each agent first gives an initial response to the question. Subsequently, the models use each other's answers to improve their initial responses. After the final round of debate, the final answer is selected from the model with the lower temperature.

[0053] The natural language debate baseline is the most direct comparison point with the approach proposed in this paper, as it reflects debate in addition to its natural language communication protocol.

[0054] Most experiments are conducted on LLaMA2-70B, which is currently the largest open source model available, which also supports long context windows of up to 4000 tokens. The proposed method is also experimented on other models such as LLaMA-65B, Falcono-40B, MPT-30B, and WizardMath-70B-V1.0 to ensure the robustness and generalization ability of the proposed method.

[0055] Figure 5A Table 1 in shows the accuracy of debate between two models of the same LLaMA family operating at different temperatures. Both self-consistency (Major@5) and natural language debate improve the performance of directly generating a single response (Single Answer). CIPHER debate further increases the gap, consistently outperforming natural language debate by xy%.

[0056] Regarding the prompt words, in the experiments, rare examples are used in combination with thought chain prompts and zero-shot learning instructions ("Let's think step by step") to encourage the agent to generate not only the final answer, but also the reasoning steps leading to it. This helps improve response accuracy and also provides important information for other language models in the whole debate. For each open source language model, only the prompt words are adjusted until the single answer accuracy matches the numbers reported in the literature. It is worth noting that the method proposed in this paper is orthogonal to the choice of prompt word technology. Therefore, alternative prompt word engineering methods can also be easily adopted on top of CIPHER.

[0057] Regarding the datasets used for experiments, CIPHER is evaluated on 6 reasoning datasets across 4 different domains: (i) GSM8K: evaluated on 200 questions randomly sampled from approximately 1K of its test set, (ii) MMLU (Hendrycks et al., 2020): three datasets from three different categories are selected to evaluate the proposed method. Specifically, the formal logic dataset is selected from the human category, the high school mathematics dataset from the STEM category, and the professional psychology dataset from the social science category, (iii) Arithmetic: the model is evaluated on 200 mathematical expressions consisting of six unique two-digit numbers including addition, multiplication, and subtraction operations, and (iv) Biology: Factual is evaluated by asking the model to generate biographies of famous computer scientists.

[0058] For debates between the same language models operating at different temperatures, the effectiveness of the debate is evaluated based on the correctness of the last round response generated by the language model with the lowest temperature. For CIPHER debaters, their final embedded responses are converted back to natural language for evaluation purposes using nearest neighbor search. To evaluate the effectiveness of debates between different language models sharing the same tokenizer, the correctness of all last round responses is tested.

[0059] As follows, the performance of CIPHER debate is evaluated against the baseline in a 3-round debate between two language models. The choice of debate rounds and participants is based on observations from some previous works.

[0060] A comprehensive evaluation of CIPHER debates is performed on 6 reasoning datasets using the LLaMA family of language models (LLaMA-65B and LLaMA2-70B). Table 1 presents the results of debates between two identical LLaMA family language models operating at different temperatures. It should be noted that the evaluation of the debates is based on the final response of the agent with the lowest temperature, resulting in a total of 5 responses per debate. For fair comparison, the self-consistency baseline (labeled as Major@5) also utilizes 5 responses.

[0061] Figure 5B and Figure 5C Tables 2 and 3 in present the results of the debate between LLaMA-65B and LLaMA2-70B on the arithmetic and GSM8K datasets, respectively. Table 2 shows the debate between LLaMA2-70B and LLaMA-65B on the arithmetic dataset. The proportion of response changes compared to the previous round is described in parentheses. Table 3 shows the debate between LLaMA2-70B and LLaMA-65B on the GSM8K dataset. The proportion of response changes compared to the previous round is described in parentheses.

[0062] When facilitating a CIPHER debate between LLaMA-65B and LLaMA2-70B, even though both models utilize the same tokenizer, they possess different vocabulary embedding mappings. This issue can be addressed by maintaining a bijective mapping between the vocabulary embeddings of both models. While the debate was beneficial for both agents, LLaMA-65B experienced substantial improvements, increasing its accuracy from 35% to 60.5%. It was also observed that the less powerful agents tended to improve their responses more during the debate.

[0063] While recent research has shown rapidly diminishing marginal returns to increasing the number of debaters and rounds, these cases were included for the sake of a comprehensive evaluation. This allows for a more complete picture of the performance of the CIPHER debates.

[0064] Regarding temperature sensitivity, in natural language debates, the best performance is usually achieved when all debaters are set to low temperatures. This can be attributed to the inherent information loss during the token sampling process in natural language generation. At higher temperatures, language models tend to generate insensitive or misleading responses, which may even undermine the performance of stronger word segmenters. In contrast, for CIPHER debates, the best performance is usually obtained when the debaters are set to a wider temperature distribution. This is most likely because CIPHER retains the trust of the language model over the entire vocabulary set. At higher temperatures, the information in the CIPHER embedding tends to be less credible vocabulary choices, effectively supplementing the information transmitted by other CIPHER debaters operating at lower temperatures. It is also worth noting that in order to convert the returned final response into a readable natural language by nearest neighbor search, it is necessary to have at least one CIPHER debater set at a low temperature.

[0065] The problem of position bias when utilizing language models as evaluators has received increasing attention in recent studies. Although the proposed multi-agent debate setting differs from the “language model as evaluator” in that it should be recognized that the order in which previous turn responses are fed into subsequent turns of the debate still has a non-negligible impact on the outcome of the debate.

[0066] To elucidate the mechanisms that contribute to the performance gain brought by CIPHER, an ablation study is performed using partial CIPHER response generation. Specifically, CIPHER response generation is invoked only at positions where the model exhibits high uncertainty about the next token. To quantify the uncertainty at a given position l, a generation distribution p is employed over the vocabulary set. (l) Shannon entropy When switching to CIPHER response generation when the entropy exceeds this threshold, the threshold is set to ε>0, and whenever H(p (p) )<ε adopts natural language token sampling. This targeted application of CIPHER is used to illustrate the importance of information preservation in moments of model uncertainty.

[0067] Although existing research has shown significant performance enhancements achieved through language model debate. However, the limitations of this performance gain remain an area of ​​research concern. In order to investigate the upper limit of performance that can be achieved through multi-agent debate, experiments were conducted in which language model debaters were pitted against professional word segmenters. Professional word segmenters are represented by making them always give true answers. Conversely, in order to establish a lower limit on performance, experiments were conducted in which the model debaters constantly received insensitive feedback from other word segmenters. Specifically, 2 fake debaters were employed, one with an extremely high temperature that gave no perception, and the other giving non-relevant responses by using unaligned true answers from other questions in the batch. It has been shown that debate can be harmful when the model has low capacity, but it does not cause much damage in the case of more powerful models.

[0068] In the disclosed embodiments, the proposed CIPHER debate demonstrates promising results across various reasoning tasks. Without the need for special training, CIPHER enhances the debate effectiveness of smaller open source language models, outperforming majority voting and natural language debate. Our research not only highlights the great potential of trust information for language models, but also serves as the first step to fully unlock such potential in language model debate. Given the fact that language models are only trained to ingest embeddings of natural language text tokens, it remains to be inferred whether there is a more efficient way to pass trust information between language models.

[0069] The current limitation of the applicability of CIPHER is critically focused on language models with a shared vocabulary. Extending the proposed approach to language models with different tokenizers requires strict alignment across their corresponding embedding-vocabulary mappings. Such alignment is challenging due to the fact that tokenizers often adopt very dissimilar text segmentation strategies. For example, while one tokenizer may decompose a word into subwords or characters, another tokenizer may recognize it as a whole token. Overcoming this bottleneck can pave the way for building more robust and efficient language model proxy systems, potentially unlocking unprecedented collaborative capabilities across different language models.

[0070] Figure 6 FIG. 6 is a flowchart of a process 600 for model training according to some example embodiments of the present disclosure. The process 600 may be implemented in Figure 1 At the dialogue platform 110 shown.

[0071] At box 610, the dialogue platform 110 generates a first response embedding based on the query embedding of the question by a first agent among the multiple agents, and the first response embedding can be converted into a first candidate answer in natural language to the question, and each agent among the multiple agents uses a corresponding model to generate a response.

[0072] At box 620, the dialogue platform 110 obtains at least one second response embedding from at least one other agent among the multiple agents, the at least one second response embedding is generated by the at least one other agent based on the query embedding of the question, and the at least one second response embedding is convertible into at least one second candidate answer in natural language to the question.

[0073] At block 630 , the conversation platform 110 updates, by the first agent, the first response embedding to a third response embedding based on the query embedding and the at least one second response embedding.

[0074] At block 640 , the dialogue platform 110 determines a target answer in natural language to the question based at least in part on the third response embedding and at least one fourth response embedding generated by at least one other agent.

[0075] In some embodiments, the first agent further provides the first response embedding to at least one other agent to generate at least one fourth response embedding to the query embedding.

[0076] In some embodiments, multiple agents share a tokenizer configured to map response embeddings into a vocabulary.

[0077] Figure 7 1 shows a block diagram of a computing device / server 700 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Figure 7 The computing device / server 700 shown in FIG. 7 is merely an example and should not suggest any limitation on the functionality and scope of the embodiments described herein.

[0078] like Figure 7 As shown, computing device / server 700 is in the form of a general-purpose computing device. Components of computing device / server 700 may include, but are not limited to, one or more processors or processing units 710, memory 720, storage device 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. Processing unit 710 may be an actual or virtual processor and may perform various processes based on a program stored in memory 720. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to increase the parallel processing capabilities of computing device / server 700.

[0079] The computing device / server 700 typically includes a plurality of computer storage media. Such media may be any available media accessible to the computing device / server 700, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 720 may be a volatile memory (e.g., registers, caches, random access memory (RAM)), a non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory) or any combination thereof. The storage device 730 may be any removable or non-removable media, and may include machine-readable media, such as a flash drive, a disk, or any other media that may be used to store information and / or data (such as training data for training) and may be accessed within the computing device / server 700.

[0080] The computing device / system 700 may also include additional removable / non-removable, volatile / non-volatile storage media. Although not described herein, Figure 7 , but a disk drive for reading from or writing to a removable, non-volatile disk (such as a "floppy disk"), and an optical drive for reading from or writing to a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) via one or more data media interfaces. Memory 720 may include a computer program product 725 having one or more program modules configured to perform various methods or actions of various embodiments of the present disclosure.

[0081] The communication unit 740 communicates with another computing device via a communication medium. In addition, the functions of the components in the computing device / system 700 can be implemented by a single computing cluster or multiple computing machines that can communicate via a communication connection. Therefore, the computing device / system 700 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0082] Input device 750 may be one or more input devices such as a mouse, keyboard, trackball, etc. External devices (such as storage devices, display devices, etc.) communicate with one or more devices that enable a user to interact with computing device / system 700, or communicate with any device (e.g., a network card, a modem, etc.) that enables computing device / system 700 to communicate with one or more other computing devices. Such communication may be performed via an input / output (I / O) interface (not shown).

[0083] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions or computer programs are stored, and the computer-executable instructions are executed by the processor to implement the above method. According to an exemplary implementation of the present disclosure, a computer program product is also provided. The computer program product is tangibly stored in a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by the processor to implement the above method.

[0084] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0085] These computer-readable program instructions can be provided to the processing unit of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine, and when these instructions are executed by a computer or other programmable data processing device, the machine generates a device for realizing the function / action specified in one or more boxes in the flow chart and / or the block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium. These instructions enable a computer, a programmable data processing device and / or other equipment to work in an ad hoc manner. Therefore, the computer-readable medium comprising instructions includes a product, and the product includes instructions for realizing the various aspects of the function / action specified in one or more boxes in the flow chart and / or the block diagram.

[0086] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps can be executed on the computer, other programmable data processing apparatus, or other device to generate a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0087] The flow charts and block diagrams in the accompanying drawings illustrate possible architectures, functions and operations of the systems, methods and computer program products implemented according to the present disclosure. In this regard, each box in the flow chart or block diagram may represent a part of a module, a program fragment or an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the box may also occur in an order different from those marked in the accompanying drawings. For example, two continuous blocks may actually be executed in parallel, and may also be executed in reverse order sometimes, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart and the combination of boxes in the block diagram and / or flow chart may be implemented by a dedicated hardware-based system or by a combination of dedicated hardware and computer instructions that performs a specified function or action.

[0088] Each implementation of the present disclosure has been described above. The above description is exemplary rather than exhaustive and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used in this disclosure is intended to best explain the principles, practical applications or improvements of the technology in each implemented market, or to enable other persons of ordinary skill in the art to understand the various embodiments disclosed herein.

Claims

1. A method comprising: A first agent among the plurality of agents generates a first response embedding based on a query embedding of the question, wherein the first response embedding is convertible into a first candidate answer in a natural language to the question, and each agent among the plurality of agents generates a response using a corresponding model; obtaining at least one second response embedding from at least one other agent among the plurality of agents, the at least one second response embedding being generated by the at least one other agent based on the query embedding of the question and convertible into at least one second candidate answer in the natural language to the question; updating, by the first agent, the first response embedding to a third response embedding based on the query embedding and the at least one second response embedding; as well as A target answer in the natural language to the question is determined based at least in part on the third response embedding and at least one fourth response embedding generated by the at least one other agent.

2. The method according to claim 1, further comprising: The first response embedding is provided to the at least one other agent to generate the at least one fourth response embedding to the query embedding.

3. The method of claim 1, wherein the plurality of proxies share a tokenizer configured to map response embeddings into a vocabulary.

4. A device comprising: at least one processor; as well as at least one memory communicatively coupled to the at least one processor and comprising computer readable instructions that, when executed by the at least one processor, cause the at least one processor to: A first agent among the plurality of agents generates a first response embedding based on a query embedding of the question, wherein the first response embedding is convertible into a first candidate answer in a natural language to the question, and each agent among the plurality of agents generates a response using a corresponding model; obtaining at least one second response embedding from at least one other agent among the plurality of agents, the at least one second response embedding being generated by the at least one other agent based on the query embedding of the question and convertible into at least one second candidate answer in the natural language to the question; updating, by the first agent, the first response embedding to a third response embedding based on the query embedding and the at least one second response embedding; as well as A target answer in the natural language to the question is determined based at least in part on the third response embedding and at least one fourth response embedding generated by the at least one other agent.

5. The apparatus of claim 4, wherein the computer-readable instructions further cause the at least one processor to: The first response embedding is provided to the at least one other agent to generate the at least one fourth response embedding to the query embedding.

6. The apparatus of claim 4, wherein the plurality of proxies share a tokenizer configured to map response embeddings into a vocabulary.

7. A non-transitory computer-readable storage medium storing computer-readable instructions, which when executed by a device cause the device to perform actions, the actions comprising: A first agent among the plurality of agents generates a first response embedding based on a query embedding of the question, wherein the first response embedding is convertible into a first candidate answer in a natural language to the question, and each agent among the plurality of agents generates a response using a corresponding model; obtaining at least one second response embedding from at least one other agent among the plurality of agents, the at least one second response embedding being generated by the at least one other agent based on the query embedding of the question and convertible into at least one second candidate answer in the natural language to the question; updating, by the first agent, the first response embedding to a third response embedding based on the query embedding and the at least one second response embedding; as well as A target answer in the natural language to the question is determined based at least in part on the third response embedding and at least one fourth response embedding generated by the at least one other agent.

8. The computer-readable storage medium according to claim 7, wherein: The actions also include: The first response embedding is provided to the at least one other agent to generate the at least one fourth response embedding to the query embedding.

9. The computer-readable storage medium of claim 7, wherein the plurality of agents share a tokenizer configured to map response embeddings into a vocabulary.

10. A computer program product tangibly stored in a computer storage medium and comprising computer executable instructions which, when executed by a device, cause the device to perform the method according to any one of claims 1 to 3.

Images