Evaluation method and device based on multi-model interaction

Abstract

The invention discloses an evaluation method and device based on multi-model interaction, and the method comprises the steps: building a multi-model interaction evaluation platform which supports a plurality of models to participate in debate; obtaining a debate theme; the multi-model interaction evaluation platform automatically distributes a debate site for the multiple models participating in debate, and meanwhile, the multiple models perform multiple rounds of debate based on the debate theme to obtain the performance of each model in the multiple rounds of debate; and performing quantitative analysis on the performance of each model in multiple rounds of debate, and automatically generating a performance report of each model based on a preset evaluation index. According to the evaluation method based on multi-model interaction, dynamic evaluation based on multi-model debate is introduced, so that the evaluation precision of the model in a complex scene can be greatly improved, and the limitation of existing evaluation is broken through. In addition, through combination of multiple rounds of debate and user preference data, developers can be helped to better understand and optimize the model, and comprehensive performance evaluation feedback is provided.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an evaluation method and device based on multi-model interaction. Background Art

[0002] At present, artificial intelligence models, especially large models, are widely used in many fields. With the continuous exploration of the application potential of large models, the application form has also evolved from single model calls to intelligent agents, and then to the use of multiple models, and various new application forms have gradually emerged. However, the existing large model evaluation methods mainly focus on the accuracy, reasoning ability and generation effect of a single model, and lack a comprehensive evaluation of the interaction capabilities of multiple models. Especially in model dialogue and debate scenarios, there is no systematic evaluation method for the interaction performance, conflict of opinions and collaboration effects between different models. In addition, the current model evaluation focuses on static evaluation and cannot accurately reflect the actual performance of the model in dynamic interaction scenarios. Specifically, the existing evaluation standards fail to cover the following points:

[0003] (1) Static evaluation is easy to be broken

[0004] The current model evaluation method is a relatively fixed way to test the model's capabilities. Static fixed-number data sets can easily be used by model manufacturers for targeted optimization. Currently, each manufacturer demonstrates the model's capabilities on several popular data sets (such as MMLU, GSM, etc.), but it is difficult to determine whether the model has been optimized for these data sets or whether the model's capabilities have been truly improved.

[0005] (2) Dynamic evaluation content

[0006] Although the latest Arena evaluation method (https: / / chat.lmsys.org) solves the problem of static input, this dynamic input is completely obtained by the user's free input. This kind of user's free input cannot guarantee the boundary of the model's ability, and there is no interaction between any models, which leads to certain limitations in the model evaluation results.

[0007] (3) The logical reasoning ability of the display model is insufficient

[0008] At present, the reasoning and logical abilities of large models are relatively lacking. Mainstream evaluation methods also want to conduct targeted reasoning ability evaluations for this ability, but the current evaluation methods mostly use various common reasoning questions in the human world to verify the reasoning ability of the model, which makes it difficult to fully demonstrate the logical reasoning ability of more models.

[0009] (4) Stability of model preference and performance

[0010] Different models often exhibit different preferences and opinion stability during the interaction process, which cannot be effectively measured by existing methods. Summary of the invention

[0011] In order to solve the problems existing in the prior art, the present invention provides the following technical solutions.

[0012] A first aspect of the present invention provides an evaluation method based on multi-model interaction, comprising the following steps:

[0013] S1. Establish a multi-model interactive evaluation platform, which supports multiple models to participate in the debate;

[0014] S2. Obtaining a debate topic based on the multi-model interactive evaluation platform;

[0015] S3. The multi-model interactive evaluation platform automatically assigns debate positions to multiple models participating in the debate. At the same time, based on the debate topic, multiple models conduct multiple rounds of debates to obtain the performance of each model in the multiple rounds of debates.

[0016] S4. Quantitatively analyze the performance of each model in multiple rounds of debates, and automatically generate a performance report for each model based on preset evaluation indicators.

[0017] Furthermore, obtaining debate topics based on the multi-model interactive evaluation platform includes: generating debate topics based on hot search content on social media captured by the multi-model interactive evaluation platform, a classic debate topic library preset in the multi-model interactive evaluation platform, or extraction by the multi-model interactive evaluation platform from a topic library submitted by the community or users.

[0018] Furthermore, step S3 also includes debugging each model based on preset prompt words and reasoning parameters before multiple models conduct multiple rounds of debates to ensure that each model can conduct effective debate under the acquired debate topic.

[0019] Furthermore, the model interaction evaluation platform is also equipped with a standardized debugging interface for model participants to optimize and debug the performance of the model, as well as to adjust the model's prompt words and reasoning parameters.

[0020] Furthermore, the debugged model will generate the corresponding debate content to be published in this round based on the historical records of the conversation, and the historical records include the background and topic of the debate, the debate content of the model with the same position, and the debate content of the model with the other party's position; among them, the background and debate topic are passed into the model in the form of a system instruction system, the debate content of the model with the same position will be put into the historical records in the form of an artificial intelligence assistant role, and the debate content of the model with the other party's position will be put into the historical records in the form of a user role.

[0021] Furthermore, before step S4, it also includes: the model interaction evaluation platform collects user preference data, professional review data of judges and experts, and / or peer review data of participants participating in the evaluation, and compares and analyzes the performance of each model in multiple rounds of debates based on the user preference data, professional review data of judges and experts, and / or peer review data of participants participating in the evaluation, to form a user preference report.

[0022] Furthermore, the preset evaluation indicators include at least logical reasoning ability, consistency of opinions, opinion mediation and response ability, and opinion mediation and response ability; wherein, the logical reasoning ability is used to evaluate whether the reasoning chain of each model in the debate is coherent and clear; the consistency of opinions is used to detect the changes and consistency of opinions of each model in multiple rounds of debates; the opinion mediation and response ability is used to evaluate the response and mediation ability of each model when facing the opposing debate; the opinion mediation and response ability is used to combine user preference data to evaluate whether the debate results of the model are in line with user preferences.

[0023] A second aspect of the present invention provides an evaluation device based on multi-model interaction, comprising:

[0024] A multi-model interactive evaluation platform establishment module, used to establish a multi-model interactive evaluation platform, wherein the multi-model interactive evaluation platform supports multiple models to participate in the debate;

[0025] The debate topic acquisition module is used to obtain debate topics based on the multi-model interactive evaluation platform;

[0026] The multi-model debate module is used in the multi-model interactive evaluation platform to automatically assign debate positions to multiple models participating in the debate. At the same time, based on the debate topic, multiple models conduct multiple rounds of debates to obtain the performance of each model in multiple rounds of debates;

[0027] The evaluation module is used to quantitatively analyze the performance of each model in multiple rounds of debates and automatically generate a performance report for each model based on preset evaluation indicators.

[0028] The third aspect of the present invention provides a memory storing a plurality of instructions, wherein the instructions are used to implement the evaluation method based on multi-model interaction described in the first aspect of the present invention.

[0029] The fourth aspect of the present invention provides an electronic device, comprising a processor and a memory connected to the processor, the memory storing a plurality of instructions, the instructions being loaded and executed by the processor so that the processor can execute the evaluation method based on multi-model interaction described in the first aspect of the present invention.

[0030] The beneficial effects of the present invention include at least: the evaluation method based on multi-model interaction provided by the present invention can significantly improve the evaluation accuracy of the model in complex scenarios by introducing dynamic evaluation based on multi-model debate, breaking through the limitations of existing evaluation. In addition, through the combination of multiple rounds of debate and user preference data, it can help developers better understand and optimize the model and provide comprehensive performance evaluation feedback. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A schematic diagram of the process of the evaluation method based on multi-model interaction according to the present invention;

[0032] Figure 2 This is a functional structure diagram of the evaluation device based on multi-model interaction according to the present invention;

[0033] Figure 3 A matrix diagram of the number of times debates are won and lost between multiple models according to a preferred embodiment of the present invention;

[0034] Figure 4 A user preference data collection interface provided by the model interaction evaluation platform described in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0036] The method provided by the present invention can be implemented in the following terminal environment, and the terminal may include one or more of the following components: a processor, a memory, and a display screen. The memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the method described in the following embodiment.

[0037] The processor may include one or more processing cores. The processor uses various interfaces and lines to connect various parts of the entire terminal, and executes various functions of the terminal and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory.

[0038] The memory may include random access memory (RAM) or read-only memory (ROM). The memory may be used to store instructions, programs, codes, and commands.

[0039] The display screen is used to display the user interface of each application.

[0040] In addition, those skilled in the art can understand that the structure of the above terminal does not constitute a limitation on the terminal, and the terminal may include more or fewer components, or combine certain components, or arrange the components differently. For example, the terminal also includes components such as a radio frequency circuit, an input unit, a sensor, an audio circuit, and a power supply, which will not be described in detail here.

[0041] Embodiment 1

[0042] like Figure 1 As shown, the embodiment of the present invention provides an evaluation method based on multi-model interaction, comprising the following steps:

[0043] S1. Establish a multi-model interactive evaluation platform, which supports multiple models to participate in the debate;

[0044] S2, obtain debate topics based on the multi-model interactive evaluation platform;

[0045] S3. The multi-model interactive evaluation platform automatically assigns debate positions to multiple models participating in the debate. At the same time, based on the debate topic, multiple models conduct multiple rounds of debates to obtain the performance of each model in multiple rounds of debates.

[0046] S4. Quantitatively analyze the performance of each model in multiple rounds of debates, and automatically generate a performance report for each model based on preset evaluation indicators.

[0047] It should be noted that the acquisition of debate topics based on the multi-model interactive evaluation platform in step S2 includes the hot search content of social media captured by the multi-model interactive evaluation platform, the classic debate topic library preset in the multi-model interactive evaluation platform, or the extraction of the multi-model interactive evaluation platform from the topic library submitted by the community or users to generate debate topics. Specifically, the multi-model interactive evaluation platform can capture the hot search content of social media such as Weibo and Twitter, and automatically generate debate topics related to current hot spots. For example, if the hot search content is "This rainstorm is extremely extreme", then the automatically generated debate topic is "Should mandatory leave under extreme weather conditions be included in legal provisions?" In addition, the multi-model interactive evaluation platform can also include a preset classic debate topic library, or extract debate topics from the topic library submitted by the community or users, so as to ensure the diversity and breadth of debate topics and ensure the coverage of the evaluation. For example, the multi-model interactive evaluation platform extracts "Topic 1: Is the key to success hard work or opportunity?" from the question bank submitted by the community or users. The affirmative argument is "The key is hard work, and hard work is more important than opportunity", and the negative argument is "The key is opportunity, and opportunity is more important than hard work."

[0048] Step S3 also includes debugging each model based on preset prompts and reasoning parameters before multiple models conduct multiple rounds of debates to ensure that each model can conduct effective debates under the acquired debate topics. In a preferred embodiment, multiple models conduct five rounds of debates, and each model speaks alternately. Through multiple rounds of debates, the model interaction evaluation platform can evaluate the performance of each model in terms of opinion expression, logical reasoning, and response to challenges. In a preferred embodiment, there are 7 models participating in the debate, Figure 3 The matrix of the number of wins and losses in the debate between multiple models shows the results of the games of 7 models. Each row shows the number of times a model won in the game against the other 6 models; the second column of the first row shows that the 7th model won 19 times in the game against the 6th model; conversely, the first column of the second row shows that the 7th model lost 9 times in the game against the 6th model. It should be noted that the model interaction evaluation platform allows users or developers to adjust the prompt words and reasoning parameters of the model through a standardized debugging interface, but strictly limits the model to a single call in the reasoning process of each round of debate, that is, the model is prohibited from performing multi-step reasoning (such as the react form). Such restrictions can avoid the intervention of external tools such as search engines and calculators, and ensure the fairness and comparability of the evaluation. At the same time, the standardized debugging interface set by the model interaction evaluation platform is also used by model participants to optimize and debug the performance of the model. After debugging, the model will return to the standard evaluation process for verification. Specifically, the debugged model will generate the corresponding debate content to be published in this round according to the historical records of the conversation, and the historical records include the background and topic of the debate, the debate content of the model with the same position, and the debate content of the model with the other party's position. The background and the topic of the debate are transmitted to the model in the form of system instructions, the debate content of the model with the same position will be put into the historical records in the form of an artificial intelligence assistant role, and the debate content of the model with the other party's position will be put into the historical records in the form of a user role. The conversation process of a conventional large model is simulated by different historical conversation records of the artificial intelligence assistant role and the user role, so that the conventional large language model used for conversation can have multiple rounds of debates with different large models. The present invention completes the process of debate between models without changing the interaction mode of the existing large language model, and the form and rules of the entire debate can be adjusted by adjusting the system instructions.

[0049] In addition, the model interaction evaluation platform also has a standardized API interface (application programming interface) to receive responses from each model and generate corresponding debate content based on the acquired debate topics and assigned debate positions. For example, sending a debate request through the service interface / debate returns the results of the speeches of both sides. The Python code is as follows:

[0050]

[0051] The adjusted model generates debate dialogues through the generate_response method, which can be customized and optimized by users or developers. The final optimization plan will be submitted to the model interaction evaluation platform for unified acceptance.

[0052] The evaluation indicators preset in step S4 at least include logical reasoning ability, viewpoint consistency, viewpoint mediation and response ability, and viewpoint mediation and response ability; wherein,

[0053] Logical reasoning ability is used to assess whether the model's reasoning chain in the debate is coherent and clear;

[0054] Opinion consistency is used to detect the change and consistency of the model's opinions in multiple rounds of debate;

[0055] Opinion mediation and coping ability are used to evaluate the model's coping and mediation abilities when facing opposing debates;

[0056] Opinion mediation and response capabilities are used to combine user preference data to evaluate whether the model's debate results are consistent with user preferences.

[0057] The model interaction evaluation platform automatically generates performance reports for each model based on the above evaluation indicators. The performance report will list in detail the strengths, weaknesses and improvement directions of the model, provide a basis for model optimization for developers and researchers, and help model providers improve their models.

[0058] Before step S4, it also includes: the model interaction evaluation platform collects user preference data, professional review data of judges and experts, and / or peer review data of participants participating in the evaluation, and compares and analyzes the performance of each model in multiple rounds of debates based on the user preference data, professional review data of judges and experts, and / or peer review data of participants participating in the evaluation, to form a user preference report. Specifically, the model interaction evaluation platform publishes debate content related to hot searches on multiple social platforms, encourages users to express their support or opposition to the pros and cons of the arguments through likes, comments, etc., and collects user preference data. The collected user preference data can be used to measure the effectiveness of the model debate. In addition, users can participate in simulated debates and score the performance of each model in multiple rounds of debates through online voting, scoring, etc., providing an additional evaluation dimension for the evaluation of the model interaction evaluation platform. In a preferred embodiment, the user preference data collection interface provided by the model interaction evaluation platform is as follows: Figure 4As shown in the figure, the model interaction evaluation platform can collect user preference data extensively. The model interaction evaluation platform compares and analyzes the collected user preference data with the performance of each model in multiple rounds of debates to form a user preference report. Through these reports, developers can understand the popularity and adaptability of each model in actual application scenarios. The model interaction evaluation platform will automatically adjust the tuning direction of each model based on user feedback to further optimize the performance and reasoning ability of the model.

[0059] Embodiment 2

[0060] like Figure 2 As shown, the present invention also includes a functional module architecture that is completely consistent with the aforementioned method flow, that is, the embodiment of the present invention also provides an evaluation device based on multi-model interaction, including:

[0061] A multi-model interactive evaluation platform establishment module 201 is used to establish a multi-model interactive evaluation platform, wherein the multi-model interactive evaluation platform supports multiple models to participate in the debate;

[0062] A debate topic acquisition module 202, used to acquire a debate topic based on the multi-model interactive evaluation platform;

[0063] The multi-model debate module 203 is used for the multi-model interactive evaluation platform to automatically assign debate positions to multiple models participating in the debate, and based on the debate topic, multiple models conduct multiple rounds of debates to obtain the performance of each model in the multiple rounds of debates;

[0064] The evaluation module 204 is used to quantitatively analyze the performance of each model in multiple rounds of debates and automatically generate a performance report for each model based on preset evaluation indicators.

[0065] The device can be implemented by the evaluation device based on multi-model interaction provided in the above-mentioned embodiment 1. The specific implementation method can be found in the description of embodiment 1, which will not be repeated here.

[0066] The present invention also provides a memory storing a plurality of instructions, wherein the instructions are used to implement an evaluation method based on multi-model interaction as described in Example 1.

[0067] The present invention also provides an electronic device, comprising a processor and a memory connected to the processor, wherein the memory stores a plurality of instructions, and the instructions can be loaded and executed by the processor so that the processor can execute an evaluation method based on multi-model interaction as described in Example 1.

[0068] Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. An evaluation method based on multi-model interaction, characterized in that: The following steps are involved: S1. Establish a multi-model interactive evaluation platform, which supports multiple models to participate in the debate; S2. Obtaining a debate topic based on the multi-model interactive evaluation platform; S3. The multi-model interactive evaluation platform automatically assigns debate positions to multiple models participating in the debate. At the same time, based on the debate topic, multiple models conduct multiple rounds of debates to obtain the performance of each model in the multiple rounds of debates. S4. Quantitatively analyze the performance of each model in multiple rounds of debates, and automatically generate a performance report for each model based on preset evaluation indicators.

2. The evaluation method based on multi-model interaction according to claim 1, characterized in that: The acquiring of debate topics based on the multi-model interactive evaluation platform includes: generating debate topics based on hot search content on social media captured by the multi-model interactive evaluation platform, a classic debate topic library preset in the multi-model interactive evaluation platform, or extraction from a topic library submitted by the multi-model interactive evaluation platform from a community or user.

3. The evaluation method based on multi-model interaction according to claim 1, characterized in that: Step S3 also includes: before multiple models conduct multiple rounds of debate, each model is debugged based on preset prompt words and reasoning parameters to ensure that each model can conduct effective debate under the acquired debate topic.

4. The evaluation method based on multi-model interaction according to claim 1, characterized in that: The model interaction evaluation platform also has a standardized debugging interface for model participants to optimize and debug the performance of the model, as well as adjust the model's prompt words and inference parameters.

5. The evaluation method based on multi-model interaction as claimed in claim 4, characterized in that: The debugged model will generate the corresponding debate content to be published in this round based on the historical records of the conversation. The historical records include the background and topic of the debate, the debate content of the model with the same position, and the debate content of the model with the other party's position: the background and topic of the debate are passed into the model in the form of a system instruction system, the debate content of the model with the same position is put into the historical records in the form of an artificial intelligence assistant role, and the debate content of the model with the other party's position is put into the historical records in the form of a user role.

6. The evaluation method based on multi-model interaction according to claim 1, characterized in that: Before step S4, it also includes: the model interaction evaluation platform collects user preference data, professional review data of judges and experts, and / or peer review data of participants participating in the evaluation, and compares and analyzes the performance of each model in multiple rounds of debates based on the user preference data, professional review data of judges and experts, and / or peer review data of participants participating in the evaluation, to form a user preference report.

7. The evaluation method based on multi-model interaction according to claim 1, characterized in that: The preset evaluation indicators include at least logical reasoning ability, viewpoint consistency, viewpoint mediation and response ability, and viewpoint mediation and response ability; among which, The logical reasoning ability is used to evaluate whether the reasoning chain of each model in the debate is coherent and clear: The viewpoint consistency is used to detect the change and consistency of viewpoints of each model in multiple rounds of debates; The aforementioned opinion mediation and coping abilities are used to evaluate the coping and mediation abilities of each model when facing opposing debates; The opinion mediation and response capabilities are used to combine user preference data to evaluate whether the debate results of the model are consistent with user preferences.

8. An evaluation device based on multi-model interaction, characterized in that: include: A multi-model interactive evaluation platform establishment module, used to establish a multi-model interactive evaluation platform, wherein the multi-model interactive evaluation platform supports multiple models to participate in the debate; The debate topic acquisition module is used to obtain debate topics based on the multi-model interactive evaluation platform; The multi-model debate module is used in the multi-model interactive evaluation platform to automatically assign debate positions to multiple models participating in the debate. At the same time, based on the debate topic, multiple models conduct multiple rounds of debates to obtain the performance of each model in multiple rounds of debates: The evaluation module is used to quantitatively analyze the performance of each model in multiple rounds of debates and automatically generate a performance report for each model based on preset evaluation indicators.

9. A memory, characterized in that: Multiple instructions are stored, and the instructions are used to implement the evaluation method based on multi-model interaction as described in any one of claims 1-7.

10. An electronic device, characterized in that: It includes a processor and a memory connected to the processor, the memory stores multiple instructions, and the instructions can be loaded and executed by the processor so that the processor can execute the evaluation method based on multi-model interaction as described in any one of claims 1-7.

Images