Intelligent agent evaluation method and device based on multiple rounds of dialogues, electronic equipment and medium

Abstract

The invention provides an agent evaluation method and device based on multiple rounds of dialogues, electronic equipment and a medium. The method comprises the following steps: constructing an evaluation sample set only containing an input sample-output sample pair; simulating a user to ask a to-be-evaluated intelligent agent according to an input sample in the evaluation sample set on the basis of the confrontation intelligent agent; determining a feedback result of the current round of dialogue based on the judgment agent according to the questioning question, the corresponding questioning result and an output sample corresponding to the input sample; and determining an evaluation result of the to-be-evaluated intelligent agent according to the current dialogue round and the corresponding feedback result. According to the method, an adversarial agent is introduced to simulate a user to ask a to-be-evaluated agent, and meanwhile, a judgment agent is introduced to judge the reasonability and accuracy of question answering of the to-be-evaluated agent in each round of dialogue, so that the problems that time and labor are consumed when an evaluation data set is constructed by an existing agent evaluation method; according to the method, the problems that the intellectual property is poor and the middle interaction process cannot be effectively simulated are solved, comprehensive evaluation of the intellectual body in the aspects of understanding ability, coherence, reply quality and the like is achieved, and powerful support is provided for optimization of an intelligent dialogue system.

Description

Technical Field

[0001] The present invention relates to the field of real estate transaction technology, and in particular to an intelligent agent evaluation method, device, electronic equipment and medium based on multi-round dialogue. Background Art

[0002] With the rapid development of artificial intelligence (AI), intelligent conversational applications have been widely used in many fields. However, developing intelligent conversational applications suitable for real estate transactions faces many challenges.

[0003] Specifically, real estate transactions involve a large number of professional terms and complex legal and financial processes. Intelligent dialogue applications need to ensure the accuracy and effectiveness of the dialogue content. At the same time, there are significant differences in market policies, transaction habits, etc. in different regions, and intelligent dialogue applications need to adapt to this urban differentiation. In addition, intelligent dialogue applications also need to achieve seamless integration with existing online systems (such as real estate companies' CRM systems, property management systems, and other back-end systems) to ensure data accuracy.

[0004] In order to ensure that intelligent conversation applications can be more intelligent, efficient, and better serve consumers, a large number of sample evaluations are often required before the intelligent conversation application products are launched. However, creating a comprehensive and high-quality evaluation data set is a very time-consuming and costly task. In addition, multiple rounds of conversation processes need to be simulated during sample evaluation, and the simulation process cannot guarantee the coherence, logic, and accuracy of the conversation.

[0005] Therefore, how to solve the problem that the existing intelligent agent evaluation methods are time-consuming and labor-intensive to construct evaluation data sets and cannot effectively simulate the intermediate interaction process is an important issue that needs to be urgently solved in the real estate transaction field. Summary of the invention

[0006] The present invention provides an intelligent agent evaluation method, device, electronic device and medium based on multi-round dialogue, which are used to overcome the defects of existing intelligent agent evaluation methods in that it is time-consuming and labor-intensive to construct evaluation data sets and cannot effectively simulate the intermediate interaction process, and realize a comprehensive evaluation of the intelligent agent in terms of comprehension ability, coherence, response quality, etc., and provide strong support for the optimization of intelligent dialogue systems.

[0007] On the one hand, the present invention provides an agent evaluation method based on multi-round dialogues, including: constructing an evaluation sample set that only contains input sample-output sample pairs; based on an adversarial agent, simulating a user to ask questions to the agent to be evaluated according to the input samples in the evaluation sample set; based on a referee agent, determining the feedback result of the current round of dialogue according to the question asked and the corresponding question result, as well as the output sample corresponding to the input sample; and determining the evaluation result of the agent to be evaluated according to the current dialogue round and the corresponding feedback result.

[0008] Furthermore, the method based on the adversarial intelligent agent simulates the user asking questions to the intelligent agent to be evaluated according to the input samples in the evaluation sample set, including: based on the adversarial intelligent agent, generating questions to be asked to the intelligent agent to be evaluated according to the input samples in the evaluation sample set; based on the intelligent agent to be evaluated, determining the question results according to the questions asked.

[0009] Furthermore, the feedback result of the current round of dialogue is determined based on the referee agent according to the questions asked and the corresponding question results, and the output samples corresponding to the input samples, including: based on the referee agent, the rationality of the agent to be evaluated in the current round of dialogue is determined according to the questions asked and the corresponding question results; based on the referee agent, the accuracy of the agent to be evaluated in the current round of dialogue is determined according to the question results and the output samples; wherein the rationality and accuracy of the agent to be evaluated in the current round of dialogue are both included in the feedback result.

[0010] Furthermore, the evaluation result of the intelligent agent to be evaluated is determined based on the current dialogue round and the corresponding feedback result, including: when the current dialogue round is lower than the set round, guiding the adversarial intelligent agent to correct itself or ask new questions based on the feedback result of the current round of dialogue to start the next round of dialogue; when the current dialogue round is equal to the set round, determining the evaluation result based on the question result of the current round of dialogue.

[0011] Furthermore, when the current dialogue round is lower than the set round, the adversarial agent is guided to correct itself or raise new questions based on the feedback results of the current dialogue round, including: determining that the current dialogue round is lower than the set round, and the feedback results do not meet the preset conditions; guiding the adversarial agent to correct itself based on the feedback results of the current dialogue round, and regenerate questions to ask the to-be-evaluated agent; or, determining that the current dialogue round is lower than the set round, and the feedback results meet the preset conditions; guiding the adversarial agent to raise new questions based on the feedback results of the current dialogue round and the new input samples in the evaluation sample set to start a new round of dialogue.

[0012] Furthermore, when the current dialogue round is equal to the set round, the evaluation result is determined according to the question result of the current dialogue round, including: determining that the current dialogue round is equal to the set round, and the semantic similarity between the question result and the output sample is lower than the set threshold, and determining the evaluation result as the first evaluation result; or, determining that the current dialogue round is equal to the set round, and the semantic similarity between the question result and the output sample reaches the set threshold, and determining the evaluation result as the second evaluation result; wherein the first evaluation result includes a first evaluation score and evaluation process data, the second evaluation result includes a second evaluation score and evaluation process data, and the second evaluation score is higher than the first evaluation score.

[0013] In a second aspect, the present invention also provides an intelligent agent evaluation device based on multi-round dialogues, comprising: an evaluation sample set construction module, used to construct an evaluation sample set containing only input sample-output sample pairs; a simulated questioning module, used to simulate a user asking questions to the intelligent agent to be evaluated based on an adversarial intelligent agent and according to input samples in the evaluation sample set; a dialogue feedback module, used to determine the feedback result of the current round of dialogue based on a referee intelligent agent according to the question asked and the corresponding question result, as well as the output sample corresponding to the input sample; an evaluation result determination module, used to determine the evaluation result of the intelligent agent to be evaluated based on the current dialogue round and the corresponding feedback result.

[0014] In a third aspect, the present invention also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the intelligent agent evaluation method based on multi-round dialogue as described in any one of the above is implemented.

[0015] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described intelligent agent evaluation methods based on multi-round dialogues.

[0016] In a fifth aspect, the present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-described intelligent agent evaluation methods based on multi-round dialogues.

[0017] The agent evaluation method based on multi-round dialogue provided by the present invention constructs an evaluation sample set containing only input sample-output sample pairs, and based on the adversarial agent, simulates the user to ask questions to the agent to be evaluated according to the input samples in the evaluation sample set, and then based on the referee agent, determines the feedback result of the current round of dialogue according to the question and the corresponding question result, as well as the output sample corresponding to the input sample, thereby determining the evaluation result of the agent to be evaluated according to the current dialogue round and the corresponding feedback result. The method introduces an adversarial agent to simulate the user to ask questions to the agent to be evaluated, and introduces a referee agent to determine the rationality and accuracy of the answers to the questions of the agent to be evaluated in each round of dialogue, thereby solving the problem that the existing agent evaluation method is time-consuming and labor-intensive to construct the evaluation data set, and cannot effectively simulate the intermediate interaction process, and realizes a comprehensive evaluation of the agent in terms of comprehension ability, coherence, and response quality, and provides strong support for the optimization of the intelligent dialogue system. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a flowchart of an intelligent agent evaluation method based on multi-round dialogue provided by an embodiment of the present invention.

[0020] Figure 2 It is a schematic diagram of the overall framework of the intelligent agent evaluation method based on multi-round dialogue provided by an embodiment of the present invention.

[0021] Figure 3 It is a structural diagram of an intelligent agent evaluation device based on multi-round dialogue provided by an embodiment of the present invention.

[0022] Figure 4 It is a schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] It should be noted that in order to ensure that intelligent dialogue applications can be more intelligent and efficient and can better serve consumers, a large number of sample evaluations are often required before the intelligent dialogue application products are launched. However, creating a comprehensive and high-quality evaluation data set is a very time-consuming and costly task, and during sample evaluation, multiple rounds of dialogue processes need to be simulated, and the simulation process cannot guarantee the coherence, logic and accuracy of the dialogue.

[0025] In view of this, the present invention proposes an agent evaluation method based on multi-round dialogue, specifically, Figure 1 A flow chart of an intelligent agent evaluation method based on multi-round dialogues provided in an embodiment of the present invention is shown.

[0026] like Figure 1 As shown, the method includes steps S110-S140, and steps S110-S140 and related steps will be described in detail below.

[0027] S110, construct an evaluation sample set containing only input sample-output sample pairs.

[0028] It is easy to understand that before evaluating the intelligent dialogue application to be launched (hereinafter referred to as the intelligent agent to be evaluated), an evaluation sample set needs to be prepared. Unlike the existing intelligent agent evaluation method, which requires the construction of a large number of evaluation data sets including input, output and intermediate process data, the evaluation sample set constructed in this embodiment only needs to include input (input samples) and output (output samples), and does not need to include intermediate process data.

[0029] Specifically, we first need to clarify the application scenarios of the intelligent body to be evaluated, such as applying it to the field of real estate transactions, providing customers with real estate information inquiries, answering questions about the home buying process, etc.

[0030] Subsequently, frequently asked questions (input samples) were collected. The specific collection plan included: (1) collecting the most concerned issues of potential buyers, sellers, and tenants through questionnaires, social media, etc.; (2) using the company's existing "Frequently Asked Questions" documents and customer service records to obtain a list of frequently asked questions; and (3) referring to the latest research reports in the field of real estate transactions to understand current hot issues.

[0031] Next, accurate and reasonable answers (output samples) are provided for the collected common questions. This can be achieved through manual annotation or determined by existing "Frequently Asked Questions" documents, which are not specifically limited here.

[0032] Thus, a test sample set for testing the agent can be constructed, which includes input samples and corresponding output samples. The input samples include but are not limited to basic real estate information, purchase process, legal conditions explanation, and market trend analysis.

[0033] For example, an input sample is "What is the down payment ratio?", and its corresponding input sample is "The down payment ratio usually depends on your loan conditions and local policy regulations. Generally, the minimum down payment ratio is about 20% of the total house price. Please consult our financial advisor for more accurate information."

[0034] It is worth mentioning that when constructing the evaluation sample set, this embodiment only needs to collect input samples and corresponding output samples, without collecting intermediate process data, which can effectively save the time and manpower costs of constructing the evaluation sample set.

[0035] On the basis of constructing the evaluation sample set containing only input sample-output sample pairs in step S110, further, step S120 is performed.

[0036] S120, based on the adversarial agent, according to the input samples in the evaluation sample set, simulates the user asking questions to the evaluation agent.

[0037] It is easy to understand that in order to effectively simulate the intermediate interaction process between the user and the intelligent agent to be evaluated, this embodiment introduces an AI entity for simulating the input behavior of human users, namely, an adversarial agent. The adversarial agent in this embodiment can perceive, make decisions and act in a specific environment, thereby simulating an interaction process similar to that of a real user.

[0038] Specifically, after starting the evaluation task, the current dialogue round is initialized, and the input samples in the evaluation sample set are used as prompt words for the adversarial agent, so that the adversarial agent can simulate real users to ask questions to the agent to be evaluated, thereby achieving the interactive effect of real users and the agent to be evaluated.

[0039] For example, the input of the adversarial agent is "the input sample is A, please ask questions to the agent to be evaluated based on the content A of the input sample", and the output of the adversarial agent is a specific question related to the input sample A.

[0040] It is worth mentioning that, by introducing adversarial agents, this embodiment can not only effectively alleviate the difficulty of preparing a large number of evaluation data sets, but also effectively simulate the intermediate interaction process between real users and the agents to be evaluated.

[0041] In step S120, based on introducing the adversarial agent and asking questions to the agent to be evaluated according to the input samples in the evaluation sample set, further, step S130 is executed.

[0042] S130, based on the referee agent, determine the feedback result of the current round of dialogue according to the question asked and the corresponding question result, and the output sample corresponding to the input sample.

[0043] It is easy to understand that in order to comprehensively evaluate the understanding ability, coherence, response quality, etc. of the intelligent agent to be evaluated, this embodiment also introduces a referee intelligent agent for judging the rationality and accuracy of the dialogue process. The referee intelligent agent can monitor and analyze each link of the dialogue process in real time, including the simulated user input, the response of the intelligent agent to be evaluated, and the intermediate interaction process. Through the built-in knowledge base and algorithm, the referee intelligent agent can judge whether the dialogue process conforms to the preset business process and logical rules, thereby ensuring the coherence and rationality of the dialogue.

[0044] Specifically, after the adversarial agent generates questions based on the input samples in the evaluation sample set, it will pass the questions to the agent to be evaluated. Subsequently, the agent to be evaluated will answer the questions and obtain the corresponding answer, which is the question result.

[0045] Next, the output of the adversarial agent (that is, the input of the agent to be evaluated) and the output of the agent to be evaluated are used as the input of the referee agent. The referee agent determines whether the input and output of the agent to be evaluated comply with the business rules and whether the output of the agent to be evaluated is accurate, thereby obtaining the feedback result of the current round of dialogue.

[0046] Among them, the feedback results of the current round of dialogue include but are not limited to the rationality and accuracy of the answers to the questions answered by the agent to be evaluated in the current round of dialogue.

[0047] In step S130, based on the referee agent, the feedback result of the current round of dialogue is determined according to the question asked and the corresponding question result and the output sample corresponding to the input sample, and then step S140 is further executed.

[0048] S140, determining the evaluation result of the to-be-evaluated intelligent agent according to the current dialogue round and the corresponding feedback result.

[0049] It is easy to understand that the system will record the number of rounds of dialogue in each round. Each round of dialogue starts when the adversarial agent asks a question and ends when the referee agent determines the feedback result. This is a round of dialogue.

[0050] The current dialogue round will be updated at the beginning of each dialogue round. If the current dialogue round has not reached the set round, the adversarial agent questioning step and the referee agent judgment step will be executed in a loop. If and only if the current dialogue round reaches the set round, the dialogue loop will end and the evaluation result of the agent to be evaluated will be output.

[0051] The evaluation results include evaluation scores and evaluation process data. The evaluation process data includes the input and output of the adversarial agent in multiple rounds of dialogue, the input and output of the agent to be evaluated, and the input and output of the referee agent.

[0052] The number of rounds can be set according to actual conditions and is not specifically limited here. For example, in a specific embodiment, the number of rounds is set to 100 times.

[0053] It should be noted that the adversarial agent and referee agent described in this embodiment are AI entities that can accurately judge and answer questions related to the real estate transaction field. Specifically, they can be large language models for learning different tasks, which are not specifically limited here.

[0054] In this embodiment, by constructing an evaluation sample set containing only input sample-output sample pairs, and based on the adversarial agent, simulating the user to ask questions to the subject to be evaluated according to the input samples in the evaluation sample set, and then based on the referee agent, according to the question and the corresponding question result, and the output sample corresponding to the input sample, the feedback result of the current round of dialogue is determined, thereby determining the evaluation result of the subject to be evaluated according to the current dialogue round and the corresponding feedback result. This method introduces an adversarial agent to simulate the user to ask questions to the subject to be evaluated, and introduces a referee agent to determine the rationality and accuracy of the answers to the questions of the subject to be evaluated in each round of dialogue, thereby solving the problem that the existing subject evaluation method is time-consuming and labor-intensive to construct the evaluation data set, and cannot effectively simulate the intermediate interaction process, and realizes a comprehensive evaluation of the subject in terms of comprehension ability, coherence, and response quality, providing strong support for the optimization of the intelligent dialogue system.

[0055] On the basis of the above-mentioned embodiments, the following will further describe in detail the process in which the adversarial intelligent agent simulates a real user to ask questions to the intelligent agent to be evaluated.

[0056] Based on the adversarial agent, according to the input samples in the evaluation sample set, the user is simulated to ask questions to the evaluation agent, including: based on the adversarial agent, according to the input samples in the evaluation sample set, questions to be asked to the evaluation agent are generated; based on the evaluation agent, according to the questions asked, the question results are determined.

[0057] It is easy to understand that, first, select an input sample from the evaluation sample set, and input the selected input sample into the pre-created adversarial agent to get the output question. For example, the input of the adversarial agent is "the input sample is 'which schools are near this community', please generate questions based on the content of the input sample", and the output of the adversarial agent is "which schools are near this community?".

[0058] Then, the questions generated by the adversarial agent are input into the agent to be evaluated, and the questions are answered to obtain the results. For example, the output of the agent to be evaluated is "There are schools such as XX Primary School and YY Middle School around this community."

[0059] Therefore, the adversarial agent can ask questions to the agent to be evaluated based on the input samples in the evaluation sample set, just like the interaction process between real users and the agent to be evaluated. This process does not require intermediate process data, which can save a lot of time and manpower costs.

[0060] On the basis of the above-mentioned embodiments, the judgment process of the referee intelligent agent will be described in detail below.

[0061] Based on the referee agent, the feedback result of the current round of dialogue is determined according to the questions asked and the corresponding question results, as well as the output samples corresponding to the input samples, including: based on the referee agent, the rationality of the agent to be evaluated in the current round of dialogue is determined according to the questions asked and the corresponding question results; based on the referee agent, the accuracy of the agent to be evaluated in the current round of dialogue is determined according to the question results and the output samples; wherein, the rationality and accuracy of the agent to be evaluated in the current round of dialogue are both included in the feedback result.

[0062] It is easy to understand that the referee agent needs to judge the input and output of the evaluated agent, including but not limited to the two dimensions of rationality and accuracy.

[0063] Specifically, the questions generated by the adversarial agent and the question results of the subject to be evaluated are input into the referee agent to determine whether the questions and the question results conform to the business rules of the real estate transaction field, thereby obtaining the rationality of the subject to be evaluated in the current round of dialogue. The rationality here can be a rationality level or a rationality score, which is not specifically limited here.

[0064] It should be noted that when determining the rationality of the current round of dialogue, the business rules related to the questions asked and the results of the questions in the real estate transaction field can be used as prompts for the referee agent to make the rationality of the output more accurate.

[0065] At the same time, the referee agent compares the question result with the output sample (corresponding to the current input sample) to determine the difference / similarity between the question result and the output sample, thereby obtaining the accuracy of the agent to be evaluated in the current round of dialogue. The accuracy here can be an accuracy level or an accuracy score, which is not specifically limited here.

[0066] Thus, we can obtain the judgment result of the referee intelligent agent on the answer of the evaluation intelligent agent, that is, the feedback result.

[0067] It is worth mentioning that the rationality and accuracy of the agent to be evaluated in the current round of dialogue can serve as a guide for the next round of dialogue to ensure the coherence, logic and accuracy of the dialogue of the agent to be evaluated. For example, if the rationality of the current round of dialogue is not enough, then a prompt can be given, such as "the question result is unreasonable because XXX", so that the adversarial agent can correct itself according to the feedback results and prompts and regenerate questions.

[0068] On the basis of the above-mentioned embodiments, the process of determining the evaluation result of the intelligent agent to be evaluated will be described in detail below.

[0069] The evaluation result of the agent to be evaluated is determined according to the current dialogue round and the corresponding feedback results, including: when the current dialogue round is lower than the set round, according to the feedback results of the current round of dialogue, guiding the adversarial agent to correct itself or ask new questions to start the next round of dialogue; when the current dialogue round is equal to the set round, the evaluation result is determined according to the question results of the current round of dialogue.

[0070] It is easy to understand that at the end of each round of dialogue, the current dialogue round will be compared with the set round to determine whether the current dialogue round has reached the set round. If the current dialogue round has not reached the set round, the dialogue process will continue in a loop; if the current dialogue round has reached the set round, the dialogue process will end in a loop and the final evaluation result will be output.

[0071] Specifically, when the current dialogue round is lower than the set round, the adversarial agent is guided to correct itself or ask new questions based on the feedback results of the current round of dialogue to start the next round of dialogue.

[0072] Specifically, it is first determined whether the current round of dialogue is indeed lower than the set round, and then it is further determined whether the feedback result of the current round of dialogue meets the preset conditions, that is, whether the rationality of the current round of dialogue reaches the preset rationality level or the preset rationality threshold, and whether the accuracy of the current round of dialogue reaches the preset accuracy level or the preset accuracy threshold.

[0073] If any of the rationality and accuracy of the current round of dialogue does not meet the corresponding preset conditions, it means that the feedback result of the current round of dialogue does not meet the preset conditions. In this case, according to the feedback result of the current round of dialogue, guide the adversarial agent to correct itself and regenerate questions to ask the subject to be evaluated. In this process, the referee agent will give unreasonable or inaccurate reasons, guide the adversarial agent to adjust the questions, and re-ask the subject to be evaluated.

[0074] If the rationality and accuracy of the current round of dialogue meet the corresponding preset conditions, it means that the feedback results of the current round of dialogue meet the preset conditions. In this case, according to the feedback results of the current round of dialogue and the new input samples in the evaluation sample set, the adversarial agent is guided to ask new questions to start a new round of dialogue. In this process, the adversarial agent re-selects an input sample from the evaluation sample set to generate new questions and start a new round of dialogue.

[0075] When the current dialogue round is equal to the set round, the evaluation result is determined according to the question results of the current dialogue round.

[0076] Specifically, first, it is determined whether the current dialogue turn is indeed equal to the set turn, and then it is further determined whether the semantic similarity between the question result and the output sample is lower than the set threshold.

[0077] If the semantic similarity between the question result of the current round of dialogue and the output sample is lower than the set threshold, the evaluation of the intelligent agent to be evaluated is judged to be unqualified, and the corresponding first evaluation result includes a first evaluation score and evaluation process data.

[0078] If the semantic similarity between the question result of the current round of dialogue and the output sample is greater than or equal to the set threshold, the evaluation of the intelligent agent to be evaluated is determined to be qualified, and the corresponding second evaluation result includes a second evaluation score and evaluation process data.

[0079] The threshold value may be set according to actual needs and is not specifically limited herein. For example, in a specific embodiment, the threshold value is set to 97%.

[0080] The first evaluation score is higher than the second evaluation score. The higher the evaluation score, the better the dialogue effect of the agent to be evaluated. The first evaluation score and the second evaluation score can be replaced by corresponding evaluation levels. For example, the first evaluation score corresponds to unqualified and the second evaluation score corresponds to qualified.

[0081] In this embodiment, when the current dialogue round is lower than the set round, the adversarial agent is guided to correct itself or ask new questions according to the feedback results of the current dialogue round to start the next dialogue round. When the current dialogue round is equal to the set round, the evaluation result is determined according to the question results of the current dialogue round. This method introduces adversarial agents to simulate users to ask questions to the subject to be evaluated, and introduces referee agents to judge the rationality and accuracy of the answers of the subject to be evaluated in each dialogue round. It solves the problem that the existing agent evaluation method is time-consuming and labor-intensive to construct the evaluation data set, and cannot effectively simulate the intermediate interaction process, and realizes the comprehensive evaluation of the agent in terms of comprehension ability, coherence, and response quality, providing strong support for the optimization of the intelligent dialogue system.

[0082] In some other embodiments, Figure 2 The figure shows the overall framework of the intelligent agent evaluation method based on multi-round dialogue provided by an embodiment of the present invention.

[0083] like Figure 2 As shown, the interaction starts with the adversarial agent. Specifically, first, the adversarial agent asks questions to the evaluation agent based on the input samples in the evaluation sample set.

[0084] Then, the agent to be evaluated answers the questions asked by the adversarial agent and returns the answers, i.e., the question results.

[0085] Next, the referee agent judges the rationality and accuracy of the questions and the corresponding results based on business rules, and gives corresponding feedback, namely the feedback results.

[0086] Subsequently, based on the evaluation sample set and feedback results, the adversarial agent asks questions in a loop, repeating the above process of answering questions, judging questions and answers.

[0087] Until the number of dialogue turns in the loop reaches the set number of turns, in which case the loop ends and the accuracy of the question result of the current dialogue round is judged, that is, the semantic similarity between the question result of the current dialogue round and the corresponding output sample.

[0088] Finally, the final evaluation result is determined based on the judgment result of semantic similarity, and the evaluation ends.

[0089] Corresponding to the intelligent agent evaluation method based on multi-round dialogue described in the above embodiments, the present invention also proposes an intelligent agent evaluation device based on multi-round dialogue.

[0090] Specifically, Figure 3 A schematic diagram of the structure of an intelligent agent evaluation device based on multi-round dialogue provided by an embodiment of the present invention is shown.

[0091] like Figure 3 As shown, the device includes: an evaluation sample set construction module 310, which is used to construct an evaluation sample set containing only input sample-output sample pairs; a simulated questioning module 320, which is used to simulate a user asking questions to the subject to be evaluated based on an adversarial agent and according to input samples in the evaluation sample set; a dialogue feedback module 330, which is used to determine the feedback result of the current round of dialogue based on the referee agent according to the question asked and the corresponding question result, as well as the output sample corresponding to the input sample; an evaluation result determination module 340, which is used to determine the evaluation result of the subject to be evaluated based on the current dialogue round and the corresponding feedback result.

[0092] In this embodiment, the evaluation sample set construction module 310 constructs an evaluation sample set containing only input sample-output sample pairs, the simulation question module 320 is based on the adversarial agent, and simulates the user asking questions to the evaluation agent according to the input samples in the evaluation sample set, and then the dialogue feedback module 330 is based on the referee agent, and determines the feedback result of the current round of dialogue according to the question and the corresponding question result, as well as the output sample corresponding to the input sample, so that the evaluation result determination module 340 determines the evaluation result of the evaluation agent according to the current dialogue round and the corresponding feedback result. The device introduces an adversarial agent to simulate the user asking questions to the evaluation agent, and introduces a referee agent to determine the rationality and accuracy of the answers of the evaluation agent in each round of dialogue, thereby solving the problem that the existing agent evaluation method is time-consuming and labor-intensive to construct the evaluation data set, and cannot effectively simulate the intermediate interaction process, and realizes a comprehensive evaluation of the agent in terms of comprehension ability, coherence, and response quality, and provides strong support for the optimization of the intelligent dialogue system.

[0093] It should be noted that the intelligent agent evaluation device based on multi-round dialogue provided in the embodiment of the present invention and the intelligent agent evaluation method based on multi-round dialogue described in the above embodiments can correspond to each other and will not be described in detail here.

[0094] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the agent evaluation method based on multiple rounds of dialogue, the method comprising: constructing an evaluation sample set containing only input sample-output sample pairs; based on the adversarial agent, simulating the user to ask questions to the agent to be evaluated according to the input samples in the evaluation sample set; based on the referee agent, determining the feedback result of the current round of dialogue according to the question and the corresponding question result, and the output sample corresponding to the input sample; and determining the evaluation result of the agent to be evaluated according to the current dialogue round and the corresponding feedback result.

[0095] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0096] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the agent evaluation method based on multi-round dialogues provided by the above methods, which method includes: constructing an evaluation sample set that only contains input sample-output sample pairs; based on an adversarial agent, simulating a user to ask questions to the agent to be evaluated according to the input samples in the evaluation sample set; based on a referee agent, determining the feedback result of the current round of dialogue according to the question asked and the corresponding question result, as well as the output sample corresponding to the input sample; and determining the evaluation result of the agent to be evaluated according to the current dialogue round and the corresponding feedback result.

[0097] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the agent evaluation method based on multi-round dialogues provided by the above-mentioned methods, the method comprising: constructing an evaluation sample set containing only input sample-output sample pairs; based on an adversarial agent, simulating a user to ask questions to the agent to be evaluated according to the input samples in the evaluation sample set; based on a referee agent, determining the feedback result of the current round of dialogue according to the question asked and the corresponding question result, as well as the output sample corresponding to the input sample; and determining the evaluation result of the agent to be evaluated according to the current dialogue round and the corresponding feedback result.

[0098] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0099] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for evaluating an intelligent agent based on multi-round dialogues, characterized in that: include: Construct an evaluation sample set containing only input sample-output sample pairs; Based on the adversarial agent, the user is simulated to ask questions to the evaluation agent according to the input samples in the evaluation sample set; Based on the referee agent, determine the feedback result of the current round of dialogue according to the question asked and the corresponding question result, and the output sample corresponding to the input sample; The evaluation result of the agent to be evaluated is determined according to the current dialogue round and the corresponding feedback result.

2. The agent evaluation method based on multi-round dialogue according to claim 1 is characterized in that: The method based on the adversarial agent, based on the input samples in the evaluation sample set, simulates the user to ask questions to the evaluation agent, including: Based on the adversarial agent, questions to be asked to the agent to be evaluated are generated according to the input samples in the evaluation sample set; Based on the intelligent agent to be evaluated, the question result is determined according to the question asked.

3. The agent evaluation method based on multi-round dialogue according to claim 1 is characterized in that: The judging agent determines the feedback result of the current round of dialogue according to the question and the corresponding question result, and the output sample corresponding to the input sample, including: Based on the referee agent, determine the rationality of the agent to be evaluated in the current round of dialogue according to the questions asked and the corresponding question results; Based on the referee agent, determine the accuracy of the agent to be evaluated in the current round of dialogue according to the question result and the output sample; Among them, the rationality and accuracy of the intelligent agent to be evaluated in the current round of dialogue are included in the feedback results.

4. The agent evaluation method based on multi-round dialogue according to claim 1 is characterized in that: Determining the evaluation result of the to-be-evaluated agent according to the current dialogue round and the corresponding feedback result includes: When the current round of dialogue is lower than the set round, the adversarial agent is guided to correct itself or ask new questions based on the feedback of the current round of dialogue to start the next round of dialogue; When the current dialogue round is equal to the set round, the evaluation result is determined according to the question results of the current dialogue round.

5. The agent evaluation method based on multi-round dialogue according to claim 4 is characterized in that: When the current dialogue round is lower than the set round, the adversarial agent is guided to correct itself or raise new questions based on the feedback results of the current dialogue round, including: Determine that the current dialogue round is lower than the set round and the feedback result does not meet the preset conditions; guide the adversarial agent to correct itself based on the feedback result of the current dialogue round and regenerate questions to ask the agent to be evaluated; or, Determine whether the current round of dialogue is lower than the set round and the feedback result meets the preset conditions; based on the feedback result of the current round of dialogue and the new input samples in the evaluation sample set, guide the adversarial agent to ask new questions to start a new round of dialogue.

6. The agent evaluation method based on multi-round dialogue according to claim 3 is characterized in that: When the current dialogue round is equal to the set round, the evaluation result is determined according to the question result of the current dialogue round, including: Determine that the current dialogue turn is equal to the set turn, and the semantic similarity between the question result and the output sample is lower than the set threshold, and determine the evaluation result as the first evaluation result; or, Determine that the current dialogue turn is equal to the set turn, and the semantic similarity between the question result and the output sample reaches a set threshold, and determine the evaluation result as the second evaluation result; Among them, the first evaluation result includes a first evaluation score and evaluation process data, and the second evaluation result includes a second evaluation score and evaluation process data, and the second evaluation score is higher than the first evaluation score.

7. An agent evaluation device based on multi-round dialogue, characterized in that: include: An evaluation sample set construction module is used to construct an evaluation sample set containing only input sample-output sample pairs; The simulated questioning module is used to simulate the user asking questions to the evaluation agent based on the adversarial agent and the input samples in the evaluation sample set; A dialogue feedback module is used to determine the feedback result of the current round of dialogue based on the referee intelligent agent, according to the questions asked and the corresponding question results, and the output samples corresponding to the input samples; The evaluation result determination module is used to determine the evaluation result of the to-be-evaluated intelligent agent according to the current dialogue round and the corresponding feedback result.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the agent evaluation method based on multi-round dialogue is implemented as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for evaluating an intelligent agent based on multi-round dialogue as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for evaluating an intelligent agent based on multi-round dialogue as described in any one of claims 1 to 6 is implemented.

Images