Conversational robot evaluation method and device, storage medium, and computer device

Abstract

The application discloses an evaluation method and device of a dialogue robot, a storage medium and a computer device, belongs to the technical field of man-machine dialogue, and mainly solves the problems that the evaluation of a robot reply lacks objective consistency, is low in efficiency, and influences online effect by adopting an artificial evaluation mode, and comprises the following steps: obtaining dialogue context and a robot reply of a dialogue robot to be evaluated, and forming a plurality of to-be-evaluated samples; performing evaluation processing on the to-be-evaluated samples based on a preset prompt word and an external evaluation model, obtaining a first evaluation value; performing evaluation processing on the to-be-evaluated samples based on a pre-trained dialogue evaluation reward model, and obtaining a second evaluation value; performing score fusion on the first evaluation value and the second evaluation value, obtaining a comprehensive evaluation value corresponding to each to-be-evaluated sample; determining an evaluation result of each to-be-evaluated sample based on the comprehensive evaluation value, and performing labeling processing on the robot reply based on the evaluation result.

Description

Technical Field

[0001] This invention relates to the field of human-computer dialogue technology, and in particular to an evaluation method and apparatus for dialogue robots, a storage medium, and a computer device. Background Technology

[0002] With the continuous maturation of deep learning and natural language processing technologies, the field of human-computer dialogue has achieved remarkable development in recent years, and the quality and effectiveness of human-computer dialogue systems have been greatly improved. Currently, human-computer dialogue systems are widely used in various fields, including customer service, intelligent assistants, smart homes, online education, financial insurance, and healthcare. The development of human-computer dialogue has also promoted the popularization of products such as intelligent robots, virtual assistants, and smart speakers. To provide users with a better interactive experience and service, products need to be evaluated before being launched on the market, making objective judgments on the rationality, relevance, and accuracy of the robot's responses.

[0003] Currently, chatbots are primarily evaluated through subjective human assessment. Before deployment, offline performance evaluations are conducted, typically using multi-turn dialogue data as a test dataset. Based on the user's context, the chatbot responds with appropriate phrases, and human annotators, using their business experience, label the chatbot's responses as "reasonable" or "unreasonable." However, relying on human annotators is susceptible to subjective bias or misunderstanding of business knowledge, making it difficult to ensure fairness, objectivity, and consistency in the evaluation. Furthermore, due to high labor costs and low efficiency, large-scale evaluations can easily lead to significant discrepancies between offline and online results. Summary of the Invention

[0004] In view of this, the present invention provides an evaluation method and apparatus, storage medium and computer equipment for chatbots, the main purpose of which is to solve the problems that the evaluation of chatbot responses by manual evaluation lacks objectivity and consistency, and is inefficient and affects the effectiveness of online implementation.

[0005] According to one aspect of the present invention, an evaluation method for a chatbot is provided, comprising:

[0006] Obtain the dialogue context and robot responses of the chatbot to be evaluated, forming multiple samples to be evaluated;

[0007] The sample to be evaluated is evaluated based on preset prompts and an external evaluation model to obtain a first evaluation value; and the sample to be evaluated is evaluated based on a pre-trained dialogue evaluation and reward model to obtain a second evaluation value.

[0008] The scores of the first evaluation value and the second evaluation value are fused to obtain a comprehensive evaluation value corresponding to each sample to be evaluated.

[0009] The evaluation results for each of the samples to be evaluated are determined based on the comprehensive evaluation value, and the robot's response is annotated based on the evaluation results.

[0010] Furthermore, before evaluating the sample to be evaluated using preset prompts and an external evaluation model, the method further includes:

[0011] The prompts are set according to the assessment requirements. The prompts include role declaration, specific tasks, task requirements and output format; the output format includes the output format of the first assessment value.

[0012] Furthermore, the evaluation process based on preset prompts and an external evaluation model to obtain the first evaluation value includes:

[0013] The preset prompt words are concatenated with the sample to be evaluated to obtain the concatenated text.

[0014] The concatenated text is transmitted to an external evaluation model via an external interface, so that the external evaluation model can evaluate the concatenated text and obtain a first evaluation value.

[0015] Receive the first evaluation value returned by the external evaluation model.

[0016] Furthermore, the step of concatenating the preset prompt words with the sample to be evaluated to obtain the concatenated text includes:

[0017] In the samples to be evaluated, the dialogue context and the role corresponding to the robot's response are respectively marked;

[0018] Add a semantic end marker to the end of each dialogue in the sample to be evaluated after the roles are identified;

[0019] The sample to be evaluated, after having the semantic end marker added, is appended to the preset prompt word to obtain the concatenated text.

[0020] Furthermore, before the pre-trained dialogue evaluation reward model performs evaluation processing on the sample to be evaluated, the method further includes:

[0021] Obtain historical business dialogue text data, and construct question-and-answer sample pairs based on the historical business dialogue text data, wherein the question-and-answer sample pairs include positive samples and negative samples;

[0022] A reward model is constructed using a pairwise loss function, and the reward model is trained based on the positive and negative samples to obtain the dialogue evaluation reward model.

[0023] Furthermore, the step of fusing the scores of the first evaluation value and the second evaluation value to obtain a comprehensive evaluation value corresponding to each sample to be evaluated includes:

[0024] Calculate the product of the first evaluation value and the second evaluation value to obtain the comprehensive evaluation value corresponding to each sample to be evaluated; or,

[0025] The first evaluation value and the second evaluation value are standardized, and the standardized first evaluation value and the second evaluation value are weighted and summed to obtain the comprehensive evaluation value corresponding to each sample to be evaluated.

[0026] Furthermore, the step of determining the evaluation result of each of the samples to be evaluated based on the comprehensive evaluation value, and performing annotation processing on the robot's response based on the evaluation result, includes:

[0027] Compare the comprehensive evaluation value with a reasonable evaluation threshold;

[0028] If the comprehensive evaluation value is greater than or equal to the reasonable evaluation threshold, the evaluation result will be determined as reasonable, and reasonable will be marked on the corresponding robot response;

[0029] If the comprehensive evaluation value is less than the reasonable evaluation threshold, the evaluation result will be determined as unreasonable, and unreasonableness will be marked on the corresponding robot response.

[0030] According to another aspect of the present invention, an evaluation apparatus for a chatbot is provided, comprising:

[0031] The sample acquisition module is used to acquire the dialogue context and robot responses of the chatbot to be evaluated, forming multiple samples to be evaluated.

[0032] The model evaluation module is used to evaluate the sample to be evaluated based on preset prompts and an external evaluation model to obtain a first evaluation value; and to evaluate the sample to be evaluated based on a pre-trained dialogue evaluation and reward model to obtain a second evaluation value.

[0033] The comprehensive evaluation module is used to fuse the scores of the first evaluation value and the second evaluation value to obtain a comprehensive evaluation value corresponding to each sample to be evaluated.

[0034] The result determination module is used to determine the evaluation result of each of the samples to be evaluated based on the comprehensive evaluation value, and to perform annotation processing on the robot's response based on the evaluation result.

[0035] Furthermore, the device also includes a setting module for:

[0036] The prompts are set according to the assessment requirements. The prompts include role declaration, specific tasks, task requirements and output format; the output format includes the output format of the first assessment value.

[0037] Furthermore, the model evaluation module includes a first evaluation unit, used for:

[0038] The preset prompt words are concatenated with the sample to be evaluated to obtain the concatenated text.

[0039] The concatenated text is transmitted to an external evaluation model via an external interface, so that the external evaluation model can evaluate the concatenated text and obtain a first evaluation value.

[0040] Receive the first evaluation value returned by the external evaluation model.

[0041] Furthermore, the first evaluation unit is also used for:

[0042] In the samples to be evaluated, the dialogue context and the role corresponding to the robot's response are respectively marked;

[0043] Add a semantic end marker to the end of each dialogue in the sample to be evaluated after the roles are identified;

[0044] The sample to be evaluated, after having the semantic end marker added, is appended to the preset prompt word to obtain the concatenated text.

[0045] Furthermore, the model evaluation module also includes a second evaluation unit, used for:

[0046] Obtain historical business dialogue text data, and construct question-and-answer sample pairs based on the historical business dialogue text data, wherein the question-and-answer sample pairs include positive samples and negative samples;

[0047] A reward model is constructed using a pairwise loss function, and the reward model is trained based on the positive and negative samples to obtain the dialogue evaluation reward model.

[0048] Furthermore, the comprehensive evaluation module is also used for:

[0049] Calculate the product of the first evaluation value and the second evaluation value to obtain the comprehensive evaluation value corresponding to each sample to be evaluated; or,

[0050] The first evaluation value and the second evaluation value are standardized, and the standardized first evaluation value and the second evaluation value are weighted and summed to obtain the comprehensive evaluation value corresponding to each sample to be evaluated.

[0051] Furthermore, the result determination module is also used for:

[0052] Compare the comprehensive evaluation value with a reasonable evaluation threshold;

[0053] If the comprehensive evaluation value is greater than or equal to the reasonable evaluation threshold, the evaluation result will be determined as reasonable, and reasonable will be marked on the corresponding robot response;

[0054] If the comprehensive evaluation value is less than the reasonable evaluation threshold, the evaluation result will be determined as unreasonable, and unreasonableness will be marked on the corresponding robot response.

[0055] According to another aspect of the present invention, a storage medium is provided, wherein at least one executable instruction is stored therein, the executable instruction causing a processor to perform an operation corresponding to the evaluation method of the above-described chatbot.

[0056] According to another aspect of the present invention, a computer device is provided, including a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus;

[0057] The memory is used to store at least one executable instruction that causes the processor to perform an operation corresponding to the evaluation method of the dialogue robot described above.

[0058] By employing the above-described technical solutions, the technical solutions provided by the embodiments of the present invention have at least the following advantages:

[0059] This invention provides a method, apparatus, storage medium, and computer device for evaluating chatbots. Compared with existing technologies, this invention obtains the dialogue context and robot responses of the chatbot to be evaluated, forming multiple evaluation samples. The evaluation samples are then processed based on preset prompts and an external evaluation model to obtain a first evaluation value. A second evaluation value is obtained by further processing the samples based on a pre-trained dialogue evaluation reward model. The first and second evaluation values ​​are then fused to obtain a comprehensive evaluation value corresponding to each sample. The evaluation result for each sample is determined based on the comprehensive evaluation value, and the robot responses are annotated based on the evaluation results. This achieves automatic evaluation of chatbots, characterized by low evaluation cost and high efficiency. Furthermore, consistency is maintained before and after evaluation, eliminating subjective judgment by annotators and making the evaluation more fair and objective. In addition, this invention uses an external evaluation model and a pre-trained dialogue evaluation reward model simultaneously for evaluation, which not only improves the accuracy of the evaluation but also makes the evaluation conclusions more professional.

[0060] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0061] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0062] Figure 1 A flowchart illustrating an evaluation method for a chatbot provided by an embodiment of the present invention is shown.

[0063] Figure 2 A flowchart illustrating another method for evaluating a chatbot provided in an embodiment of the present invention is shown.

[0064] Figure 3 A flowchart illustrating another method for evaluating a chatbot provided in an embodiment of the present invention is shown.

[0065] Figure 4 A flowchart illustrating another method for evaluating a chatbot provided in an embodiment of the present invention is shown.

[0066] Figure 5A schematic diagram of the structure of an evaluation device for a conversational robot provided in an embodiment of the present invention is shown;

[0067] Figure 6 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention is shown. Detailed Implementation

[0068] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0069] This invention provides an evaluation method for chatbots, such as... Figure 1 As shown, the method includes:

[0070] 101. Obtain the dialogue context and robot responses of the chatbot to be evaluated, forming multiple samples to be evaluated;

[0071] In this embodiment of the invention, the current execution terminal acquires the dialogue context and robot responses of the chatbot to be evaluated, forming multiple samples to be evaluated. Each sample to be evaluated represents a complete set of dialogue content, typically in a question-and-answer format. For example, in an insurance and financial planning scenario, if a user asks, "Is this insurance free?", and the robot replies, "This insurance requires payment; it's calculated based on your age," this question-and-answer dialogue constitutes one sample to be evaluated. Furthermore, samples to be evaluated can also be from scenarios such as intelligent assistants, smart homes, online education, and healthcare; this embodiment of the invention does not impose specific limitations.

[0072] 102. The sample to be evaluated is evaluated based on preset prompt words and an external evaluation model to obtain a first evaluation value; and the sample to be evaluated is evaluated based on a pre-trained dialogue evaluation and reward model to obtain a second evaluation value.

[0073] In this embodiment of the invention, the current execution end evaluates the sample to be evaluated based on preset prompts and an external evaluation model to obtain a first evaluation value. The preset prompts are a type of natural language used in Natural Language Processing (NLP) to guide or stimulate an artificial intelligence model to complete a specific task; they are also known as prompt instructions. The external evaluation model represents a type of model that can better understand the input intent based on the preset prompts and continuously improve and optimize by answering user questions through supervised learning. This model implements its functions through an external interface, such as calling OpenAI's ChatGPT interface to call the GPT4 model to evaluate the sample to be evaluated. This embodiment of the invention does not specifically limit the model's functionality.

[0074] In this embodiment of the invention, the current execution end further evaluates the sample to be evaluated based on a pre-trained dialogue evaluation model to obtain a second evaluation value. The dialogue evaluation model is used to represent a series of models in the field of reinforcement learning for processing natural language. After processing by the dialogue evaluation model, the output is a probability value; the higher the score, the more valuable and reasonable the robot's response.

[0075] 103. The scores of the first evaluation value and the second evaluation value are fused to obtain the comprehensive evaluation value corresponding to each sample to be evaluated;

[0076] In this embodiment of the invention, the current execution end performs score fusion on the first evaluation value and the second evaluation value to obtain a comprehensive evaluation value corresponding to each sample to be evaluated. The score fusion process characterizes the entire process of combining evaluation values ​​obtained from different channels, effectively preventing bias in evaluation results from a single channel and improving the accuracy of the chatbot evaluation.

[0077] 104. Determine the evaluation result of each of the samples to be evaluated based on the comprehensive evaluation value, and perform annotation processing on the robot's response based on the evaluation result.

[0078] In this embodiment of the invention, the current execution end determines the evaluation result of each sample to be evaluated based on a comprehensive evaluation value. The evaluation result includes "reasonable" and "unreasonable," and this embodiment does not impose specific limitations. After obtaining the evaluation result, the current execution end performs annotation processing on the robot's response based on the evaluation result. For example, it may directly annotate "reasonable" on robot responses with an evaluation result of "reasonable" and "unreasonable" on robot responses with an evaluation result of "unreasonable," or it may use numbers, symbols, letters, etc. for annotation processing. This embodiment does not impose specific limitations.

[0079] Furthermore, as a refinement and extension of the specific implementation of the above embodiments, in order to guide or stimulate the external evaluation model to complete the evaluation processing task, another evaluation method for a chatbot is provided. Before the steps of evaluating the sample to be evaluated based on preset prompt words and the external evaluation model are performed, the method further includes: setting prompt words according to the evaluation requirements.

[0080] In this embodiment of the invention, the current execution end sets prompt words according to the evaluation requirements. These prompt words include role declaration, specific tasks, task requirements, and output format; the output format includes the output format of the first evaluation value. It should be noted that, to better understand the evaluation logic of the external evaluation model, the external evaluation model can be required to provide an explanation of its evaluation when setting the prompt words. In an insurance and financial planning scenario, the prompt could be set as follows: "Please act as a fair referee and evaluate the quality of the agent's answers to customer questions. Your evaluation should consider factors such as the reasonableness, relevance, and accuracy of the answers. Your evaluation should focus on the agent's answer to the customer's last question. Please begin by providing a brief explanation. Please evaluate as objectively as possible. After providing the explanation, you need to give a rating in the following format, strictly following the rating range of 1 to 10: \"[[rating]]\", for example: \"Rating: [[6]]\". In the above prompt, the role description corresponds to referee, the specific task corresponds to evaluating the quality of the agent's answers to customer questions, the task requirements correspond to considering factors such as the reasonableness, relevance, and accuracy of the answers, and focusing on the agent's answer to the customer's last question, and the output format of the first evaluation value corresponds to "Rating: [[rating]]", etc. This embodiment of the invention does not make specific limitations.

[0081] Furthermore, as a refinement and extension of the specific implementation of the above embodiments, in order to reduce the computational load and development cost of the current execution end without affecting the accuracy of the evaluation, another evaluation method for chatbots is provided, such as... Figure 2 As shown, the steps involve evaluating the sample to be evaluated based on preset prompts and an external evaluation model to obtain a first evaluation value, including:

[0082] 201. Perform text concatenation processing between the preset prompt words and the sample to be evaluated to obtain the concatenated text;

[0083] In this embodiment of the invention, the current execution end performs text concatenation processing on the Yisuhe prompt word and the sample to be evaluated. First, the roles corresponding to the dialogue context and the robot's response are marked in the sample to be evaluated; for example, in the insurance and financial management scenario, the roles of user and agent are marked in the dialogue context and robot response; in the health and medical scenario, the roles of patient and intelligent doctor are marked in the dialogue context and robot response, etc. This embodiment of the invention does not make specific limitations. The current execution end adds semantic end markers to the end of each dialogue in the sample to be evaluated after marking the roles for semantic segmentation, such as using the semantic end marker "\n" for semantic segmentation, making the concatenated text after semantic segmentation easier for the model to understand and improving the accuracy of model evaluation. Finally, the sample to be evaluated with the added semantic end markers is concatenated to the preset prompt word to obtain the concatenated text.

[0084] 202. The concatenated text is transmitted to an external evaluation model via an external interface, so that the external evaluation model can evaluate the concatenated text and obtain a first evaluation value.

[0085] 203. Receive the first evaluation value returned by the external evaluation model.

[0086] In this embodiment of the invention, the current execution terminal obtains an external interface corresponding to the external evaluation model and transmits the concatenated text to the external evaluation model based on the external interface. For example, it may obtain OpenAI's ChatGPT interface to transmit the concatenated text to the GPT4 model for evaluation processing, etc. This embodiment of the invention does not specifically limit the method. Upon receiving the concatenated text, the external evaluation model initiates evaluation processing of the concatenated text, obtains a first evaluation value, and returns the first evaluation value to the current execution terminal. The current execution terminal receives the first evaluation value returned by the external evaluation model, completing one evaluation process for the sample to be evaluated.

[0087] Furthermore, as a refinement and extension of the specific implementation methods described above, and in order to improve the professionalism and accuracy of the evaluation in various business scenarios, another evaluation method for chatbots is provided, such as... Figure 3 As shown, before evaluating the sample to be evaluated based on a pre-trained dialogue evaluation reward model, the method further includes:

[0088] 301. Obtain historical business dialogue text data, and construct question-and-answer sample pairs based on the historical business dialogue text data, wherein the question-and-answer sample pairs include positive samples and negative samples;

[0089] In this embodiment of the invention, the current execution terminal acquires historical business dialogue text data. This historical dialogue text data includes business scenarios such as customer service, intelligent assistants, smart homes, online education, financial insurance, and healthcare. Historical dialogue text data from relevant scenarios can be reasonably selected according to current business needs; this embodiment of the invention does not impose specific limitations. After acquiring the historical business dialogue text data, the current execution terminal constructs question-and-answer sample pairs based on the historical dialogue text data. These question-and-answer sample pairs include positive samples and negative samples. Positive samples represent genuine responses in the current business scenario, while negative samples are non-genuine responses randomly obtained from historical dialogues; this embodiment of the invention does not impose specific limitations.

[0090] 302. A reward model is constructed using the Pairwise loss function, and the reward model is trained based on the positive samples and the negative samples to obtain the dialogue evaluation reward model.

[0091] In this embodiment of the invention, the current execution end uses a pairwise loss function to construct a reward model, and uses the aforementioned positive and negative samples as training samples for the reward model. Then, the reward model is trained using these training samples to obtain a dialogue evaluation reward model corresponding to the business scenario requirements. Since the question-answering samples constructed in step 301 contain more professional knowledge of the relevant business domain, they compensate for the lack of understanding of relevant business domain knowledge in the external evaluation model, thereby improving the professionalism of the dialogue robot evaluation.

[0092] Furthermore, as a refinement and extension of the specific implementation of the above embodiments, in order to improve the accuracy and professionalism of the evaluation, another evaluation method for chatbots is provided. The steps of fusing the scores of the first evaluation value and the second evaluation value to obtain a comprehensive evaluation value corresponding to each sample to be evaluated include:

[0093] Calculate the product of the first evaluation value and the second evaluation value to obtain the comprehensive evaluation value corresponding to each sample to be evaluated; or,

[0094] In this embodiment of the invention, the current execution end calculates the product of the first evaluation value and the second evaluation value, and uses the result of the product as the comprehensive evaluation value of each sample to be evaluated. For example, if the first evaluation value of the sample to be evaluated is "score: [[8]]" and the second evaluation value is 0.7, then the comprehensive evaluation value of the sample to be evaluated is 5.6, etc. This embodiment of the invention does not make specific limitations.

[0095] The first evaluation value and the second evaluation value are standardized, and the standardized first evaluation value and the second evaluation value are weighted and summed to obtain the comprehensive evaluation value corresponding to each sample to be evaluated.

[0096] In this embodiment of the invention, the current execution terminal performs standardization processing on the first evaluation value and the second evaluation value. The standardization process represents the transformation of the original data according to a certain proportion through a specific mathematical transformation, so that it falls within a small, specific interval, such as 0 to 1 or -1 to 1, eliminating differences in the properties, dimensions, orders of magnitude, and other characteristic attributes of different variables, and transforming them into a dimensionless relative value. After standardizing the first and second evaluation values, the current execution terminal performs a weighted sum of the standardized first and second evaluation values ​​to obtain the comprehensive evaluation value corresponding to each sample to be evaluated, expressed by the following formula:

[0097] Z = αx + βy

[0098] Where Z is the comprehensive evaluation value, x is the first evaluation value after standardization, y is the second evaluation value after standardization, and α and β are the weights of the first and second evaluation values, respectively. α and β can be adjusted according to the evaluation results, and this embodiment of the invention does not impose specific limitations on them.

[0099] Furthermore, as a refinement and extension of the specific implementation of the above embodiments, in order to quickly obtain evaluation results and make the evaluation results serve subsequent development work, another evaluation method for chatbots is provided, such as... Figure 4 As shown, the steps of determining the evaluation result of each of the samples to be evaluated based on the comprehensive evaluation value, and annotating the robot's response based on the evaluation result, include:

[0100] 401. Compare the comprehensive evaluation value with a reasonable evaluation threshold;

[0101] 402. If the comprehensive evaluation value is greater than or equal to the reasonable evaluation threshold, the evaluation result is determined to be reasonable, and reasonable is marked on the corresponding robot response;

[0102] 403. If the comprehensive evaluation value is less than the reasonable evaluation threshold, the evaluation result will be determined as unreasonable, and unreasonable will be marked on the corresponding robot response.

[0103] In this embodiment of the invention, the current execution end sets a reasonable evaluation threshold for the response based on the evaluation requirements, and compares the comprehensive evaluation value with the reasonable evaluation threshold. If the comprehensive evaluation value is greater than or equal to the reasonable evaluation threshold, the evaluation result is determined to be reasonable, and reasonable is marked on the corresponding robot response. For example, in the financial insurance scenario, the reasonable evaluation threshold is set to 5.5. If the comprehensive evaluation value of the dialogue between the user and the agent, "User: Does this insurance cost money?\nAgent: Our insurance requires payment, and it is calculated based on your age." is 5.6, then the evaluation result is determined to be reasonable, and reasonable is marked on the robot response, "Our insurance requires payment, and it is calculated based on your age." Conversely, if the comprehensive evaluation value is less than the reasonable evaluation threshold, the evaluation result is determined to be unreasonable, and unreasonable is marked on the corresponding robot response. This embodiment of the invention does not impose specific limitations. It should be noted that, assuming the total number of samples to be evaluated is N, we repeat the above evaluation process N times to obtain the overall evaluation result of each sample to be evaluated, i.e., the evaluation result of the chatbot.

[0104] This invention provides a method for evaluating chatbots. Compared with existing technologies, this invention obtains the dialogue context and robot responses of the chatbot to be evaluated, forming multiple evaluation samples. The evaluation samples are then processed based on preset prompts and an external evaluation model to obtain a first evaluation value. A second evaluation value is obtained by further evaluating the samples based on a pre-trained dialogue evaluation reward model. The first and second evaluation values ​​are then fused to obtain a comprehensive evaluation value corresponding to each sample. The evaluation result for each sample is determined based on the comprehensive evaluation value, and the robot responses are annotated based on the evaluation results. This method achieves automatic evaluation of chatbots, featuring low evaluation cost and high efficiency. Furthermore, it maintains consistency before and after evaluation, eliminating the subjective judgment of annotators and making the evaluation more fair and objective. In addition, this invention uses an external evaluation model and a pre-trained dialogue evaluation reward model simultaneously for evaluation, which not only improves the accuracy of the evaluation but also makes the evaluation conclusions more professional.

[0105] As a response to the above Figure 1 The implementation of the method shown in this invention provides an evaluation device for a chatbot, such as... Figure 5 As shown, the device includes:

[0106] The sample acquisition module 51 is used to acquire the dialogue context and robot responses of the chatbot to be evaluated, forming multiple samples to be evaluated.

[0107] The model evaluation module 52 is used to evaluate the sample to be evaluated based on preset prompt words and an external evaluation model to obtain a first evaluation value; and to evaluate the sample to be evaluated based on a pre-trained dialogue evaluation and reward model to obtain a second evaluation value.

[0108] The comprehensive evaluation module 53 is used to fuse the scores of the first evaluation value and the second evaluation value to obtain a comprehensive evaluation value corresponding to each sample to be evaluated.

[0109] The result determination module 54 is used to determine the evaluation result of each of the samples to be evaluated based on the comprehensive evaluation value, and to perform annotation processing on the robot response based on the evaluation result.

[0110] Furthermore, the device also includes a setting module for:

[0111] The prompts are set according to the assessment requirements. The prompts include role declaration, specific tasks, task requirements and output format; the output format includes the output format of the first assessment value.

[0112] Furthermore, the model evaluation module 52 includes a first evaluation unit, used for:

[0113] The preset prompt words are concatenated with the sample to be evaluated to obtain the concatenated text.

[0114] The concatenated text is transmitted to an external evaluation model via an external interface, so that the external evaluation model can evaluate the concatenated text and obtain a first evaluation value.

[0115] Receive the first evaluation value returned by the external evaluation model.

[0116] Furthermore, the first evaluation unit is also used for:

[0117] In the samples to be evaluated, the dialogue context and the role corresponding to the robot's response are respectively marked;

[0118] Add a semantic end marker to the end of each dialogue in the sample to be evaluated after the roles are identified;

[0119] The sample to be evaluated, after having the semantic end marker added, is appended to the preset prompt word to obtain the concatenated text.

[0120] Furthermore, the model evaluation module 52 also includes a second evaluation unit, used for:

[0121] Obtain historical business dialogue text data, and construct question-and-answer sample pairs based on the historical business dialogue text data, wherein the question-and-answer sample pairs include positive samples and negative samples;

[0122] A reward model is constructed using a pairwise loss function, and the reward model is trained based on the positive and negative samples to obtain the dialogue evaluation reward model.

[0123] Furthermore, the comprehensive evaluation module 53 is also used for:

[0124] Calculate the product of the first evaluation value and the second evaluation value to obtain the comprehensive evaluation value corresponding to each sample to be evaluated; or,

[0125] The first evaluation value and the second evaluation value are standardized, and the standardized first evaluation value and the second evaluation value are weighted and summed to obtain the comprehensive evaluation value corresponding to each sample to be evaluated.

[0126] Furthermore, the result determination module 54 is also used for:

[0127] Compare the comprehensive evaluation value with a reasonable evaluation threshold;

[0128] If the comprehensive evaluation value is greater than or equal to the reasonable evaluation threshold, the evaluation result will be determined as reasonable, and reasonable will be marked on the corresponding robot response;

[0129] If the comprehensive evaluation value is less than the reasonable evaluation threshold, the evaluation result will be determined as unreasonable, and unreasonableness will be marked on the corresponding robot response.

[0130] This invention provides an evaluation device for chatbots. Compared with existing technologies, this invention obtains the dialogue context and robot responses of the chatbot to be evaluated, forming multiple evaluation samples. The evaluation samples are then processed based on preset prompts and an external evaluation model to obtain a first evaluation value. A second evaluation value is obtained by further processing the samples based on a pre-trained dialogue evaluation reward model. The first and second evaluation values ​​are then fused to obtain a comprehensive evaluation value corresponding to each sample. The evaluation result for each sample is determined based on the comprehensive evaluation value, and the robot responses are annotated based on the evaluation results. This achieves automatic evaluation of chatbots, characterized by low evaluation cost and high efficiency. Furthermore, consistency is maintained before and after evaluation, eliminating the subjective judgment of annotators and making the evaluation more fair and objective. In addition, this invention uses an external evaluation model and a pre-trained dialogue evaluation reward model simultaneously for evaluation, which not only improves the accuracy of the evaluation but also makes the evaluation conclusions more professional.

[0131] According to one embodiment of the present invention, a storage medium is provided, the storage medium storing at least one executable instruction, the computer-executable instruction being capable of executing the evaluation method of the chatbot in any of the above method embodiments.

[0132] Figure 6 The diagram illustrates a structural schematic of a computer device according to an embodiment of the present invention. The specific embodiments of the present invention do not limit the specific implementation of the computer device.

[0133] like Figure 6 As shown, the computer device may include: a processor 602, a communications interface 604, a memory 606, and a communications bus 608.

[0134] The processor 602, communication interface 604, and memory 606 communicate with each other via communication bus 608.

[0135] Communication interface 604 is used to communicate with other network elements such as clients or other servers.

[0136] The processor 602 is used to execute program 610, which can specifically perform the relevant steps of the evaluation method of the aforementioned chatbot.

[0137] Specifically, program 610 may include program code that includes computer operation instructions.

[0138] Processor 602 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The computer device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or they may be processors of different types, such as one or more CPUs and one or more ASICs.

[0139] Memory 606 is used to store program 610. Memory 606 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0140] Specifically, program 610 can be used to cause processor 602 to perform the following operations:

[0141] Obtain the dialogue context and robot responses of the chatbot to be evaluated, forming multiple samples to be evaluated;

[0142] The sample to be evaluated is evaluated based on preset prompts and an external evaluation model to obtain a first evaluation value; and the sample to be evaluated is evaluated based on a pre-trained dialogue evaluation and reward model to obtain a second evaluation value.

[0143] The scores of the first evaluation value and the second evaluation value are fused to obtain a comprehensive evaluation value corresponding to each sample to be evaluated.

[0144] The evaluation results for each of the samples to be evaluated are determined based on the comprehensive evaluation value, and the robot's response is annotated based on the evaluation results.

[0145] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0146] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for evaluating a chatbot, characterized in that, include: Obtain the dialogue context and robot responses of the chatbot to be evaluated, forming multiple samples to be evaluated; The sample to be evaluated is evaluated based on preset prompts and an external evaluation model to obtain a first evaluation value; and the sample to be evaluated is evaluated based on a pre-trained dialogue evaluation and reward model to obtain a second evaluation value. The scores of the first evaluation value and the second evaluation value are fused to obtain a comprehensive evaluation value corresponding to each sample to be evaluated. The evaluation results for each of the samples to be evaluated are determined based on the comprehensive evaluation value, and the robot's response is annotated based on the evaluation results.

2. The method according to claim 1, characterized in that, Before evaluating the sample based on preset prompts and an external evaluation model, the method further includes: The prompts are set according to the assessment requirements. The prompts include role declaration, specific tasks, task requirements and output format; the output format includes the output format of the first assessment value.

3. The method according to claim 1, characterized in that, The evaluation process based on preset prompts and an external evaluation model to obtain the first evaluation value includes: The preset prompt words are concatenated with the sample to be evaluated to obtain the concatenated text. The concatenated text is transmitted to an external evaluation model via an external interface, so that the external evaluation model can evaluate the concatenated text and obtain a first evaluation value. Receive the first evaluation value returned by the external evaluation model.

4. The method according to claim 3, characterized in that, The step of concatenating the preset prompt words with the sample to be evaluated to obtain the concatenated text includes: In the samples to be evaluated, the dialogue context and the role corresponding to the robot's response are respectively marked; Add a semantic end marker to the end of each dialogue in the sample to be evaluated after the roles are identified; The sample to be evaluated, after having the semantic end marker added, is appended to the preset prompt word to obtain the concatenated text.

5. The method according to claim 1, characterized in that, Before the pre-trained dialogue evaluation reward model evaluates the sample to be evaluated, the method further includes: Obtain historical business dialogue text data, and construct question-and-answer sample pairs based on the historical business dialogue text data, wherein the question-and-answer sample pairs include positive samples and negative samples; A reward model is constructed using a pairwise loss function, and the reward model is trained based on the positive and negative samples to obtain the dialogue evaluation reward model.

6. The method according to claim 1, characterized in that, The step of fusing the scores of the first evaluation value and the second evaluation value to obtain the comprehensive evaluation value corresponding to each sample to be evaluated includes: Calculate the product of the first evaluation value and the second evaluation value to obtain the comprehensive evaluation value corresponding to each sample to be evaluated; or, The first evaluation value and the second evaluation value are standardized, and the standardized first evaluation value and the second evaluation value are weighted and summed to obtain the comprehensive evaluation value corresponding to each sample to be evaluated.

7. The method according to any one of claims 1 to 6, characterized in that, The process of determining the evaluation result for each of the samples to be evaluated based on the comprehensive evaluation value, and then annotating the robot's response based on the evaluation result, includes: Compare the comprehensive evaluation value with a reasonable evaluation threshold; If the comprehensive evaluation value is greater than or equal to the reasonable evaluation threshold, the evaluation result will be determined as reasonable, and reasonable will be marked on the corresponding robot response; If the comprehensive evaluation value is less than the reasonable evaluation threshold, the evaluation result will be determined as unreasonable, and unreasonableness will be marked on the corresponding robot response.

8. An evaluation device for a conversational robot, characterized in that, include: The sample acquisition module is used to acquire the dialogue context and robot responses of the chatbot to be evaluated, forming multiple samples to be evaluated. The model evaluation module is used to evaluate the sample to be evaluated based on preset prompts and an external evaluation model to obtain a first evaluation value; and to evaluate the sample to be evaluated based on a pre-trained dialogue evaluation and reward model to obtain a second evaluation value. The comprehensive evaluation module is used to fuse the scores of the first evaluation value and the second evaluation value to obtain a comprehensive evaluation value corresponding to each sample to be evaluated. The result determination module is used to determine the evaluation result of each of the samples to be evaluated based on the comprehensive evaluation value, and to perform annotation processing on the robot's response based on the evaluation result.

9. A storage medium storing at least one executable instruction that performs an operation corresponding to the evaluation method of the chatbot as described in any one of claims 1-7.

10. A computer device, comprising a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction that causes the processor to perform an operation corresponding to the evaluation method of the conversational robot as described in any one of claims 1-7.

Images