A method and apparatus for evaluating the quality of a system recovery

Abstract

A method and apparatus for evaluate that quality of a system recovery is disclosed, The method comprises: Firstly, the system evaluation index of the target dialogue system is generated, includes generating a first evaluation indicator based on the thematic relevance between each selected response of the target dialog system and the corresponding question, A second evaluation index is generated based on the semantic similarity between each selected response of the target dialogue system and the corresponding manual response, and at least one of the third evaluation indicators generated based on the likelihood of each selected reply of the target dialog system being a generic reply, and then according to the generated system evaluation index, determine the response quality of the target conversation system, It can be seen that when evaluating the response quality of the target conversation system, the present application takes into account the subject correlation between the system response and the question, the semantic similarity between the system response and the manual response, and the possibility that the system response is a general response, so that the response quality ofthe target conversation system can be evaluated more accurately.

Description

technical field [0001] The present application relates to the technical field of natural language understanding, in particular to a method and device for evaluating the quality of system responses. Background technique [0002] As an important application of artificial intelligence technology, chatbots have been widely used in various intelligent terminal devices, such as mobile phones and wearable devices. At present, chatbots can be divided into task-type chatbots and non-task-type chatbots according to their uses, and chatbots with different uses have different implementation technologies. [0003] Task-based chatbots interact with users through the task-based dialogue system to complete related tasks, such as ordering food, weather query tasks, etc. The evaluation of the response quality of the task-based dialogue system is generally based on whether the task is completed or not, and the completion of the task. It is judged by the number of dialogue rounds performed by ...

AI Technical Summary

Problems solved by technology

Among them, the search method is to find the most suitable reply from the existing dialogue database. The reply generated by the search formula is manually marked in advance, and no reply other than the dialogue database will be generated. For the evaluation of the reply quality of the retrieval system, you can use The accuracy of the retrieval is judged, for example, to judge whether the first K sentences obtained by the retrieval contain the best answer to the question, therefore, it can also evaluate the quality of the retrieval system’s reply well; while the generative method generally constructs The probability distribution model generates the reply with the highest probability under the premise of a given question sentence. Compared with the retrieval method, the generative method can generate replies that have not appeared in the existing corpus, which is flexible, but the generative method is also It may produce replies that do not conform to grammatical rules, or even replies that are less relevant to questions. Therefore, for generative methods, it is difficult to evaluate the quality of their replies, and it is difficult to evaluate the quality of their replies accurately.

Images