Question and answer system evaluation method and device, electronic equipment and storage medium

Abstract

The invention provides a question and answer system evaluation method and device, electronic equipment and a storage medium, and the method comprises the steps: inputting an evaluation question into a question and answer system, and obtaining an associated text obtained by the question and answer system through retrieval based on the evaluation question, and an output answer generated by the question and answer system based on the evaluation question and the associated text; based on the evaluation question and the associated text, determining the retrieval precision of the question and answer system; and determining an evaluation result of the question and answer system based on the retrieval precision and the output answer. According to the method and device, the electronic equipment and the storage medium provided by the invention, comprehensive evaluation is carried out on the question-answering system from two aspects of retrieval performance and answer generation performance, manual participation is not needed, subjectivity and inconsistency possibly introduced by manual evaluation are eliminated, objectivity, comprehensiveness and repeatability of evaluation of the question-answering system are ensured, and the evaluation efficiency of the question-answering system is improved. Therefore, an accurate and reliable evaluation result is obtained.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing technology, and in particular to a question-answering system evaluation method, device, electronic device and storage medium. Background Art

[0002] As an important part of human-computer interaction, the question-and-answer system can answer questions and provide suggestions for users, greatly improving the user experience.

[0003] Performance evaluation of question-answering systems can promote iterative optimization of question-answering systems. Most current evaluation schemes are implemented through manual evaluation, but manual evaluation is subjective, which makes it difficult to ensure the consistency and fairness of the evaluation results. In addition, manual evaluation is time-consuming and labor-intensive, especially for question-answering systems with huge amounts of output data. It is difficult to evaluate all interactions, which makes the evaluation results less representative and directly affects the reliability of the evaluation results.

[0004] How to conduct performance evaluation on question-answering systems is still a problem that needs to be solved urgently. Summary of the invention

[0005] The present invention provides a question-answering system evaluation method, device, electronic device and storage medium, which are used to solve the defect of low reliability of manually evaluating question-answering systems in related technologies.

[0006] The present invention provides a question-answering system evaluation method, comprising: Inputting the evaluation question into the question-answering system, obtaining the associated text retrieved by the question-answering system based on the evaluation question, and the output answer generated by the question-answering system based on the evaluation question and the associated text; Determining the retrieval accuracy of the question-answering system based on the evaluation question and the associated text; Based on the retrieval accuracy and the output answer, an evaluation result of the question-answering system is determined.

[0007] According to a question-answering system evaluation method provided by the present invention, determining the retrieval accuracy of the question-answering system based on the evaluation question and the associated text includes: The retrieval accuracy of the question-answering system is determined based on the semantic similarity between the evaluation question and the associated text, and / or the text relevance between the evaluation question and the associated text.

[0008] According to a question-answering system evaluation method provided by the present invention, determining the evaluation result of the question-answering system based on the retrieval accuracy and the output answer includes: Determining the generation quality of the question-answering system based on the reference answer corresponding to the evaluation question and the output answer; Based on the retrieval accuracy and the generation quality, an evaluation result of the question-answering system is determined.

[0009] According to a question-answering system evaluation method provided by the present invention, determining the generation quality of the question-answering system based on the reference answer corresponding to the evaluation question and the output answer includes: The generation quality of the question-answering system is determined based on the semantic similarity between the reference answer and the output answer, and the fluency of the output answer.

[0010] According to a question-answering system evaluation method provided by the present invention, the fluency of the output answer is determined based on at least one of the sentence structure, grammatical correctness and naturalness of the output answer.

[0011] According to a question-answering system evaluation method provided by the present invention, the generation quality of the question-answering system is determined based on the semantic similarity between the reference answer and the output answer, and the fluency of the output answer, including: The generation quality of the question-answering system is determined based on the manual evaluation results of the output answer, at least one of the word overlap and content completeness of the output answer compared with the reference answer, the semantic similarity between the reference answer and the output answer, and the fluency of the output answer.

[0012] According to a question-answering system evaluation method provided by the present invention, the question-answering system includes a cascaded retriever and generator, the retriever is used to retrieve the associated text, and the generator is used to generate the output answer.

[0013] The present invention also provides a question-answering system evaluation device, comprising: A question-answering unit, configured to input an evaluation question into a question-answering system, obtain associated text retrieved by the question-answering system based on the evaluation question, and an output answer generated by the question-answering system based on the evaluation question and the associated text; An accuracy evaluation unit, used to determine the retrieval accuracy of the question-answering system based on the evaluation question and the associated text; A result output unit is used to determine the evaluation result of the question-answering system based on the retrieval accuracy and the output answer.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the question-answering system evaluation method described above is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the question-answering system evaluation methods described above.

[0016] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned question-answering system evaluation methods.

[0017] The question-answering system evaluation method, device, electronic device and storage medium provided by the present invention are combined with the process of the question-answering system first retrieving and then generating answers during the question-answering process, and the retrieval performance of the question-answering system is measured using the retrieval accuracy obtained based on the evaluation questions and related texts, and the answer generation performance of the question-answering system is measured based on the output answers. The question-answering system is comprehensively evaluated from two aspects: retrieval performance and answer generation performance, without the need for human intervention, thus eliminating the subjectivity and inconsistency that may be introduced by manual evaluation, and ensuring the objectivity, comprehensiveness and repeatability of the question-answering system evaluation, thereby obtaining accurate and reliable evaluation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a flow chart of the question-answering system evaluation method provided by the present invention.

[0020] Figure 2 It is a structural schematic diagram of the question-answering system evaluation device provided by the present invention.

[0021] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] With the rapid development of Natural Language Processing (NLP) technology, question-answering systems have emerged. As an important part of human-computer interaction, question-answering systems can answer questions and provide suggestions for users, greatly improving user experience.

[0024] Performance evaluation of question-answering systems can promote iterative optimization of question-answering systems. Currently, most evaluation schemes are implemented through manual evaluation, which has the following problems: First, manual evaluation is subjective. Different evaluators may have different evaluations of the performance of the same question-answering system due to personal preferences, experience, understanding, and other factors, which makes it difficult to ensure the consistency and fairness of the evaluation results.

[0025] Second, manual evaluation is time-consuming and laborious. When facing a question-answering system with a huge amount of output data, it is difficult to evaluate all interactions, which makes the evaluation results less representative and directly affects the reliability of the evaluation results.

[0026] Third, manual evaluation lacks comprehensiveness. Limited by labor and time costs, it is difficult to fully capture all aspects of the performance of the question-answering system during manual evaluation, such as the ability to handle complex or subtle problems, or the ability to conduct natural and engaging conversations. And when facing different scenarios, the question-answering system may show different styles and personalities, which makes it very difficult to develop fair evaluation indicators that are applicable to all question-answering systems. In addition, the continuous evolution of conversation patterns and user expectations also brings challenges to the updating of evaluation indicators.

[0027] In view of the above problems, an embodiment of the present invention provides a question-answering system evaluation method. Figure 1 It is a flow chart of the question-answering system evaluation method provided by the present invention. Figure 1 As shown, the method includes: Step 110, input the evaluation question into the question-answering system, obtain the associated text retrieved by the question-answering system based on the evaluation question, and the output answer generated by the question-answering system based on the evaluation question and the associated text.

[0028] The question-answering system here refers to the question-answering system that needs to be evaluated for performance, such as a question-answering chat system. The question-answering system can combine retrieval and generation technologies to achieve answers to input questions. For example, a question-answering system can be built based on the Retrieval-Augmented Generation (RAG) technology. The generated question-answering system can first retrieve information related to the question from a large number of documents based on retrieval technology, and then generate answers corresponding to the question and the retrieved information based on generation technology.

[0029] In the process of evaluating the question-answering system, the evaluation questions can be determined first. The evaluation questions here are the questions that are input to the question-answering system when evaluating the question-answering system. The number of evaluation questions can be one or more, and the evaluation questions can be questions in the question-answering dialogue collected in advance. It can be understood that each set of question-answering dialogues includes a question and an answer corresponding to the question.

[0030] Question-answering conversations can be obtained manually, that is, people can be hired to conduct and record the conversations, thereby ensuring the quality of the question-answering conversations, but the cost is relatively high; or they can be collected through web crawlers, specifically, question-answering conversations can be captured from question-answering websites, forums, etc., thereby obtaining a large number of question-answering conversations, but the quality of the question-answering conversations may vary; or public data sets can be selected to obtain question-answering conversations. This acquisition method may be limited by the amount of data or the type of data, but the quality of the question-answering conversations obtained is relatively high. Specifically, the appropriate way to obtain question-answering conversations can be selected based on the scale, quality, and budget of the question-answering conversations actually required.

[0031] Optionally, after obtaining the question-answering dialogue, data processing and evaluation can be performed on the question-answering dialogue, thereby using the questions in the question-answering dialogue after data processing and data evaluation as evaluation questions. Among them, data processing can include operations such as data cleaning, standardization, and annotation. Data cleaning can remove repeated, meaningless, or erroneous information, thereby ensuring data quality; standardization can achieve unified word segmentation, remove punctuation marks, etc., to facilitate subsequent processing; annotation can specifically be used to annotate questions and answers in the question-answering dialogue, such as annotating question types, answer types, etc. Data evaluation can include data volume evaluation, data quality evaluation, etc. Data volume evaluation can evaluate whether the data volume of the question-answering dialogue is large enough to meet the evaluation requirements of the question-answering system, and data quality evaluation can evaluate whether the question-answering dialogue is accurate, whether there is noise and erroneous information, and whether the question-answering dialogue is evenly distributed to avoid data bias.

[0032] After determining the evaluation question, the evaluation question can be input into the question-answering system, and the question-answering system searches for the input evaluation question to obtain the associated text of the evaluation question. Here, the associated text is the result of the question-answering system searching for the evaluation question. The associated text can be in the form of a text paragraph, and one evaluation question can correspond to one or more associated texts. Subsequently, the question-answering system can combine the input evaluation question and the retrieved associated text to generate and output the final answer. Here, the answer generated by the question-answering system is recorded as the output answer.

[0033] That is, after the evaluation question is input into the question-answering system, the associated text retrieved by the question-answering system and the output answer generated by the question-answering system can be obtained.

[0034] Step 120: Determine the retrieval accuracy of the question-answering system based on the evaluation question and the associated text.

[0035] Specifically, after obtaining the relevant text, the evaluation questions can be combined with the relevant text to measure the retrieval accuracy of the question-answering system. The retrieval accuracy here is a measurement indicator for the retrieval performance of the question-answering system, which can specifically characterize the retrieval accuracy of the question-answering system. It can be understood that a high retrieval accuracy indicates that the question-answering system can find text related to the input question, providing a good basis for answering the question. A low retrieval accuracy indicates that the question-answering system has difficulty finding text related to the input question, and the answer may be biased or wrong.

[0036] The determination of retrieval accuracy can be achieved by evaluating the correlation between the evaluation question and the associated text. For example, the semantic similarity between the evaluation question and the associated text can be calculated. The higher the similarity, the higher the retrieval accuracy. For another example, the correlation between the evaluation question and the associated text can be determined, that is, whether the evaluation question and the associated text are related. The higher the proportion of associated text related to the evaluation question in the associated text, the higher the retrieval accuracy.

[0037] Step 130: Determine the evaluation result of the question-answering system based on the retrieval accuracy and the output answer.

[0038] Specifically, after obtaining the retrieval accuracy, the performance of the question-answering system can be measured by combining the retrieval accuracy and the output answer, thereby obtaining the evaluation results of the question-answering system.

[0039] The output answer can intuitively reflect the answer generation performance of the question-answering system. For example, the answer generation performance of the question-answering system can be evaluated by evaluating the quality of the output answer. Here, the quality of the output answer can be evaluated by measuring the semantic similarity between the output answer and the reference answer corresponding to the evaluation question, or by measuring the fluency of the output answer itself, which is not specifically limited in the embodiment of the present invention.

[0040] Therefore, the evaluation results of the question-answering system can be determined by combining the retrieval performance of the question-answering system reflected by the retrieval accuracy and the answer generation performance of the question-answering system reflected by the output answers.

[0041] In the method provided in the embodiment of the present invention, the process of the question-answering system first searching and then generating answers during the question-answering process is combined, the retrieval performance of the question-answering system is measured using the retrieval accuracy obtained based on the evaluation questions and related texts, and the answer generation performance of the question-answering system is measured based on the output answers. The question-answering system is comprehensively evaluated from two aspects: retrieval performance and answer generation performance. No human intervention is required, thus eliminating the subjectivity and inconsistency that may be introduced by manual evaluation, ensuring the objectivity, comprehensiveness and repeatability of the question-answering system evaluation, thereby obtaining accurate and reliable evaluation results.

[0042] Moreover, by combining the retrieval performance and answer generation performance to evaluate the question-answering system, the functions of the question-answering system in information retrieval, response fluency, coherence, and dialogue flow can be fully covered, making the evaluation more comprehensive. The automated evaluation process can effectively adapt to the large number of interactive needs of the question-answering system, and adapt to the ever-changing dialogue patterns and user expectations. Moreover, objective and consistent evaluation indicators can be applied to different question-answering systems to ensure the fairness and repeatability of the evaluation. The evaluation results obtained in this way can identify the strengths and weaknesses of the question-answering system's performance, thereby guiding targeted improvements. Moreover, by creating basic evaluation results and comparative analysis, the evaluation performance of the same question-answering system at different times can be tracked, and industry best practices can be identified.

[0043] Based on the above embodiment, in step 120, determining the retrieval accuracy of the question-answering system based on the evaluation question and the associated text includes: The retrieval accuracy of the question-answering system is determined based on the semantic similarity between the evaluation question and the associated text, and / or the text relevance between the evaluation question and the associated text.

[0044] Specifically, when measuring the retrieval accuracy of the question-answering system based on the evaluation question and the associated text, reference may be made to the semantic similarity between the two and / or the text relevance between the two.

[0045] The semantic similarity between the evaluation question and the associated text refers to the degree of closeness between the evaluation question and the associated text at the semantic level. Semantic similarity can be calculated by, for example, word embedding, sentence embedding, or semantic matching algorithms. Specifically, the evaluation question and the associated text can be converted into vectors in a high-dimensional space, and the semantic similarity between the evaluation question and the associated text can be evaluated by calculating the cosine similarity, Euclidean distance, etc. between the vectors.

[0046] It can be understood that the higher the semantic similarity between the evaluation question and the associated text, the higher the retrieval accuracy of the question-answering system; the lower the semantic similarity between the associated texts, the lower the retrieval accuracy of the question-answering system. Therefore, the retrieval accuracy of the question-answering system can be determined based on the semantic similarity between the evaluation question and the associated text.

[0047] The text relevance between the evaluation question and the related text refers to whether there is a direct association or connection between the two texts. In the case of binary classification, text relevance is usually simplified to 0 or 1, that is, relevant or irrelevant. Text relevance can be obtained based on the pre-trained binary classification model.

[0048] It can be understood that, among the related texts retrieved, the more related texts are related to the evaluation question or the higher their proportion, the higher the retrieval accuracy of the question-answering system; the fewer related texts are related to the evaluation question or the lower their proportion, the lower the retrieval accuracy of the question-answering system. Therefore, the retrieval accuracy of the question-answering system can be determined based on the text correlation between the evaluation question and the related texts.

[0049] In addition, the retrieval accuracy of the question-answering system can be determined by combining the semantic similarity between the evaluation question and the associated text, as well as the text relevance between the evaluation question and the associated text. For example, the retrieval accuracy of the question-answering system can be calculated by the following formula:

[0050] In the formula, is the number of associated texts, To evaluate the problem, is the i-th associated text, To evaluate the semantic similarity between the question and the i-th associated text, To measure the text relevance between the question and the i-th associated text, The value of is 0 or 1.

[0051] The method provided in the embodiment of the present invention realizes accurate measurement of the retrieval accuracy of the question-answering system based on semantic similarity and / or text relevance, and provides conditions for realizing accurate and reliable question-answering system evaluation.

[0052] Based on any of the above embodiments, in step 130, determining the evaluation result of the question-answering system based on the retrieval accuracy and the output answer includes: Determining the generation quality of the question-answering system based on the reference answer corresponding to the evaluation question and the output answer; Based on the retrieval accuracy and the generation quality, an evaluation result of the question-answering system is determined.

[0053] Specifically, when evaluating the question-answering system, the question-answering system can be comprehensively evaluated from two aspects: retrieval performance and answer generation performance. Among them, the retrieval accuracy obtained in step 120 can characterize the retrieval performance of the question-answering system, and the answer generation performance can be characterized by the generation quality of the question-answering system. The generation quality here can reflect the strength of the generation ability of the question-answering system. If the generation quality is high, it means that the question-answering system can generate smooth, fluent and semantically consistent text, providing users with a good answer experience. If the generation quality is low, it means that the text generated by the question-answering system may contain grammatical errors, semantic errors or logical errors, which affect the user experience.

[0054] Here, the generation quality of the question-answering system can be determined by combining the reference answers corresponding to the evaluation questions and the output answers generated by the question-answering system. The reference answers are the answers contained in a set of question-answering dialogues to which the evaluation questions belong. The reference answers can be understood as the standard answers obtained by answering the evaluation questions and can be used as references.

[0055] When measuring the generation quality of the question-answering system, the output answer can be compared with the reference answer, and the generation quality can be determined by measuring the similarities and differences between the output answer and the reference answer. For example, the semantic similarity between the output answer and the reference answer can be calculated. The higher the semantic similarity between the two, the closer the output answer is to the reference answer, and the higher the generation quality of the question-answering system is. The embodiment of the present invention does not specifically limit this.

[0056] After obtaining the retrieval accuracy and generation quality of the question-answering system, the evaluation result of the question-answering system can be determined by combining the two. For example, a weighted summation can be performed on the retrieval accuracy and generation quality, and the result of the weighted summation can be used as the evaluation result of the question-answering system, which can be specifically expressed as the following formula:

[0057] In the formula, The preset weights.

[0058] In the method provided in the embodiment of the present invention, the performance of the question-answering system is evaluated by combining the retrieval accuracy and the generation quality, which can fully reflect the overall performance of the question-answering system.

[0059] It is understandable that if the evaluation result reflecting the overall performance of the question-answering system has a high score, it means that the question-answering system performs well in both retrieval and generation capabilities and can provide users with high-quality question-answering services. If the evaluation result score is low, it means that the question-answering system is insufficient in retrieval or generation capabilities and needs to be improved.

[0060] By analyzing the evaluation results, we can effectively evaluate the performance of the question-answering system and provide a basis for improving the question-answering system. It should be noted that during the analysis process, attention should be paid to the quality of the evaluation data, the rationality of the evaluation indicators, and the interpretability of the analysis results.

[0061] Based on any of the above embodiments, in step 130, determining the generation quality of the question-answering system based on the reference answer corresponding to the evaluation question and the output answer includes: The generation quality of the question-answering system is determined based on the semantic similarity between the reference answer and the output answer, and the fluency of the output answer.

[0062] Specifically, when measuring the generation quality of a question-answering system, one can not only compare the similarities and differences between the output answers and the reference answers, but also combine the characteristics of the reference answers themselves.

[0063] The semantic similarity between the reference answer and the output answer refers to the degree of closeness between the reference answer and the output answer at the semantic level. Specifically, the reference answer and the output answer can be converted into vectors in a high-dimensional space, and the semantic similarity between the reference answer and the output answer can be evaluated by calculating the cosine similarity, Euclidean distance, etc. between the vectors.

[0064] It can be understood that the semantic similarity between the reference answer and the output answer can characterize the similarities and differences between the reference answer and the output answer. The higher the semantic similarity, the more accurate and comprehensive the output answer is, and the higher the generation quality of the question and answer system is. The lower the semantic similarity, the worse the accuracy and comprehensiveness of the output answer is, and the lower the generation quality of the question and answer system is.

[0065] The fluency of the output answer is a measure of whether the output answer conforms to the grammatical rules and logical structure of natural language and whether it has good expressive ability. The fluency determines whether the user can understand the content of the output answer smoothly and unimpeded when reading the output answer. The fluency can be determined by scoring the output answer through a pre-trained language model, such as BERT (Bidirectional Encoder Representations from Transformers), or can be evaluated from the grammatical correctness of the output answer, the complexity of the sentence structure, etc. through a syntactic analysis model. The embodiment of the present invention does not specifically limit this.

[0066] It can be understood that the higher the fluency of the output answer, the higher the generation quality of the question-answering system, and the lower the fluency of the output answer, the lower the generation quality of the question-answering system.

[0067] After obtaining the semantic similarity between the reference answer and the output answer, as well as the fluency of the output answer, the generation quality of the question-answering system can be determined by combining the two. For example, the semantic similarity and fluency can be weighted and summed, and the result of the weighted sum can be used as the generation quality of the question-answering system, which can be expressed as the following formula:

[0068] In the formula, To output the answer, For reference answer, is the semantic similarity between the reference answer and the output answer, To output the answer smoothly, and The preset weights.

[0069] Based on any of the above embodiments, the fluency of the output answer is determined based on at least one of the sentence structure, grammatical correctness and naturalness of the output answer.

[0070] Specifically, sentence structure refers to the arrangement and relationship between words, phrases or clauses in the output answer. A clear and reasonable sentence structure helps users understand the semantics of the output answer. When evaluating the sentence structure of the output answer, you can focus on whether the output answer has a clear subject, predicate and object, and whether the relationship between them is clear; or, you can analyze whether the attributive, adverbial, complement and other components in the output answer are complete, and whether their positions are reasonable. It is understandable that the clearer and more reasonable the sentence structure of the output answer, the higher the fluency of the output answer.

[0071] Grammatical correctness reflects whether the output answer conforms to the grammatical rules of the language. When evaluating the grammatical correctness of the output answer, you can focus on whether the tense and voice of the output answer are correct and whether they are consistent; or you can focus on whether the subject and predicate of the output answer are consistent; in addition, you can check whether the usage of nouns, verbs, adjectives, adverbs and other vocabulary in the output answer is correct, and check whether the use of punctuation marks in the output answer is appropriate and whether they contribute to the understanding and reading of the output answer. It can be understood that grammatical correctness directly affects the fluency of the output answer. The fluency of the grammatically correct output answer is higher than that of the grammatically incorrect output answer.

[0072] Naturalness reflects whether the output answer is natural and fluent in expression and whether it conforms to the reader's reading habits and thinking logic. When evaluating the naturalness of the output answer, you can pay attention to whether the output answer is clear and easy to understand, whether there is ambiguity; you can also pay attention to whether the language style of the output answer is consistent; you can also pay attention to whether the output answer uses appropriate transitional vocabulary to connect sentences or paragraphs, etc. It is understandable that the higher the naturalness of the output answer, the higher the fluency of the output answer.

[0073] Therefore, the fluency of the output answer can be determined based on at least one of the sentence structure, grammatical correctness and naturalness of the output answer. For example, the fluency can be calculated based on the sentence structure, grammatical correctness and naturalness of the output answer respectively, and then the three fluencies obtained are weightedly summed to obtain the final fluency.

[0074] Based on any of the above embodiments, in step 130, determining the generation quality of the question-answering system based on the semantic similarity between the reference answer and the output answer, and the fluency of the output answer, includes: The generation quality of the question-answering system is determined based on the manual evaluation results of the output answer, at least one of the word overlap and content completeness of the output answer compared with the reference answer, the semantic similarity between the reference answer and the output answer, and the fluency of the output answer.

[0075] Specifically, in addition to applying the semantic similarity between the reference answer and the output answer, and the fluency of the output answer to determine the generation quality of the question-answering system, the manual evaluation results of the output answer, and at least one of the word overlap and content completeness of the output answer compared to the reference answer can also be combined to assist in determining the generation quality.

[0076] The manual evaluation results of the output answers may be the results obtained by manual evaluation of the output answers. That is, a person may be invited to judge whether the output answers generated by the question-answering system are accurate, reasonable, and natural based on their subjective feelings and professional knowledge. Specifically, the manual evaluation results may be collected by designing questionnaires, conducting user surveys, and the like.

[0077] The word overlap between the output answer and the reference answer may be an indicator that compares the overlap between the output answer and the reference answer in units of words. For example, the word overlap between the output answer and the reference answer may be represented by BLEU (Bilingual Evaluation Understudy).

[0078] The completeness of the output answer compared to the reference answer can be determined from the perspective of the content contained in the output answer and the content contained in the reference answer. For example, ROUGE (Recall-Oriented Understudy for Gisting Evaluation) can be used to represent the completeness of the output answer compared to the reference answer.

[0079] In the method provided in an embodiment of the present invention, the manual evaluation results of the output answers, and at least one of the word overlap and content completeness of the output answers compared with the reference answers are used as factors to assist in determining the generation quality of the question-answering system, which helps to further ensure the rationality and reliability of the generation quality determination.

[0080] Based on any of the above embodiments, the question-answering system includes a cascaded retriever and generator, wherein the retriever is used to retrieve the associated text, and the generator is used to generate the output answer.

[0081] Specifically, the question-answering system can be built based on the RAG architecture. That is, the question-answering system includes a retriever and a generator.

[0082] Among them, the retriever is responsible for retrieving the relevant text paragraphs of the input question from the candidate text library as the associated text. The retriever is specifically a model based on text retrieval, and various retrieval technologies can be used. The retriever can be a bag-of-words model, TF-IDF (Term Frequency-Inverse Document Frequency), BM25 (Best Matching 25), BERT, etc. The input of the retriever is usually a question or query raised by the user. For example, in the process of evaluating the question-answering system, the input of the retriever is the evaluation question. The output of the retriever is the text paragraph or document related to the input question retrieved from the candidate text library. For example, in the process of evaluating the question-answering system, the output of the retriever is the associated text. The candidate text library can be constructed based on a large amount of text data collected from the Internet or other sources. In the process of constructing the candidate text library, the text data can also be processed, such as word segmentation, punctuation removal, encoding, etc.

[0083] In the question-answering system, the role of the retriever is to retrieve relevant text paragraphs or documents from the candidate text library based on the input question to provide more contextual information and background knowledge to support the generator in generating more accurate output answers.

[0084] The generator is responsible for generating the final output answer based on the input question and the related text retrieved. The generator is usually a text generation model, such as RNN (Recurrent Neural Network), LSTM (Long Short-Term Memory), Transformer, etc. The input of the generator includes the question asked by the user and the related text paragraphs or documents retrieved from the retriever. For example, in the process of evaluating the question-answering system, the input of the generator is the evaluation question and the related text. The output of the generator is the final response or answer generated based on the input question and the related text paragraph. For example, in the process of evaluating the question-answering system, the output of the generator is the output answer.

[0085] In question-answering systems, the role of the generator is to generate natural and fluent text responses based on the input questions and related text paragraphs to answer users' questions or complete tasks.

[0086] In the question-answering system, the retriever and generator are integrated together to form an end-to-end model. Therefore, in the question-answering process, the questions raised by the user are first processed by the retriever, which retrieves the text paragraphs related to the question from the candidate text library, and then transmits the retrieved text paragraphs together with the question to the generator. The generator then generates the final output answer based on the question and the retrieved text paragraphs.

[0087] In the process of building a question-answering system, the integrated retriever and generator can be trained to learn how to effectively retrieve relevant text paragraphs and generate high-quality output answers. The training process usually includes supervised learning using annotated datasets to optimize model parameters.

[0088] Specifically, during the training process, the model parameters can be initialized first, including randomly assigning initial values ​​to each parameter in the retriever and generator. Subsequently, the parameters in the retriever and generator can be updated and iterated through forward transmission and back propagation. In this process, forward propagation refers to inputting the input question into the retriever until the output answer generated by the generator is obtained, and back propagation refers to calculating the error between the output answer and the true answer, and using the error back propagation to update the parameters. The updated parameters here refer to the synchronous update and optimization of the parameters of the retriever and generator.

[0089] Moreover, during the training process, a suitable loss function can be used during back propagation, for example, a cross entropy loss function or other suitable loss functions can be used to measure the error between the output answer and the true answer. In addition, during back propagation, optimization algorithms such as stochastic gradient descent algorithm and Adam algorithm can be used to update parameters. In addition, regularization techniques such as L1 regularization and L2 regularization can be used to prevent model overfitting. In addition, early stopping techniques can also be used to avoid model overfitting.

[0090] In addition, after training is completed, you can use indicators such as accuracy, F1 value, BLEU score, etc. to evaluate the training effect. Among them, accuracy refers to the proportion of correct answers generated. F1 value refers to the harmonic mean of the proportion of correct answers generated and recall. BLEU score refers to the similarity between the generated answer and the true answer.

[0091] Based on any of the above embodiments, for the question-answering system, the evaluation can be implemented based on the following steps: First, collect the evaluation data: The evaluation data here may be in the form of a question-and-answer dialogue, that is, it may be a combination of evaluation questions and reference answers.

[0092] Secondly, calculate the evaluation indicators: The evaluation question can be input into the question-answering system to obtain the associated text retrieved by the retriever in the question-answering system for the evaluation question, and the output answer generated by the generator in the question-answering system based on the evaluation question and the associated text.

[0093] Then, the evaluation index can be calculated based on the associated text and the output answer. Specifically, the retrieval accuracy of the question-answering system can be determined based on the relevance between the evaluation question and the associated text; the generation quality of the question-answering system can be determined based on the similarity between the output answer and the reference answer, as well as the fluency of the output answer. Finally, the evaluation result of the question-answering system is determined by combining the retrieval accuracy and generation quality.

[0094] The question-answering system evaluation device provided by the present invention is described below. The question-answering system evaluation device described below and the question-answering system evaluation method described above can be referenced to each other.

[0095] Figure 2 is a schematic diagram of the structure of the question-answering system evaluation device provided by the present invention, such as Figure 2 As shown, the device comprises: A question-answering unit 210, configured to input an evaluation question into a question-answering system, obtain associated text retrieved by the question-answering system based on the evaluation question, and output an answer generated by the question-answering system based on the evaluation question and the associated text; An accuracy evaluation unit 220, configured to determine the retrieval accuracy of the question-answering system based on the evaluation question and the associated text; The result output unit 230 is used to determine the evaluation result of the question-answering system based on the retrieval accuracy and the output answer.

[0096] In the device provided in the embodiment of the present invention, combined with the process of the question-answering system first searching and then generating answers during the question-answering process, the retrieval performance of the question-answering system is measured by the retrieval accuracy obtained based on the evaluation questions and related texts, and the answer generation performance of the question-answering system is measured based on the output answers. The question-answering system is comprehensively evaluated from two aspects: retrieval performance and answer generation performance. No human intervention is required, thus eliminating the subjectivity and inconsistency that may be introduced by manual evaluation, ensuring the objectivity, comprehensiveness and repeatability of the question-answering system evaluation, thereby obtaining accurate and reliable evaluation results.

[0097] Based on any of the above embodiments, the accuracy evaluation unit is specifically used for: The retrieval accuracy of the question-answering system is determined based on the semantic similarity between the evaluation question and the associated text, and / or the text relevance between the evaluation question and the associated text.

[0098] Based on any of the above embodiments, the result output unit is specifically used for: Determining the generation quality of the question-answering system based on the reference answer corresponding to the evaluation question and the output answer; Based on the retrieval accuracy and the generation quality, an evaluation result of the question-answering system is determined.

[0099] Based on any of the above embodiments, the result output unit is specifically used for: The generation quality of the question-answering system is determined based on the semantic similarity between the reference answer and the output answer, and the fluency of the output answer.

[0100] Based on any of the above embodiments, the fluency of the output answer is determined based on at least one of the sentence structure, grammatical correctness and naturalness of the output answer.

[0101] Based on any of the above embodiments, the result output unit is specifically used for: The generation quality of the question-answering system is determined based on the manual evaluation results of the output answer, at least one of the word overlap and content completeness of the output answer compared with the reference answer, the semantic similarity between the reference answer and the output answer, and the fluency of the output answer.

[0102] Based on any of the above embodiments, the question-answering system includes a cascaded retriever and generator, wherein the retriever is used to retrieve the associated text, and the generator is used to generate the output answer.

[0103] Figure 3 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communication interface 320 and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute the question-answering system evaluation method, which includes: Inputting the evaluation question into the question-answering system, obtaining the associated text retrieved by the question-answering system based on the evaluation question, and the output answer generated by the question-answering system based on the evaluation question and the associated text; Determining the retrieval accuracy of the question-answering system based on the evaluation question and the associated text; Based on the retrieval accuracy and the output answer, an evaluation result of the question-answering system is determined.

[0104] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0105] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the question-answering system evaluation method provided by the above methods, the method includes: Inputting the evaluation question into the question-answering system, obtaining the associated text retrieved by the question-answering system based on the evaluation question, and the output answer generated by the question-answering system based on the evaluation question and the associated text; Determining the retrieval accuracy of the question-answering system based on the evaluation question and the associated text; Based on the retrieval accuracy and the output answer, an evaluation result of the question-answering system is determined.

[0106] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the question-answering system evaluation method provided by the above methods is implemented, and the method includes: Inputting the evaluation question into the question-answering system, obtaining the associated text retrieved by the question-answering system based on the evaluation question, and the output answer generated by the question-answering system based on the evaluation question and the associated text; Determining the retrieval accuracy of the question-answering system based on the evaluation question and the associated text; Based on the retrieval accuracy and the output answer, an evaluation result of the question-answering system is determined.

[0107] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0108] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A question-answering system evaluation method, characterized in that: include: Inputting the evaluation question into the question-answering system, obtaining the associated text retrieved by the question-answering system based on the evaluation question, and the output answer generated by the question-answering system based on the evaluation question and the associated text; Determining the retrieval accuracy of the question-answering system based on the evaluation question and the associated text; Based on the retrieval accuracy and the output answer, an evaluation result of the question-answering system is determined.

2. The question-answering system evaluation method according to claim 1, characterized in that: The step of determining the retrieval accuracy of the question-answering system based on the evaluation question and the associated text includes: The retrieval accuracy of the question-answering system is determined based on the semantic similarity between the evaluation question and the associated text, and / or the text relevance between the evaluation question and the associated text.

3. The question-answering system evaluation method according to claim 1, characterized in that: The step of determining the evaluation result of the question-answering system based on the retrieval accuracy and the output answer includes: Determining the generation quality of the question-answering system based on the reference answer corresponding to the evaluation question and the output answer; Based on the retrieval accuracy and the generation quality, an evaluation result of the question-answering system is determined.

4. The question-answering system evaluation method according to claim 3, characterized in that: The determining the generation quality of the question-answering system based on the reference answer corresponding to the evaluation question and the output answer includes: The generation quality of the question-answering system is determined based on the semantic similarity between the reference answer and the output answer, and the fluency of the output answer.

5. The question-answering system evaluation method according to claim 4, characterized in that: The fluency of the output answer is determined based on at least one of sentence structure, grammatical correctness and naturalness of the output answer.

6. The question-answering system evaluation method according to claim 4, characterized in that: The determining the generation quality of the question-answering system based on the semantic similarity between the reference answer and the output answer, and the fluency of the output answer, includes: The generation quality of the question-answering system is determined based on the manual evaluation results of the output answer, at least one of the word overlap and content completeness of the output answer compared with the reference answer, the semantic similarity between the reference answer and the output answer, and the fluency of the output answer.

7. The question-answering system evaluation method according to any one of claims 1 to 6, characterized in that: The question-answering system includes a cascaded retriever and a generator, wherein the retriever is used to retrieve the associated text, and the generator is used to generate the output answer.

8. A question-answering system evaluation device, characterized in that: include: A question-answering unit, configured to input an evaluation question into a question-answering system, obtain associated text retrieved by the question-answering system based on the evaluation question, and an output answer generated by the question-answering system based on the evaluation question and the associated text; An accuracy evaluation unit, used to determine the retrieval accuracy of the question-answering system based on the evaluation question and the associated text; A result output unit is used to determine the evaluation result of the question-answering system based on the retrieval accuracy and the output answer.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the question-answering system evaluation method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the question-answering system evaluation method according to any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the question-answering system evaluation method according to any one of claims 1 to 7 is implemented.

Images