Evaluation optimization method and device for recall result of knowledge question-answering system, equipment and medium

Abstract

The invention provides an evaluation optimization method and device for recall results of a knowledge question-answering system, equipment and a medium, and the method carries out finer-grained position matching on labeled document fragments and recall document fragments based on a dynamically adjusted threshold value and a constructed document tree, thereby carrying out precise evaluation on the knowledge question-answering system. The anti-interference capability of the system is evaluated by introducing a negative sample problem, so that the robustness of the knowledge question-answering system to noise data is improved; and by introducing the similarity between the original document fragment vector and the annotated document fragment vector, the problem of low recall rate caused by incompleteness of the annotated document fragment is avoided.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and in particular to an evaluation optimization method, device, equipment and medium for recall results of a knowledge question answering system. Background Art

[0002] With the widespread application of large-scale pre-trained language models in the field of natural language processing, the knowledge base question answering system RAG based on large models has gradually become an effective tool for solving complex problems. The RAG system retrieves document fragments of original documents in the knowledge base and generates answers in combination with the generative model to provide complete answers to user questions; in the knowledge question answering system, the recall result refers to the set of all potential answers retrieved by the system for a given question. This set includes all answer candidates that the system believes are relevant to the user's question, regardless of whether these answers are ultimately selected as the correct answer.

[0003] However, the traditional recall result evaluation and optimization method is based on the document level, lacking precise evaluation at a smaller granularity, which in turn affects the optimization quality of the knowledge base question answering system. Summary of the invention

[0004] In view of this, the purpose of this application is to provide an evaluation optimization method, device, equipment and medium for the recall results of a knowledge question and answer system, which can perform more fine-grained position matching of annotated document fragments and recalled document fragments based on dynamically adjusted thresholds and document trees, thereby accurately evaluating the knowledge question and answer system and improving the optimization quality of the knowledge base question and answer system.

[0005] In a first aspect, an embodiment of the present application provides a method for optimizing the evaluation of recall results of a knowledge question answering system, and the method for optimizing the evaluation of recall results of a knowledge question answering system includes:

[0006] Obtain the annotated document fragments and recall results corresponding to the target user's question; the annotated document fragment refers to the document fragment containing the standard answer corresponding to the target user's question; the recall result includes the recalled document fragments of all potential answers retrieved by the target knowledge question answering system in the original document for the target user's question; the recalled document fragment carries its position information in the original document;

[0007] The document fragments obtained by splitting the original document based on each fine-grained level are used as nodes to construct a document tree; each node in the document tree carries its position information in the original document;

[0008] The position information of the node in the document tree whose similarity with the annotated document fragment is greater than the target similarity threshold and has the highest similarity is determined as the position information corresponding to the annotated document fragment; the target similarity threshold is obtained by adjusting the general similarity threshold based on the characteristics of the annotated document fragment itself;

[0009] The recall result of the target knowledge question answering system is evaluated according to the overlap length between the position information corresponding to the recalled document fragment and the position information corresponding to the annotated document fragment.

[0010] In a possible implementation, the characteristics of the annotated document segment include the length of the annotated document segment and the density of professional terms; the target similarity threshold is determined by the following steps:

[0011] If the length of the annotated document segment is greater than a preset length or the density of professional terms is greater than a preset density, the value corresponding to the general similarity threshold is increased to obtain a target similarity threshold.

[0012] In a possible implementation, the evaluation index of the recall result includes the position information recall rate, the position information accuracy, the position information average accuracy rate and the position information average reciprocal ranking; the recall result of the target knowledge question answering system is evaluated according to the overlap length between the position information corresponding to the recalled document fragment and the position information corresponding to the annotated document fragment, including:

[0013] Calculate the coverage between the recalled document fragment and the annotated document fragment according to the overlap length between the position information corresponding to the recalled document fragment and the position information corresponding to the annotated document fragment;

[0014] According to the coverage between the recalled document fragments and the annotated document fragments, the location information recall rate, location information precision, location information average accuracy rate and location information average reciprocal ranking are calculated.

[0015] In a possible implementation, the evaluation index of the recall result further includes the normalized discounted cumulative gain of the position information; and the recall result of the target knowledge question answering system is evaluated according to the overlap length between the position information corresponding to the recalled document fragment and the position information corresponding to the annotated document fragment, further including:

[0016] Calculate the correlation coefficient between the recalled document segment and each annotated document segment according to the overlap length between the position information corresponding to the recalled document segment and the position information corresponding to each annotated document segment and the length of the position information corresponding to each annotated document segment;

[0017] All recalled document fragments are arranged in descending order based on the maximum value of the correlation coefficient to obtain the ideal ranking of the recalled document fragments; and the ranking obtained by the target knowledge question answering system based on the importance of the answers corresponding to the target user's questions generated by the recalled document fragments is determined as the actual ranking of the recalled document fragments;

[0018] The actual loss cumulative gain is calculated based on the maximum value of the actual ranking and relevance coefficient corresponding to all recalled document fragments; and the ideal loss cumulative gain is calculated based on the maximum value of the ideal ranking and relevance coefficient corresponding to all recalled document fragments;

[0019] The ratio of the actual impairment cumulative gain to the ideal impairment cumulative gain is calculated to obtain the normalized impairment cumulative gain of the position information.

[0020] In a possible implementation, calculating the actual impairment cumulative gain or the ideal impairment cumulative gain includes:

[0021] Substitute the maximum value of the correlation coefficients corresponding to all recalled document fragments and the actual ranking or ideal ranking corresponding to all recalled document fragments into the following formula to obtain the actual depreciation cumulative gain or the ideal depreciation cumulative gain;

[0022]

[0023] Where DCG is the actual cumulative loss gain or the ideal cumulative loss gain, n is the number of recalled document fragments, is the maximum value of the relevance coefficient of the recalled document fragments with actual ranking or ideal ranking i.

[0024] In a possible implementation, the method further includes:

[0025] Obtain the rejection rate and wrong answer rate of the target knowledge question answering system for negative sample questions; negative sample questions refer to questions that are irrelevant to the original documents used to generate answers in the target knowledge question answering system;

[0026] According to the rejection answer rate and the answer error rate, the anti-interference ability index of the target knowledge question answering system is calculated.

[0027] In a possible implementation, before determining the position information of the node in the document tree whose similarity to the annotated document fragment is greater than a target similarity threshold and has the highest similarity as the position information corresponding to the annotated document fragment, the method further includes:

[0028] Vectorize the original document fragments and the annotated document fragments obtained by splitting the original document to obtain the original document fragment vector and the annotated document fragment vector;

[0029] Calculate the similarity between the original document fragment vector and the annotated document fragment vector;

[0030] If the similarity is greater than the preset similarity, the original document fragment corresponding to the original document fragment vector is added to the annotated document fragment.

[0031] In a second aspect, an embodiment of the present application further provides an evaluation and optimization device for a recall result of a knowledge question answering system, and the evaluation and optimization device for a recall result of a knowledge question answering system comprises:

[0032] The acquisition module is used to obtain the annotated document fragments and recall results corresponding to the target user's question; the annotated document fragment refers to the document fragment containing the standard answer corresponding to the target user's question; the recall result includes the recalled document fragments of all potential answers retrieved by the target knowledge question answering system in the original document for the target user's question; the recalled document fragment carries its position information in the original document;

[0033] The splitting module is used to construct a document tree by using the document fragments obtained by splitting the original document based on each fine-grained level as nodes; each node in the document tree carries its position information in the original document;

[0034] A determination module is used to determine the position information of the node in the document tree whose similarity with the annotated document fragment is greater than a target similarity threshold and has the highest similarity as the position information corresponding to the annotated document fragment; the target similarity threshold is obtained by adjusting the general similarity threshold based on the characteristics of the annotated document fragment itself;

[0035] The evaluation module is used to evaluate the recall result of the target knowledge question answering system according to the overlap length between the position information corresponding to the recalled document fragment and the position information corresponding to the annotated document fragment.

[0036] In the third aspect, an embodiment of the present application also provides an electronic device, comprising: a processor, a storage medium and a bus, the storage medium storing machine-readable instructions executable by the processor, when the electronic device is running, the processor and the storage medium communicate through the bus, and the processor executes the machine-readable instructions to perform the steps of the method for evaluating and optimizing the recall results of the knowledge question and answer system as described in any one of the first aspects.

[0037] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method for evaluating and optimizing the recall results of the knowledge question and answer system as described in any one of the first aspects are executed.

[0038] The embodiment of the present application provides an evaluation optimization method, device, equipment and medium for the recall result of a knowledge question and answer system, the method comprising: obtaining annotated document fragments and recall results corresponding to the target user's question; constructing a document tree using the document fragments obtained by splitting the original document based on each fine-grained level as nodes; each node in the document tree carries its position information in the original document; the position information of the node in the document tree whose similarity with the annotated document fragment is greater than the target similarity threshold and has the highest similarity is determined as the position information corresponding to the annotated document fragment; the target similarity threshold is obtained by adjusting the general similarity threshold based on the characteristics of the annotated document fragment itself; the recall result of the target knowledge question and answer system is evaluated according to the overlap length between the position information corresponding to the recalled document fragment and the position information corresponding to the annotated document fragment. The present application determines the position information corresponding to the annotated document fragment by constructing a document tree based on the document fragments obtained by splitting based on each fine-grained level, and then performs a more fine-grained evaluation on the recall result of the knowledge question and answer system based on the position information corresponding to the recalled document fragment and the annotated document fragment, thereby improving the optimization quality of the knowledge base question and answer system. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 A flowchart of a method for evaluating and optimizing recall results of a knowledge question answering system provided in an embodiment of the present application is shown;

[0041] Figure 2 A flowchart of another method for evaluating and optimizing the recall results of a knowledge question answering system provided in an embodiment of the present application is shown;

[0042] Figure 3 A schematic diagram of the structure of a device for evaluating and optimizing the recall results of a knowledge question-answering system provided in an embodiment of the present application is shown;

[0043] Figure 4 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0044] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0045] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0046] In order to enable those skilled in the art to use the content of this application, the following implementation is provided in conjunction with the specific application scenario "field of natural language processing". For those skilled in the art, the general principles defined herein can be applied to other embodiments and application scenarios without departing from the spirit and scope of this application. Although this application is mainly described around the "field of natural language processing", it should be understood that this is only an exemplary embodiment.

[0047] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0048] The following is a detailed description of a method for evaluating and optimizing the recall results of a knowledge question-answering system provided in an embodiment of the present application.

[0049] Reference Figure 1 As shown, it is a flow chart of a method for evaluating and optimizing the recall results of a knowledge question answering system provided in an embodiment of the present application. The exemplary steps of the embodiment of the present application are described below:

[0050] S101, obtaining the annotated document fragments and recall results corresponding to the target user's question.

[0051] In the implementation manner of the present application, the target knowledge question and answer system refers to a knowledge question and answer system that needs to evaluate the recall results. The target user questions refer to the questions asked by the user when using the target knowledge question and answer system; the target knowledge question and answer system is pre-set with a knowledge base, which contains a large number of original documents; the target knowledge question and answer system can retrieve document fragments related to the user questions from these original documents, and then generate answers to the user questions based on the retrieved document fragments. The annotated document fragment refers to a document fragment containing the standard answer corresponding to the target user question; the recall result includes the recalled document fragments of all potential answers retrieved by the target knowledge question and answer system for the target user question in the original document, and the recalled answer generated by the target knowledge question and answer system based on the recalled document fragment; the recalled document fragment carries its position information in the original document.

[0052] Here, when evaluating the recall results of the knowledge question-answering system, the traditional evaluation optimization method manually annotates the annotated document fragments corresponding to the user questions in the original documents. However, there are many original documents in the knowledge base, and the manual annotation method usually only covers part of the documents in the knowledge base. The correct answers may also exist in the unannotated original documents, resulting in a low recall rate. Therefore, in addition to manual annotation by the annotation team or business department, the embodiment of the present application can also generate more user questions and corresponding annotated document fragments through the annotation large model. Specifically, all original documents and target user questions are input into the annotation large model to obtain annotated document fragments and standard answers corresponding to other user questions related to the content in the original document and related to the target user questions; and other user questions are also added to the target user questions to enrich the number of user questions and the coverage of the annotated document fragments.

[0053] Among them, target user questions, annotated document fragments and recalled document fragments can be divided into structured data (such as records in tabular form) and unstructured data (such as text conversations). All data must be cleaned and normalized before application: (1) Noise data cleaning: delete blank records, duplicate records and conversation fragments not related to financial business. (2) Unified data format: ensure that data from different sources are unified in format, and timestamps, currency units, etc. should be standardized; for example, use a unified date format YYYY-MM-DD. (3) Evaluation of the quality of annotated document fragments: conduct a preliminary inspection of existing annotated document fragments through large model classification, screen out low-quality annotated samples, and feedback to the annotation team or business department for further correction or supplement of these samples; among them, the input of the large model classification is user questions and annotated document fragments, and the output is the quality of the annotated document fragments (such as qualified, unqualified, etc.).

[0054] S102: Using document fragments obtained by splitting the original document based on each fine-grained level as nodes to construct a document tree; each node in the document tree carries its position information in the original document.

[0055] In the implementation mode of the present application, a natural language processing tool splitter class is customized to ensure that the split document fragments have logical independence. This class splits the split document fragments of the i-1th fine-grained level according to the i-th fine-grained level through a customized regular expression and manually set splitting rules to obtain the split document fragments of the i-th fine-grained level; the initial value of i is 1; the split document fragments of the 0th fine-grained level are the original document; after the split is completed, i=i+1, and jumps to the i-th fine-grained level to split the document fragments obtained by splitting according to the i-1th fine-grained level and continue to execute until the division according to all fine-grained levels is completed; the higher the level of fine-grainedness, the finer the split. The split document fragments are used as nodes to build a document tree; each original document will be built into a document tree. In the document tree, the original document is the root node, and each node except the root node is contained in its corresponding parent node; the document fragments obtained by splitting at the same fine-grained level are at the same level in the document tree. Constructing a document tree can help determine the level and position of the annotated document fragment in the original document later.

[0056] Among them, regular expressions refer to general rules used to split original documents into various fine-grained levels; each level corresponds to its own regular expression; the manually set splitting rules are determined based on the business scenarios involved in the original documents in the knowledge base; for example, in financial dialogue scenarios, there may be many text description features that are different from other fields, so the splitting can be further limited by splitting rules based on regular expressions to improve the splitting effect.

[0057] Specifically, according to specific scenarios and data characteristics, some customized splitting standards and logics ensure that the splitting process is more accurate and in line with specific business scenarios. Specific punctuation segmentation: Punctuation marks such as periods, question marks, exclamation marks, and semicolons are common segmentation points. Ignore punctuation in certain specific locations. For example, abbreviations in English (such as "Dr." and "US") will not trigger segmentation. Paragraphs and line breaks: Rules can include paragraph or line break processing; for example, double line breaks are used to split paragraphs, and single line breaks are used to split sentences. When processing multi-line content, it is necessary to force segmentation at line breaks to ensure logical integrity. Identification of titles and subheadings: For texts containing titles (such as documents or reports), use titles or subheadings as segmentation points to ensure that each segmented segment is a complete content segment (by detecting keywords such as "Chapter X" or "Section X" to segment the text into specific chapters or subsections). Special character or keyword processing: Some technical documents or reports may contain special characters (such as "###", "--", etc.) to segment content; some common keywords (such as "summary", "conclusion") may indicate the end of a paragraph or the beginning of a new paragraph, and also serve as segmentation points.

[0058] Optionally, in order to avoid the problem that the annotated document fragment only covers part of the original document, while the unannotated original document contains correct answers in fragments other than the annotated document fragment, thereby causing a low recall rate, before S103, determining the position information of the node in the document tree whose similarity with the annotated document fragment is greater than the target similarity threshold and has the highest similarity, as the position information corresponding to the annotated document fragment, the method further includes:

[0059] Step 1: vectorize the original document fragments and the annotated document fragments obtained by splitting the original document respectively to obtain the original document fragment vector and the annotated document fragment vector.

[0060] In the implementation manner of the present application, the original document fragment refers to the document fragment obtained after the original document is split into multiple levels in S102; the original document fragment is vectorized using a large model such as BERT to obtain an original document fragment vector; the annotated document fragment is vectorized using a large model such as BERT to obtain an annotated document fragment vector.

[0061] Step 2: Calculate the similarity between the original document fragment vector and the annotated document fragment vector.

[0062] Step 3: If the similarity is greater than a preset similarity, the original document fragment corresponding to the original document fragment vector is added to the annotated document fragment.

[0063] In an implementation manner of the present application, the preset similarity can be set to 0.85. If the similarity is greater than 0.85, the original document fragment corresponding to the original document fragment vector is also used as an annotated document fragment to avoid the problem of low recall rate due to the annotated document fragment only covering the part of the original document that contains the standard answer.

[0064] S103: Determine the position information of the node in the document tree whose similarity to the annotated document fragment is greater than the target similarity threshold and has the highest similarity as the position information corresponding to the annotated document fragment.

[0065] In the implementation mode of the present application, the similarity between the annotated document fragment and each node in the document tree is calculated using algorithms such as cosine similarity or Jaccard similarity; the greater the similarity, the better the match; the document in which the document fragment is located and its position in the document are taken into account when calculating the similarity; for example, if the two document fragments are no longer in the same document, the similarity is 0; the position information of the node with the highest similarity and a similarity greater than the similarity threshold corresponding to the annotated document fragment is determined as the position information corresponding to the annotated document fragment. The target similarity threshold is obtained by adjusting the general similarity threshold based on the inherent characteristics of the annotated document fragment; the inherent characteristics of the annotated document fragment include the length of the annotated document fragment and the density of professional terms.

[0066] Here, the document tree has the following functions in the process of determining the location information corresponding to the annotated document fragment: (1) Hierarchical positioning: The tree structure allows the various parts of the original document to be located hierarchically. This hierarchical structure helps to quickly find the exact location of the annotated document fragment in the entire original document. (2) Fast indexing: Each tree node contains the location information of its own fragment in the original document. When the annotated document fragment matches the node, its location information can be directly read from the node, which greatly improves the speed and accuracy of generating location information.

[0067] Further, the target similarity threshold is determined by the following steps: if the length of the annotated document segment is greater than a preset length or the density of professional terms is greater than a preset density, the value corresponding to the general similarity threshold is increased to obtain the target similarity threshold; otherwise, the general similarity threshold is determined as the target similarity threshold. The general similarity threshold can be increased by a preset increase value, or by a preset increase ratio, or by a difference between the length of the annotated document segment and a preset length, or between the density of professional terms and a preset density, and the specific method can be determined according to the actual situation.

[0068] Here, the longer the annotated document fragment is, the more likely it is to contain multiple topics, so a higher threshold is needed to filter out irrelevant content. The greater the density of professional terms and the more complex the annotated document fragment is, the higher the similarity threshold is needed to ensure the accuracy of the recalled fragment. If the document is long, increase the threshold to avoid recalling too many irrelevant fragments. In complex content, raising the threshold can ensure that the fragments are highly relevant.

[0069] Optionally, when determining the target similarity threshold, in addition to considering the characteristics of the annotated document fragment itself, the answer requirement corresponding to the target user's question can also be considered; if the answer requirement corresponding to the target user's question is a precise answer, then the value corresponding to the general similarity threshold is increased to obtain the target similarity threshold; if the answer requirement corresponding to the target user's question is a broad answer, then the value corresponding to the general similarity threshold is reduced to obtain the target similarity threshold; if none of the above conditions are met, then the general similarity threshold is determined as the target similarity threshold. The general similarity threshold can be reduced by a preset reduction value, or by a preset reduction ratio.

[0070] Optionally, the embodiment of the present application can also adjust the general similarity threshold, including: counting the number of annotated document fragments matched by the general similarity threshold to obtain a first number; counting the number of annotated document fragments successfully matched by the general similarity threshold to obtain a second number; determining the ratio of the second number to the first number as the matching success rate of the general similarity threshold; if the matching success rate is less than the preset minimum matching success rate value, it is considered that the general similarity threshold is too high, then reducing the value corresponding to the general similarity threshold to obtain the latest general similarity threshold, so as to expand the scope of recall and allow more relevant fragments to enter the recall results. The initial general similarity threshold is set based on experience.

[0071] S104. Evaluate the recall result of the target knowledge question answering system according to the overlap length between the position information corresponding to the recalled document fragment and the position information corresponding to the annotated document fragment.

[0072] In the implementation manner of the present application, the evaluation indicators of the recall result include the location information recall rate, the location information precision, the average accuracy rate of the location information and the average reciprocal ranking of the location information.

[0073] Among them, the position information recall rate is used to measure the proportion of successfully finding the standard answer in the recalled document fragments. The position information precision is used to measure the proportion of document fragments that are actually related to the standard answer in all recalled document fragments. The average accuracy of the position information is used to measure the accuracy and rationality of the ranking of the recalled document fragments by the target knowledge question and answer system RAG. The average reciprocal ranking of the position information is used to measure the effectiveness of the ranking of the recalled document fragments by the target knowledge question and answer system RAG.

[0074] Reference Figure 2 As shown, it is a flow chart of another method for evaluating and optimizing the recall results of a knowledge question answering system provided in an embodiment of the present application. The exemplary steps of the embodiment of the present application are described below:

[0075] S201. Calculate the coverage between the recalled document segment and the annotated document segment according to the overlap length between the position information corresponding to the recalled document segment and the position information corresponding to the annotated document segment.

[0076] In the implementation mode of the present application, the coverage IoU between the recalled document fragment and the annotated document fragment is calculated by the following formula:

[0077]

[0078] Among them, L1 is the overlapping length between the position information corresponding to the recalled document fragment and the position information corresponding to the annotated document fragment, and L2 is the total covering length of the position information corresponding to the recalled document fragment and the position information corresponding to the annotated document fragment.

[0079] For example, the position information corresponding to the recalled document fragment is 3-5, and the position information corresponding to the annotated document fragment is 4-10, then the overlapping length is 5-4=1, and the total coverage length is 10-3=7.

[0080] S202. Calculate the position information recall rate, position information precision, position information average accuracy rate and position information average reciprocal ranking according to the coverage rate between the recalled document fragments and the annotated document fragments.

[0081] Specifically, the location information recall rate Recall is calculated by the following formula:

[0082]

[0083] Wherein, D1 is the number of annotated document segments whose coverage is greater than a preset coverage threshold, and D2 is the total number of annotated document segments.

[0084] Specifically, the position information accuracy Precision is calculated by the following formula:

[0085]

[0086] Among them, D3 is the number of recalled document fragments whose coverage is greater than the preset coverage threshold, and D4 is the total number of recalled document fragments.

[0087] Specifically, the average accuracy mAP of the position information is calculated by the following formula:

[0088]

[0089] Among them, coverage_weight is the maximum coverage ratio between each recalled document fragment and the annotated document fragment, and the actual ranking of the recalled document fragment refers to the ranking obtained by the target knowledge question-answering system based on the importance of the answers corresponding to the target user's questions generated by the recalled document fragments.

[0090] Specifically, the average reciprocal ranking MRR of the location information is calculated by the following formula:

[0091]

[0092] Furthermore, the evaluation index of the recall result also includes the normalized cumulative gain of position information; the normalized cumulative gain of position information is used to measure the effect of sorting the recalled document fragments, ensuring that the recalled documents with high relevance are recalled first. Specifically, the recall result of the target knowledge question answering system is evaluated according to the overlap length between the position information corresponding to the recalled document fragment and the position information corresponding to the annotated document fragment, and also includes:

[0093] Step 1: Calculate the correlation coefficient between the recalled document fragment and each annotated document fragment based on the overlap length between the position information corresponding to the recalled document fragment and the position information corresponding to each annotated document fragment and the length of the position information corresponding to each annotated document fragment.

[0094] In the implementation manner of the present application, the correlation coefficient relevance_score between the recalled document fragment and the annotated document fragment is calculated by the following formula:

[0095]

[0096] Among them, L3 is the length of the position information corresponding to the annotated document fragment.

[0097] Step 2: Arrange all recalled document fragments in descending order based on the maximum value of the correlation coefficient to obtain the ideal ranking of the recalled document fragments; and determine the ranking obtained by the target knowledge question-answering system based on the importance of the answers corresponding to the target user's questions generated by the recalled document fragments as the actual ranking of the recalled document fragments.

[0098] Step 3: Calculate the actual loss cumulative gain based on the actual ranking corresponding to all recalled document fragments and the maximum value of the correlation coefficient; and calculate the ideal loss cumulative gain based on the ideal ranking corresponding to all recalled document fragments and the maximum value of the correlation coefficient.

[0099] In the implementation manner of the present application, the maximum value of the correlation coefficients corresponding to all recalled document fragments and the actual ranking or ideal ranking corresponding to all recalled document fragments are substituted into the following formula to obtain the actual impairment cumulative gain or the ideal impairment cumulative gain;

[0100]

[0101] Where DCG is the actual cumulative loss gain or the ideal cumulative loss gain, n is the number of recalled document fragments, is the maximum value of the relevance coefficient of the recalled document fragments with actual ranking or ideal ranking i.

[0102] Step 4: Calculate the ratio of the actual cumulative loss gain to the ideal cumulative loss gain to obtain the normalized cumulative loss gain of the position information.

[0103] Furthermore, in the embodiment of the present application, in addition to calculating the evaluation indicators of the target knowledge question-and-answer system based on the position information, it also includes evaluation indicators based on the answer; the evaluation indicators based on the answer include the Rouge-L score and the large model score. The Rouge-L score is calculated by comparing the recalled answers and standard answers of the RAG system to evaluate the semantic similarity and completeness of the answers. This indicator is based on the longest common subsequence and can measure the accuracy of the answers of the RAG system; the specific process of calculating the Rouge-L score includes: using the jieba word segmentation library to segment the sentence; using the rouge_chinese library to calculate the Rouge-L score, and the recall indicator is returned. The recalled answer refers to the answer generated by the RAG system based on the target user's question.

[0104] Among them, the big model score is an indicator for evaluating the recalled answers based on the designed big model scoring system, focusing on analyzing the semantic consistency and logical rationality of the recalled answers. The big model score will provide further in-depth analysis for answer evaluation; specifically, it includes the following contents:

[0105] (1) Context Relevance Evaluation: Context Recall is used to evaluate whether the context of the recalled answer contains the standard answer. Context Relevance is used to calculate the semantic relevance between the context of the recalled answer and the target user's question, and to determine whether the context provides effective support information for the target user's question. Answer Semantic Similarity is used to calculate the semantic similarity between the recalled answer and the standard answer to ensure that the recalled answer not only matches the surface vocabulary, but also meets the requirements in terms of semantics.

[0106] (2) Large model scoring and context re-ranking (DeepEval / UpTrain): Context re-ranking is used to re-rank the context fragments of the recalled answers based on the deep understanding ability of the large model to ensure that the most relevant context is given priority. Through context re-ranking, the recall ability of the RAG system can be enhanced. Answer accuracy evaluation is used to combine the recalled answers and the standard answers. The system evaluates the accuracy of the recalled answers through a multi-dimensional scoring framework and gives a comprehensive score of semantic correctness and grammatical fluency.

[0107] Here, the input for calculating the big model score includes the standard answer, the annotated document fragment, the recalled answer, and the recalled document fragment, and the output is all the contents of the big model score. The answer generated by the RAG system needs to be consistent with the standard answer not only on the surface, but also on the semantic level. However, existing evaluation frameworks usually focus on N-gram-based text matching (such as BLEU and ROUGE). These indicators can only measure the matching of vocabulary and cannot effectively evaluate the semantic correctness of the recalled answers. This causes the generative model to generate logically unreasonable or semantically inaccurate answers in complex problem scenarios, and the existing evaluation framework is less sensitive to this. The embodiment of the present application introduces a big model score, which can effectively combine the context for semantic evaluation and fully evaluate the quality of the recalled answer.

[0108] Furthermore, the embodiment of the present application also introduces a negative sample problem. By introducing negative samples, the system's ability to respond to irrelevant or noisy questions is tested to evaluate whether it can refuse to answer or avoid generating wrong answers. Specifically:

[0109] Step 1: Obtain the rejection rate and error rate of negative sample questions of the target knowledge question answering system; negative sample questions refer to questions that are irrelevant to the original documents used to generate answers in the target knowledge question answering system.

[0110] Step 2: Calculate the anti-interference ability index of the target knowledge question answering system based on the refusal to answer rate and the answer error rate.

[0111] In the implementation mode of the present application, the rejection answer rate and the answer error rate are substituted into the following formula to calculate the anti-interference ability index of the target knowledge question answering system;

[0112] RI = α × RR - β × ER;

[0113] Among them, RI is the anti-interference ability index, α is the weight corresponding to the refusal to answer rate, RR is the refusal to answer rate, β is the weight corresponding to the wrong answer rate, and β is the wrong answer rate.

[0114] In the implementation of the present application, the higher the RI value, the more effective the system is in rejecting wrong answers and has a stronger anti-interference ability. The lower the RI value, the more susceptible the system is to negative samples, generating wrong answers or failing to reject answers.

[0115] Here, in actual application scenarios, the knowledge base may contain a large amount of irrelevant or noisy data. How to effectively evaluate the performance of the system when processing these noisy data and ensure that the RAG system can still generate high-quality answers in this case is a problem that needs to be solved urgently. Therefore, the embodiment of the present application introduces an anti-interference index to improve the robustness of the evaluation system to noisy data.

[0116] Furthermore, all the evaluation indicators calculated will provide a clear direction for the subsequent RAG system optimization. Based on the feedback, the recall algorithm can be appropriately adjusted, such as strengthening the recall priority of a certain type of document, or adjusting the system's recall strategy to improve the overall performance of the system.

[0117] The present application provides an evaluation and optimization method for the recall results of a knowledge question-answering system, the method comprising: obtaining annotated document fragments and recall results corresponding to the target user's questions; constructing a document tree using the document fragments obtained by splitting the original document based on each fine-grained level as nodes; each node in the document tree carries its position information in the original document; the position information of the node in the document tree whose similarity with the annotated document fragment is greater than the target similarity threshold and has the highest similarity is determined as the position information corresponding to the annotated document fragment; the target similarity threshold is obtained by adjusting the general similarity threshold based on the characteristics of the annotated document fragment itself; and the recall results of the target knowledge question-answering system are evaluated according to the overlap length between the position information corresponding to the recalled document fragment and the position information corresponding to the annotated document fragment. The present application determines the position information corresponding to the annotated document fragment by constructing a document tree using the document fragments obtained by splitting based on each fine-grained level, and then performs a more fine-grained evaluation of the recall results of the knowledge question-answering system based on the position information corresponding to the recalled document fragment and the annotated document fragment, thereby improving the optimization quality of the knowledge base question-answering system.

[0118] Based on the same inventive concept, an embodiment of the present application also provides an evaluation and optimization device for the recall results of a knowledge question and answer system corresponding to the evaluation and optimization method for the recall results of a knowledge question and answer system. Since the principle of solving the problem by the device in the embodiment of the present application is similar to the evaluation and optimization method for the recall results of the knowledge question and answer system in the embodiment of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0119] Reference Figure 3 FIG. 1 is a schematic diagram of a device for evaluating and optimizing the recall results of a knowledge question answering system provided in an embodiment of the present application. The device for evaluating and optimizing the recall results of the knowledge question answering system includes:

[0120] The acquisition module 301 is used to obtain the annotated document fragments and the recall results corresponding to the target user's question; the annotated document fragments refer to the document fragments containing the standard answers corresponding to the target user's question; the recall results include the recalled document fragments of all potential answers retrieved by the target knowledge question answering system in the original document for the target user's question; the recalled document fragments carry their position information in the original document;

[0121] The splitting module 302 is used to construct a document tree by using the document fragments obtained by splitting the original document based on each fine-grained level as nodes; each node in the document tree carries its position information in the original document;

[0122] The determination module 303 is used to determine the position information of the node in the document tree whose similarity with the annotated document fragment is greater than the target similarity threshold and has the highest similarity as the position information corresponding to the annotated document fragment; the target similarity threshold is obtained by adjusting the general similarity threshold based on the characteristics of the annotated document fragment itself;

[0123] The evaluation module 304 is used to annotate the document fragment pairs according to the position information corresponding to the recalled document fragments.

[0124] In a possible implementation, the characteristics of the annotated document segment include the length of the annotated document segment and the density of professional terms; the determination module 303 is further used to:

[0125] If the length of the annotated document segment is greater than a preset length or the density of professional terms is greater than a preset density, the value corresponding to the general similarity threshold is increased to obtain a target similarity threshold.

[0126] In one possible implementation, the evaluation indicators of the recall result include the position information recall rate, the position information precision, the position information average accuracy rate and the position information average reciprocal ranking; the evaluation module 304 is specifically used to calculate the coverage rate between the recalled document fragment and the annotated document fragment based on the overlapping length between the position information corresponding to the recalled document fragment and the position information corresponding to the annotated document fragment; and calculate the position information recall rate, the position information precision, the position information average accuracy rate and the position information average reciprocal ranking based on the coverage rate between the recalled document fragment and the annotated document fragment.

[0127] In a possible implementation, the evaluation index of the recall result also includes the normalized cumulative loss gain of position information; the evaluation module 304 is specifically used to calculate the correlation coefficient between the recalled document fragment and each annotated document fragment based on the overlapping length between the position information corresponding to the recalled document fragment and the position information corresponding to each annotated document fragment, and the length of the position information corresponding to each annotated document fragment; all the recalled document fragments are arranged in descending order based on the maximum value of the correlation coefficient to obtain the ideal ranking of the recalled document fragments; and the ranking obtained by the target knowledge question and answer system based on the importance of the answer corresponding to the target user's question generated by the recalled document fragment is determined as the actual ranking of the recalled document fragment; the actual cumulative loss gain is calculated based on the actual ranking corresponding to all the recalled document fragments and the maximum value of the correlation coefficient; and the ideal cumulative loss gain is calculated based on the ideal ranking corresponding to all the recalled document fragments and the maximum value of the correlation coefficient; the ratio of the actual cumulative loss gain to the ideal cumulative loss gain is calculated to obtain the normalized cumulative loss gain of position information.

[0128] In a possible implementation, the evaluation module 304 is further configured to:

[0129] Substituting the maximum value of the correlation coefficients corresponding to all the recalled document fragments and the actual ranking or the ideal ranking corresponding to all the recalled document fragments into the following formula to obtain the actual depreciation cumulative gain or the ideal depreciation cumulative gain;

[0130]

[0131] Where DCG is the actual cumulative loss gain or the ideal cumulative loss gain, n is the number of recalled document fragments, is the maximum value of the relevance coefficient of the recalled document fragments with actual ranking or ideal ranking i.

[0132] In a possible implementation, the acquisition module 301 is further used to obtain the rejection rate and the wrong answer rate of the target knowledge question answering system for negative sample questions; negative sample questions refer to questions that are irrelevant to the original document used to generate answers in the target knowledge question answering system;

[0133] The evaluation module 304 is further used to calculate the anti-interference capability index of the target knowledge question answering system according to the rejection answer rate and the answer error rate.

[0134] In a possible implementation, the device further includes: a vectorization module 305, a calculation module 306, and an adding module 307;

[0135] The vectorization module 305 is used to vectorize the original document segments and the annotated document segments obtained by splitting the original document, respectively, before determining the position information of the node in the document tree whose similarity with the annotated document segment is greater than the target similarity threshold and has the highest similarity as the position information corresponding to the annotated document segment, to obtain the original document segment vector and the annotated document segment vector;

[0136] A calculation module 306, used to calculate the similarity between the original document segment vector and the annotated document segment vector;

[0137] The adding module 307 is used to add the original document segment corresponding to the original document segment vector to the marked document segment if the similarity is greater than a preset similarity.

[0138] The present application provides an evaluation and optimization device for the recall results of a knowledge question and answer system. The device determines the position information corresponding to the annotated document fragments by constructing a document tree based on the document fragments obtained by splitting at each fine-grained level, and then performs a more fine-grained evaluation on the recall results of the knowledge question and answer system based on the position information corresponding to the recalled document fragments and the annotated document fragments, thereby improving the optimization quality of the knowledge base question and answer system.

[0139] like Figure 4 As shown, an electronic device 400 provided in an embodiment of the present application includes: a processor 401, a memory 402 and a bus, the memory 402 stores machine-readable instructions executable by the processor 401, when the electronic device is running, the processor 401 and the memory 402 communicate through the bus, and the processor 401 executes the machine-readable instructions to perform the steps of the evaluation optimization method of the recall results of the knowledge question and answer system as mentioned above.

[0140] Specifically, the above-mentioned memory 402 and processor 401 can be general-purpose memories and processors, which are not specifically limited here. When the processor 401 runs the computer program stored in the memory 402, it can execute the evaluation optimization method of the recall results of the above-mentioned knowledge question and answer system.

[0141] Corresponding to the evaluation and optimization method of the recall results of the above-mentioned knowledge question and answer system, an embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the evaluation and optimization method of the recall results of the above-mentioned knowledge question and answer system are executed.

[0142] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0143] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0144] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0145] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the information processing method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0146] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for evaluating and optimizing the recall results of a knowledge question answering system, characterized in that: The method comprises: Obtaining an annotated document fragment and a recall result corresponding to the target user's question; the annotated document fragment refers to a document fragment containing a standard answer corresponding to the target user's question; the recall result includes the recalled document fragments of all potential answers retrieved by the target knowledge question-answering system in the original document for the target user's question; the recalled document fragment carries its position information in the original document; The document fragments obtained by splitting the original document based on each fine-grained level are used as nodes to construct a document tree; each node in the document tree carries its position information in the original document; The position information of the node in the document tree whose similarity with the annotated document fragment is greater than the target similarity threshold and has the highest similarity is determined as the position information corresponding to the annotated document fragment; the target similarity threshold is obtained by adjusting the general similarity threshold based on the characteristics of the annotated document fragment itself; The recall result of the target knowledge question answering system is evaluated according to the overlap length between the position information corresponding to the recalled document fragment and the position information corresponding to the annotated document fragment.

2. The method for evaluating and optimizing the recall results of a knowledge question answering system according to claim 1, characterized in that: The characteristics of the annotated document segment include the length of the annotated document segment and the density of professional terms. The target similarity threshold is determined by the following steps: If the length of the annotated document segment is greater than a preset length or the density of the professional terms is greater than a preset density, the value corresponding to the general similarity threshold is increased to obtain the target similarity threshold.

3. The method for evaluating and optimizing the recall results of the knowledge question answering system according to claim 2, characterized in that: The evaluation indicators of the recall result include the position information recall rate, the position information accuracy, the position information average accuracy rate and the position information average reciprocal ranking; the recall result of the target knowledge question answering system is evaluated according to the overlap length between the position information corresponding to the recalled document fragment and the position information corresponding to the annotated document fragment, including: Calculating the coverage between the recalled document segment and the annotated document segment according to the overlap length between the position information corresponding to the recalled document segment and the position information corresponding to the annotated document segment; According to the coverage between the recalled document fragment and the annotated document fragment, the position information recall rate, the position information precision, the position information average accuracy rate and the position information average reciprocal ranking are calculated.

4. The method for evaluating and optimizing the recall results of a knowledge question answering system according to claim 2, characterized in that: The evaluation index of the recall result also includes the normalized discounted cumulative gain of position information; the recall result of the target knowledge question answering system is evaluated according to the overlap length between the position information corresponding to the recalled document fragment and the position information corresponding to the annotated document fragment, and further includes: Calculating a correlation coefficient between the recalled document segment and each of the annotated document segments according to an overlap length between the position information corresponding to the recalled document segment and the position information corresponding to each of the annotated document segments and a length of the position information corresponding to each of the annotated document segments; Arrange all the recalled document fragments in descending order based on the maximum value of the correlation coefficient to obtain the ideal ranking of the recalled document fragments; and determine the ranking obtained by the target knowledge question-answering system based on the importance of the answer corresponding to the target user's question generated by the recalled document fragments as the actual ranking of the recalled document fragments; Calculating the actual depreciation cumulative gain according to the actual ranking corresponding to all the recalled document fragments and the maximum value of the correlation coefficient; and calculating the ideal depreciation cumulative gain according to the ideal ranking corresponding to all the recalled document fragments and the maximum value of the correlation coefficient; The ratio of the actual loss cumulative gain to the ideal loss cumulative gain is calculated to obtain the position information normalized loss cumulative gain.

5. The method for evaluating and optimizing the recall results of the knowledge question answering system according to claim 4, characterized in that: Calculating the actual loss accumulation gain or the ideal loss accumulation gain includes: Substituting the maximum value of the correlation coefficients corresponding to all the recalled document fragments and the actual ranking or the ideal ranking corresponding to all the recalled document fragments into the following formula to obtain the actual depreciation cumulative gain or the ideal depreciation cumulative gain; Wherein, DCG is the actual discounted cumulative gain or the ideal discounted cumulative gain, n is the number of recalled document fragments, is the maximum value of the relevance coefficients of the recalled document fragments whose actual ranking or ideal ranking is i.

6. The method for evaluating and optimizing the recall results of a knowledge question answering system according to any one of claims 1 to 5, characterized in that: The method further comprises: Obtaining a rejection rate and an error rate of answers for negative sample questions in the target knowledge question answering system; the negative sample questions refer to questions that are unrelated to the original documents used to generate answers in the target knowledge question answering system; According to the refusal to answer rate and the answer error rate, an anti-interference capability index of the target knowledge question answering system is calculated.

7. The method for evaluating and optimizing the recall results of a knowledge question answering system according to any one of claims 1 to 5, characterized in that: Before determining the position information of the node in the document tree whose similarity to the annotated document fragment is greater than a target similarity threshold and has the highest similarity as the position information corresponding to the annotated document fragment, the method further includes: Vectorizing the original document segments and the annotated document segments obtained by splitting the original document to obtain original document segment vectors and annotated document segment vectors; Calculating the similarity between the original document segment vector and the annotated document segment vector; If the similarity is greater than a preset similarity, the original document segment corresponding to the original document segment vector is added to the annotated document segment.

8. A device for evaluating and optimizing the recall results of a knowledge question answering system, characterized in that: The device comprises: An acquisition module is used to acquire the annotated document fragment and the recall result corresponding to the target user's question; the annotated document fragment refers to the document fragment containing the standard answer corresponding to the target user's question; the recall result includes the recalled document fragments of all potential answers retrieved by the target knowledge question-answering system in the original document for the target user's question; the recalled document fragment carries its position information in the original document; A splitting module, used to construct a document tree using the document fragments obtained by splitting the original document based on each fine-grained level as nodes; each node in the document tree carries its position information in the original document; A determination module, used to determine the position information of the node in the document tree whose similarity with the annotated document fragment is greater than a target similarity threshold and has the highest similarity as the position information corresponding to the annotated document fragment; the target similarity threshold is obtained by adjusting the general similarity threshold based on the characteristics of the annotated document fragment itself; An evaluation module is used to evaluate the recall result of the target knowledge question answering system according to the overlap length between the position information corresponding to the recalled document fragment and the position information corresponding to the annotated document fragment.

9. An electronic device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the storage medium communicate via the bus, and the processor executes the machine-readable instructions to perform the steps of the method for evaluating and optimizing the recall results of the knowledge question and answer system as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for evaluating and optimizing the recall results of the knowledge question and answer system according to any one of claims 1 to 7 are executed.

Images