Question-answering model evaluation method and device

Abstract

This invention relates to the field of data processing technology, and provides a method and apparatus for evaluating a question-answering model. The method includes: inputting historical question-answer pairs, multiple instruction documents, questions from user query question-answer pairs, and a first prompt word from an evaluation dataset into a question-answering model to be tested; obtaining a predicted answer and a predicted step sequence for the questions in the user query question-answer pairs, inferred by the model based on the first prompt word; wherein, the multiple instruction documents are obtained based on user query text retrieval; the historical question-answer pairs and the questions in the user query question-answer pairs are generated by a first-level model based on the instruction documents; the standard answer and the standard step sequence are generated by the first-level model based on the historical question-answer pairs, multiple instruction documents, and the questions in the user query question-answer pairs; comparing the standard answer and the predicted answer, and comparing the standard step sequence and the predicted step sequence, to obtain the evaluation result. This invention achieves effective evaluation of the question-answering model to be tested in scenarios with multi-step, procedural instructions.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a question-answering model evaluation method and apparatus. Background Technology

[0002] With the rapid development of large-scale language models, they have been widely applied in question answering, text generation, and human-computer dialogue. Evaluating the effectiveness of question answering models in answering questions (accuracy or completeness, etc.) is the primary means of evaluating their quality. Currently, mainstream question answering model evaluation methods still focus on factual questions based on single texts. For real-world question answering tasks involving multi-step, procedural instructions, especially when faced with instruction documents from multiple heterogeneous sources, these methods cannot effectively evaluate the output answers of question answering models. Summary of the Invention

[0003] This invention provides a question-answering model evaluation method and apparatus to solve the problem in the prior art that the output answer of the question-answering model cannot be effectively evaluated when users have question-answering needs for multi-step, process-oriented instructions.

[0004] This invention provides a question-answering model evaluation method, comprising the following steps:

[0005] The evaluation dataset includes historical question-and-answer pairs, multiple instruction documents, questions from user query question-and-answer pairs, and a first prompt word. These are input into the question-and-answer model under test. The model then infers and outputs the predicted answer and predicted step sequence for the questions in the user query question-and-answer pairs based on the first prompt word. The first prompt word instructs the model to analyze the multiple instruction documents and the historical question-and-answer pairs to infer and output the predicted answer and predicted step sequence for the questions in the user query question-and-answer pairs. The evaluation dataset includes historical question-and-answer pairs, user query question-and-answer pairs, standard step sequences corresponding to the standard answers in the user query question-and-answer pairs, and multiple instruction documents. The multiple instruction documents are obtained based on user query text retrieval. The questions in the historical question-and-answer pairs and user query question-and-answer pairs are generated by the first model based on the multiple instruction documents. The standard answer and standard step sequence are generated by the first model based on the historical question-and-answer pairs, the multiple instruction documents, and the questions in the user query question-and-answer pairs.

[0006] The evaluation results are obtained by comparing the standard answer and the predicted answer, and by comparing the standard step sequence and the predicted step sequence.

[0007] According to a question-answering model evaluation method provided by the present invention, before inputting historical question-answer pairs, multiple instruction documents, questions in user query question-answer pairs, and a first prompt word from the evaluation dataset into the question-answering model to be tested, and obtaining the predicted answer and prediction step sequence of the questions in the user query question-answer pairs inferred by the question-answering model to be tested according to the first prompt word, the method further includes:

[0008] Obtain the user's query text;

[0009] Retrieve multiple instruction documents obtained based on the user's query text;

[0010] Input multiple instruction documents, user query text, and second prompt words into the first large model to obtain the historical question-and-answer pairs and the questions in the user query question-and-answer pairs after the first large model analyzes multiple instruction documents and user query text according to the instructions in the second prompt words.

[0011] Input the questions and first prompt words from multiple instruction documents, historical question-and-answer pairs, and user query question-and-answer pairs into the first large model to obtain the standard answer and standard step sequence of the questions in the user query question-and-answer pairs after the first large model analyzes multiple instruction documents and historical question-and-answer pairs according to the instructions in the first prompt words.

[0012] The evaluation dataset is constructed based on the historical question-and-answer pairs, user query question-and-answer pairs, standard step sequences, and multiple instruction documents.

[0013] According to a question-answering model evaluation method provided by the present invention, obtaining the user query text includes:

[0014] Retrieve all query content entered by the user from the user query pool;

[0015] Input all the query content and the third prompt word into the second large model to obtain the user query text filtered from all the query content after the second large model analyzes all the query content according to the instructions of the third prompt word;

[0016] The third prompt word is used to instruct the second model to filter out query content containing the user's task completion requirements from all query content as the user's query text.

[0017] According to the question-answering model evaluation method provided by the present invention, after constructing the evaluation dataset based on the historical question-answer pairs, user query question-answer pairs, standard step sequences, and multiple instruction documents, the method further includes:

[0018] Receive external input for revisions of standard answers and / or standard step sequences in historical question-and-answer pairs and user query question-and-answer pairs.

[0019] According to the question-answering model evaluation method provided by the present invention, an evaluation result is obtained based on a comparison between the standard answer and the predicted answer, and a comparison between the standard step sequence and the predicted step sequence, including:

[0020] Calculate a recall-oriented summary evaluation metric based on the standard answer and the predicted answer;

[0021] Based on the comparison between the standard answer and the predicted answer, and the comparison between the standard step sequence and the predicted step sequence, the relevance score and task completion rate of the question-answering model under test are calculated; wherein, the relevance score represents the degree of relevance between the predicted answer and the question in the user query question-answering pair, and the task completion rate represents the proportion of the number of steps that appear in both the predicted step sequence and the standard step sequence to the total number of steps in the standard step sequence.

[0022] According to a question-answering model evaluation method provided by the present invention, based on the comparison between the standard answer and the predicted answer, and the comparison between the standard step sequence and the predicted step sequence, a relevance score and a task completion rate characterizing the question-answering model under test are calculated, including:

[0023] The historical question-and-answer pairs, the questions in the user query question-and-answer pairs, the standard answers, the predicted answers, the standard step sequences, the predicted step sequences, and the fourth prompt word are input into the third major model. The third major model then analyzes the historical question-and-answer pairs, the questions in the user query question-and-answer pairs, the standard answers, the predicted answers, the standard step sequences, and the predicted step sequences according to the instructions in the fourth prompt word, and outputs the relevance score and task completion rate.

[0024] The present invention also provides a question-answering model evaluation device, comprising the following modules:

[0025] The answer output unit is used to input historical question-and-answer pairs, multiple instruction documents, questions from user query question-and-answer pairs, and a first prompt word from the evaluation dataset into the question-and-answer model under test, and obtain the predicted answer and predicted step sequence of the questions in the user query question-and-answer pairs inferred by the question-and-answer model under test according to the first prompt word. The first prompt word is used to instruct the question-and-answer model under test to analyze the multiple instruction documents and the historical question-and-answer pairs to infer and output the predicted answer and predicted step sequence of the questions in the user query question-and-answer pairs. The evaluation dataset includes: historical question-and-answer pairs, user query question-and-answer pairs, standard step sequences corresponding to the standard answers in the user query question-and-answer pairs, and multiple instruction documents. The multiple instruction documents are obtained based on user query text retrieval. The questions in the historical question-and-answer pairs and user query question-and-answer pairs are generated by the first model based on the multiple instruction documents. The standard answer and standard step sequence are generated by the first model based on the questions in the historical question-and-answer pairs, multiple instruction documents, and user query question-and-answer pairs.

[0026] The answer comparison unit is used to compare the standard answer and the predicted answer, and to compare the standard step sequence and the predicted step sequence to obtain the evaluation result.

[0027] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the program to implement the question-answering model evaluation method as described above.

[0028] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the question-answering model evaluation method as described above.

[0029] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the question-answering model evaluation method as described above.

[0030] The question-answering model evaluation method and apparatus provided by this invention includes an evaluation dataset comprising multiple instruction documents corresponding to user query text. A first large model infers the questions in historical question-answer pairs and user query question-answer pairs based on multiple instruction documents, simulating multi-step (or multi-round) question-answering scenarios. Under the guidance of the first prompt word, the question-answering model under test infers the predicted answer and predicted step sequence of the questions in the user query question-answer pair based on the analysis of the multi-step (or multi-round) question-answering scenario and multiple instruction documents. The predicted answer and predicted step sequence are then compared with the standard answer and standard step sequence, respectively. This achieves effective evaluation of the output answer of the question-answering model under test in scenarios where users have question-answering needs for multi-step, procedural instructions. Moreover, the question-answering format of the multi-step (or multi-round) question-answering scenario and multiple instruction documents is closer to user habits, supports multi-round contextual reasoning and continuous interaction with context, and enhances the evaluation of the dialogue ability of the question-answering model under test. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0032] Figure 1 This is a flowchart illustrating the question-answering model evaluation method provided by the present invention.

[0033] Figure 2 This is a schematic diagram of the evaluation data in the question-answering model evaluation method provided by the present invention.

[0034] Figure 3 This is a schematic diagram of the first prompt word in the question-answering model evaluation method provided by the present invention.

[0035] Figure 4 This is a schematic diagram of the second prompt word in the question-answering model evaluation method provided by the present invention.

[0036] Figure 5 This is a schematic diagram showing the application domain distribution of the evaluation dataset in the question-answering model evaluation method provided by this invention.

[0037] Figure 6 This is a schematic diagram of the third prompt word in the question-answering model evaluation method provided by the present invention.

[0038] Figure 7 This is a schematic diagram of the fourth prompt word in the question-answering model evaluation method provided by the present invention.

[0039] Figure 8 This is a schematic diagram of the question-answering model evaluation device provided by the present invention.

[0040] Figure 9 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0042] The question-answering model evaluation method of this invention, such as... Figure 1 As shown, it includes steps S110 and S120.

[0043] Step S110: Input the historical question-and-answer pairs, multiple instruction documents, questions in user query question-and-answer pairs, and the first prompt word from the evaluation dataset into the question-and-answer model under test, and obtain the predicted answer and predicted step sequence of the questions in the user query question-and-answer pairs inferred by the question-and-answer model under test according to the first prompt word. The first prompt word is used to instruct the question-and-answer model under test to analyze multiple instruction documents and the historical question-and-answer pairs in order to infer and output the predicted answer and predicted step sequence of the questions in the user query question-and-answer pairs.

[0044] The evaluation dataset is a pre-constructed sample set, specifically including: historical question-answer pairs, user query question-answer pairs, standard step sequences corresponding to the standard answers in the user query question-answer pairs, and multiple instruction documents. The multiple instruction documents are obtained based on user query text retrieval. The questions in the historical question-answer pairs and user query question-answer pairs are generated by the first model based on the multiple instruction documents. The standard answers and standard step sequences are generated by the first model based on the questions in the historical question-answer pairs, the multiple instruction documents, and the user query question-answer pairs.

[0045] Understandably, a user query text is a question asked by the user. After the user asks a question on the internet, other users will post answers based on the question in a thread. This thread is the instruction document, instructing the user on how to proceed. Different users will provide different answers; therefore, one user query text usually corresponds to multiple instruction documents.

[0046] The questions in the historical question-and-answer pairs and user query question-and-answer pairs are generated by the first major model based on multiple instruction documents. That is, the first major model analyzes the content of multiple instruction documents to generate multiple historical question-and-answer pairs and user query question-and-answer pairs. The historical question-and-answer pairs are question-and-answer pairs that the first major model infers from multiple instruction documents, which are questions and answers that precede the user query question-and-answer pairs and have a sequential question-and-answer logical relationship with the user query question-and-answer pairs. Thus, the historical question-and-answer pairs simulate the multi-step or multi-round question-and-answer scenarios before the user query question-and-answer in daily real-world task scenarios.

[0047] The standard answer and standard step sequence are generated by the first major model based on the historical question-and-answer pairs, multiple instruction documents, and questions in the user query question-and-answer pairs. In other words, the first major model analyzes the historical question-and-answer pairs and multiple instruction documents, answers the questions in the user query question-and-answer pairs, and obtains the standard answer. The first major model further refines the standard answer into clear, structured, process-oriented operation instructions, and generates a set of standard step sequences that are not sublisted, have a clear logical order, and are semantically complete. Users who ask questions can solve their questions by following these standard step sequences.

[0048] For example: Figure 2 As shown, the user query text corresponds to four instruction documents. Question Q3 in the user query question-answer pair is the last question. Q1A1 and Q2A2 are both historical question-answer pairs inferred and output by the first model based on the four instruction documents. A3 is the standard answer to Q3 inferred and output by the first model based on the four instruction documents, as well as Q1A1 and Q2A2. The steps 1-4 below A3 are the standard step sequence obtained by the first model from further refining the standard answer.

[0049] In this step, when evaluating the question-answering model under test, the historical question-answer pairs, multiple instruction documents, questions from user query question-answer pairs, and the first prompt word from the evaluation dataset are input into the model. For example, the first prompt word might be... Figure 3 As shown. The first prompt word is used to instruct the question-answering model under test to understand and analyze multiple instruction documents and historical question-answer pairs, and to infer and output the predicted answer and prediction step sequence of the user query question-answer pair.

[0050] Step S120: Based on the comparison between the standard answer and the predicted answer, and the comparison between the standard step sequence and the predicted step sequence, an evaluation result is obtained. Specifically, the closer the predicted answer is to the standard answer, and the closer the predicted step sequence is to the standard step sequence, the better the performance of the question-answering model under test, i.e., the better the evaluation result. For example, the evaluation result can adopt the traditional recall-oriented summary evaluation metric ROUGE.

[0051] It should be noted that: since the first model generates historical question-and-answer pairs (e.g., Q1A1~Q2A2), user query questions in the question-and-answer pairs, and standard answers and standard step sequences for the user query questions in the question-and-answer pairs based on multiple instruction documents, the question-and-answer model under test generates predicted answers and predicted step sequences based on multiple instruction documents and historical question-and-answer pairs. That is, it refers to the historical question-and-answer pairs generated by the first model to generate predicted answers and predicted step sequences. Therefore, the question-and-answer model under test can be evaluated by comparing the generated predicted answers and predicted step sequences with the standard answers and standard step sequences, respectively.

[0052] In the question-answering model evaluation method of this embodiment, the evaluation dataset includes multiple instruction documents corresponding to the user's query text. A first large model infers the questions in historical question-answer pairs and user query question-answer pairs based on multiple instruction documents, simulating a multi-step (or multi-round) question-answering scenario. Under the guidance of the first prompt word, the question-answering model under test infers the predicted answer and predicted step sequence of the questions in the user query question-answer pair based on the analysis of the multi-step (or multi-round) question-answering scenario and multiple instruction documents. Then, the predicted answer and predicted step sequence are compared with the standard answer and standard step sequence, respectively. This achieves effective evaluation of the output answer of the question-answering model under test in the scenario where users have question-answering needs for multi-step, process-oriented instructions. Moreover, the question-answering format of multi-step (or multi-round) question-answering scenarios and multiple instruction documents is closer to user habits, supports multi-round contextual reasoning and continuous interaction with context, and enhances the evaluation of the dialogue ability of the question-answering model under test.

[0053] In some embodiments, prior to step S110, a step of constructing an evaluation dataset is further included, specifically including:

[0054] Step 1: Obtain the user query text. This user query text is the natural language query entered by the user when asking a question on the Internet, and can be obtained by querying the user query pool of a Q&A website or Q&A app.

[0055] Step 2: Obtain multiple instruction documents retrieved based on the user's query text. This involves retrieving multiple instruction documents from other users' responses based on the user's query text.

[0056] Step 3: Input multiple instruction documents, user query text, and second prompt words into the first large model. The first large model analyzes the multiple instruction documents and user query text according to the instructions in the second prompt words, and outputs the historical question-and-answer pairs and the questions in the user query question-and-answer pairs. The question in the user query question-and-answer pair is the last question. For example... Figure 2 Question Q3. For example, the second prompt word is as follows: Figure 4As shown, the first major model, based on the indication of the second prompt word, understands and analyzes multiple instruction documents and user query texts, inferring the questions in multiple historical question-and-answer pairs and user query question-and-answer pairs. Since the questions in the user query question-and-answer pairs correspond to the user query text and represent the currently posed question, the first major model outputs the questions in the user query question-and-answer pairs at the end of the historical question-and-answer pairs.

[0057] Step 4: Input the questions from multiple instruction documents, historical question-and-answer pairs, and user query question-and-answer pairs, along with the first prompt word, into the first large model. This yields the standard answer and standard step sequence output by the first large model after analyzing the multiple instruction documents and historical question-and-answer pairs according to the instructions in the first prompt word. The first prompt word is as follows: Figure 3 As shown, the predicted answer and prediction step sequence are the same as those of the question-answering model under test, thus avoiding the impact on the accuracy of the evaluation results due to different prompt words.

[0058] Step 5: Construct the evaluation dataset based on the historical question-and-answer pairs, user query question-and-answer pairs, standard step sequences, and multiple instruction documents. For each user query text from Step 1, there is one corresponding evaluation dataset, for example... Figure 2 The evaluation data shown.

[0059] In this embodiment, the evaluation dataset was constructed through the above steps. For example... Figure 5 As shown, to evaluate question-answering models across multiple application domains, the evaluation dataset extensively covers 13 major real-world application domains, containing 13,959 question-answering sample evaluation data. On average, each user query question-answer pair corresponds to 3.11 rounds of questions (historical question-answer pairs and user query question-answer pairs), referencing 4.09 instruction documents, ultimately generating 6.5 structured process instructions, i.e., standard step sequences. This demonstrates the high diversity, practicality, and process-orientation of the evaluation dataset, enabling effective evaluation of the question-answering model's output answers in scenarios where users require multi-step, process-oriented instructions for question-answering.

[0060] In some embodiments, obtaining the user query text specifically includes:

[0061] Retrieve all user-entered queries from the user query pool. Many Q&A websites and apps store user-entered queries in a user query pool; therefore, you can retrieve all user-entered queries by accessing this pool.

[0062] All query content and the third prompt word are input into the second large model. The second large model analyzes all query content according to the instructions of the third prompt word, and then filters out the user query text from all query content. The third prompt word instructs the second large model to filter query content containing the user's task completion requirements as the user query text. That is, the second large model filters all query content to select query content containing the user's task completion requirements. For example, the third prompt word might be... Figure 6 As shown, for queries containing user task completion requirements, the answer is "yes", thus filtering out queries containing user task completion requirements.

[0063] In this embodiment, filtering query content that includes user task completion requirements can be understood as removing content that is highly time-dependent or subjective, retaining query content with clear task objectives, and ensuring the quality and instructiveness of the constructed dataset.

[0064] In some embodiments, after constructing the evaluation dataset based on the historical question-and-answer pairs, user query question-and-answer pairs, standard step sequences, and multiple instruction documents, the method further includes: receiving external input for revisions to the standard answers and / or standard step sequences in the historical question-and-answer pairs and user query question-and-answer pairs. For example, three human annotators are introduced to review the samples, removing problematic evaluation data, such as data with question-and-answer swaps, mismatched question-and-answer content, marketing content, low-quality language generation, politically sensitive information, or incorrect step order in the standard step sequences. Additionally, 10% of the evaluation data in the dataset undergoes manual review, ultimately achieving an accuracy rate of over 95%.

[0065] In this embodiment, after constructing the evaluation dataset based on the historical question-and-answer pairs, user query question-and-answer pairs, standard step sequences, and multiple instruction documents, the standard answers and / or standard step sequences in the historical question-and-answer pairs and user query question-and-answer pairs in the evaluation dataset are manually revised. This further ensures the matching degree of each question-and-answer pair in the evaluation dataset and the correctness of the step order of the standard step sequences, thereby achieving a more accurate and effective evaluation of the question-and-answer model under test.

[0066] In some embodiments, step S120 specifically includes:

[0067] Based on the standard answer and the predicted answer, a recall-oriented summary evaluation metric is calculated. ROUGE (Recall-Oriented Summary Evaluation) is a traditional evaluation metric. In this embodiment, the ROUGE score can be represented by the literal and semantic similarity between the standard answer and the predicted answer. ROUGE's score The specific calculation formula is as follows:

[0068] ;

[0069] ;

[0070] .

[0071] in, X This represents the standard answer. m express X The length of the characters, Y Indicates the predicted answer. n express Y The length of the characters, β It's a hyperparameter. Recall rate. Indicates accuracy. This indicates that two sentences need to be solved. X and Y The function of longest common subsequence between two pairs of sequences is used to determine their similarity. The longer the longest common subsequence, the higher the similarity between the two pairs.

[0072] Based on the comparison between the standard answer and the predicted answer, and the comparison between the standard step sequence and the predicted step sequence, the relevance score and task completion rate of the question-answering model under test are calculated; wherein, the relevance score represents the degree of relevance between the predicted answer and the question in the user query question-answering pair, and the task completion rate represents the proportion of the number of steps that appear in both the predicted step sequence and the standard step sequence to the total number of steps in the standard step sequence.

[0073] For example, comparing the predicted answer output by the question-answering model under test with the standard answer, such as comparing the similarity between the two, can assess the relevance between the answer output by the question-answering model under test and the question in the user's query question-answer pair. The greater the similarity between the two, the greater the relevance, and the higher the accuracy of the predicted answer.

[0074] Task completion rate Task completion rate It can be calculated using the following formula:

[0075] .

[0076] in, M c This indicates the number of steps that appear in both the predicted step sequence and the standard step sequence. K This represents the total number of steps in the standard step sequence.

[0077] It should be noted that the number of steps in the predicted step sequence and the standard step sequence may differ. The number of steps appearing in both sequences is calculated based on the steps in the standard step sequence to ensure task completion rate. Task completion rate The value is not greater than 1. For example: the total number of steps in the prediction step sequence is five, while the total number of steps in the standard step sequence is four. If the content of two steps in the prediction step sequence is represented in one step in the standard step sequence, then the two steps in the prediction step sequence are counted as one step.

[0078] In this embodiment, by comparing the standard answer and the predicted answer, as well as the standard step sequence and the predicted step sequence, an effective evaluation of the question-answering model under test is achieved in a multi-step, procedural question-answering scenario. This evaluation considers three dimensions: ROUGE metric, total rating, and task completion rate. For each evaluation dimension, a higher score indicates a better question-answering performance of the model.

[0079] In some embodiments, based on a comparison between the standard answer and the predicted answer, and a comparison between the standard step sequence and the predicted step sequence, a relevance score and task completion rate characterizing the question-answering model's answers are calculated, including:

[0080] The historical question-and-answer pairs, the questions in the user query question-and-answer pairs, the standard answers, the predicted answers, the standard step sequences, the predicted step sequences, and the fourth prompt word are input into the third major model. The third major model then analyzes the historical question-and-answer pairs, the questions in the user query question-and-answer pairs, the standard answers, the predicted answers, the standard step sequences, and the predicted step sequences according to the instructions in the fourth prompt word, and outputs the relevance score and task completion rate.

[0081] For example, the fourth prompt word is as follows: Figure 7 As shown, in this embodiment, the third major model uses the fourth prompt word combined with historical question-and-answer pairs to analyze whether the predicted answer solves the problem in the user's query question-and-answer pair. The predicted answer is compared with the standard answer, and a relevance score of 1-10 is output. The third major model also uses the fourth prompt word to represent the completion rate of the aforementioned task. Task completion rate The calculation formula automatically calculates the task completion rate.

[0082] It should be noted that each evaluation data point in the evaluation dataset will have a relevance score and a task completion rate. The final evaluation result of the question answering model under test is the average of the relevance score and the average of the task completion rate obtained from each evaluation data point.

[0083] In this embodiment, a large model (third large model) is introduced to assist the evaluation mechanism, which standardizes and automates the evaluation process, significantly improving evaluation efficiency and consistency of evaluation results.

[0084] It is understood that the first, second, and third major models in the above embodiments can all be implemented using currently mature large language models (such as DeepSeek R1 or Qwen3).

[0085] The question-answering model evaluation device provided by the present invention is described below. The question-answering model evaluation device described below can be referred to in correspondence with the question-answering model evaluation method described above.

[0086] The question-answering model evaluation device in this embodiment, such as Figure 8 As shown, it includes:

[0087] The answer output unit 810 is used to input historical question-and-answer pairs, multiple instruction documents, questions in user query question-and-answer pairs, and a first prompt word from the evaluation dataset into the question-and-answer model under test, and obtain the predicted answer and predicted step sequence of the questions in the user query question-and-answer pairs inferred by the question-and-answer model under test according to the first prompt word. The first prompt word is used to instruct the question-and-answer model under test to analyze the multiple instruction documents and the historical question-and-answer pairs to infer and output the predicted answer and predicted step sequence of the questions in the user query question-and-answer pairs. The evaluation dataset includes: historical question-and-answer pairs, user query question-and-answer pairs, standard step sequences corresponding to the standard answers in the user query question-and-answer pairs, and multiple instruction documents. The multiple instruction documents are obtained based on user query text retrieval. The questions in the historical question-and-answer pairs and user query question-and-answer pairs are generated by the first model based on the multiple instruction documents. The standard answer and standard step sequence are generated by the first model based on the questions in the historical question-and-answer pairs, multiple instruction documents, and user query question-and-answer pairs.

[0088] The answer comparison unit 820 is used to compare the standard answer and the predicted answer, and the standard step sequence and the predicted step sequence to obtain the evaluation result.

[0089] In some embodiments, the question-answering model evaluation apparatus further includes:

[0090] The user query text acquisition unit is used to acquire the user query text.

[0091] The instruction document acquisition unit is used to acquire multiple instruction documents obtained by searching based on the user query text.

[0092] The historical question-and-answer pair output unit is used to input multiple instruction documents, user query text, and second prompt words into the first large model, and obtain the historical question-and-answer pairs and the questions in the user query question-and-answer pairs output by the first large model after analyzing multiple instruction documents and user query text according to the instructions in the second prompt words.

[0093] The standard answer output unit is used to input multiple instruction documents, historical question-and-answer pairs, questions from user query question-and-answer pairs, and a first prompt word into the first large model, and obtain the standard answer and the standard step sequence of the questions in the user query question-and-answer pairs after the first large model analyzes multiple instruction documents and historical question-and-answer pairs according to the instructions in the first prompt word.

[0094] The dataset construction unit is used to construct the evaluation dataset based on the historical question-and-answer pairs, user query question-and-answer pairs, standard step sequences, and multiple instruction documents.

[0095] In some embodiments, the user query text acquisition unit specifically includes:

[0096] The query content retrieval unit is used to retrieve all query content entered by the user from the user query pool.

[0097] The query content filtering unit is used to input all the query content and the third prompt word into the second large model, and obtain the user query text filtered from all the query content after the second large model analyzes all the query content according to the instructions of the third prompt word.

[0098] The third prompt word is used to instruct the second model to filter out query content containing the user's task completion requirements from all query content as the user's query text.

[0099] In some embodiments, the question-answering model evaluation apparatus further includes: a revision receiving unit, configured to receive external input revisions to the standard answers and / or standard step sequences in the historical question-answering pairs and user query question-answering pairs after constructing the evaluation dataset based on the historical question-answering pairs, user query question-answering pairs, standard step sequences and multiple instruction documents.

[0100] In some embodiments, the answer comparison unit 820 is specifically used for:

[0101] Calculate a recall-oriented summary evaluation metric based on the standard answer and the predicted answer;

[0102] Based on the comparison between the standard answer and the predicted answer, and the comparison between the standard step sequence and the predicted step sequence, the relevance score and task completion rate of the question-answering model under test are calculated; wherein, the relevance score represents the degree of relevance between the predicted answer and the question in the user query question-answering pair, and the task completion rate represents the proportion of the number of steps that appear in both the predicted step sequence and the standard step sequence to the total number of steps in the standard step sequence.

[0103] In some embodiments, the answer comparison unit 820 is specifically used to: input the historical question-and-answer pairs, the questions in the user query question-and-answer pairs, the standard answers, the predicted answers, the standard step sequences, the predicted step sequences, and the fourth prompt word into a third large model, and obtain the relevance score and task completion rate output by the third large model after analyzing the historical question-and-answer pairs, the questions in the user query question-and-answer pairs, the standard answers, the predicted answers, the standard step sequences, and the predicted step sequences according to the instructions in the fourth prompt word.

[0104] Figure 9 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 9 As shown, the electronic device may include: a processor 910, a communications interface 920, a memory 930, and a communication bus 940, wherein the processor 910, the communications interface 920, and the memory 930 communicate with each other via the communication bus 940. The processor 910 can call logical instructions in the memory 930 to execute a question-answering model evaluation method, which includes:

[0105] The evaluation dataset includes historical question-and-answer pairs, multiple instruction documents, questions from user query question-and-answer pairs, and a first prompt word. These are input into the question-and-answer model under test. The model then infers and outputs the predicted answer and predicted step sequence for the questions in the user query question-and-answer pairs based on the first prompt word. The first prompt word instructs the model to analyze the multiple instruction documents and the historical question-and-answer pairs to infer and output the predicted answer and predicted step sequence for the questions in the user query question-and-answer pairs. The evaluation dataset includes historical question-and-answer pairs, user query question-and-answer pairs, standard step sequences corresponding to the standard answers in the user query question-and-answer pairs, and multiple instruction documents. The multiple instruction documents are obtained based on user query text retrieval. The questions in the historical question-and-answer pairs and user query question-and-answer pairs are generated by the first model based on the multiple instruction documents. The standard answer and standard step sequence are generated by the first model based on the historical question-and-answer pairs, the multiple instruction documents, and the questions in the user query question-and-answer pairs.

[0106] The evaluation results are obtained by comparing the standard answer and the predicted answer, and by comparing the standard step sequence and the predicted step sequence.

[0107] Furthermore, the logical instructions in the aforementioned memory 930 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0108] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the question-answering model evaluation method provided by the above methods, the method comprising:

[0109] The evaluation dataset includes historical question-and-answer pairs, multiple instruction documents, questions from user query question-and-answer pairs, and a first prompt word. These are input into the question-and-answer model under test. The model then infers and outputs the predicted answer and predicted step sequence for the questions in the user query question-and-answer pairs based on the first prompt word. The first prompt word instructs the model to analyze the multiple instruction documents and the historical question-and-answer pairs to infer and output the predicted answer and predicted step sequence for the questions in the user query question-and-answer pairs. The evaluation dataset includes historical question-and-answer pairs, user query question-and-answer pairs, standard step sequences corresponding to the standard answers in the user query question-and-answer pairs, and multiple instruction documents. The multiple instruction documents are obtained based on user query text retrieval. The questions in the historical question-and-answer pairs and user query question-and-answer pairs are generated by the first model based on the multiple instruction documents. The standard answer and standard step sequence are generated by the first model based on the historical question-and-answer pairs, the multiple instruction documents, and the questions in the user query question-and-answer pairs.

[0110] The evaluation results are obtained by comparing the standard answer and the predicted answer, and by comparing the standard step sequence and the predicted step sequence.

[0111] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the question-answering model evaluation method provided by the methods described above, the method comprising:

[0112] The evaluation dataset includes historical question-and-answer pairs, multiple instruction documents, questions from user query question-and-answer pairs, and a first prompt word. These are input into the question-and-answer model under test. The model then infers and outputs the predicted answer and predicted step sequence for the questions in the user query question-and-answer pairs based on the first prompt word. The first prompt word instructs the model to analyze the multiple instruction documents and the historical question-and-answer pairs to infer and output the predicted answer and predicted step sequence for the questions in the user query question-and-answer pairs. The evaluation dataset includes historical question-and-answer pairs, user query question-and-answer pairs, standard step sequences corresponding to the standard answers in the user query question-and-answer pairs, and multiple instruction documents. The multiple instruction documents are obtained based on user query text retrieval. The questions in the historical question-and-answer pairs and user query question-and-answer pairs are generated by the first model based on the multiple instruction documents. The standard answer and standard step sequence are generated by the first model based on the historical question-and-answer pairs, the multiple instruction documents, and the questions in the user query question-and-answer pairs.

[0113] The evaluation results are obtained by comparing the standard answer and the predicted answer, and by comparing the standard step sequence and the predicted step sequence.

[0114] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0115] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A question-answering model evaluation method, characterized in that, include: The historical question-and-answer pairs, multiple instruction documents, questions in user query question-and-answer pairs, and the first prompt word in the evaluation dataset are input into the question-and-answer model under test. The predicted answer and prediction step sequence of the questions in the user query question-and-answer pairs are obtained by the question-and-answer model under test inferring the questions in the user query question-and-answer pairs according to the first prompt word. The first prompt word is used to instruct the question-and-answer model under test to analyze multiple instruction documents and the historical question-and-answer pairs in order to infer and output the predicted answer and prediction step sequence of the questions in the user query question-and-answer pairs. The evaluation dataset includes: historical question-and-answer pairs, user query question-and-answer pairs, standard step sequences corresponding to the standard answers in the user query question-and-answer pairs, and multiple instruction documents. The multiple instruction documents are obtained based on user query text retrieval. The questions in the historical question-and-answer pairs and user query question-and-answer pairs are generated by the first model based on the multiple instruction documents. The standard answers and standard step sequences are generated by the first model based on the questions in the historical question-and-answer pairs, multiple instruction documents, and user query question-and-answer pairs. Based on the comparison between the standard answer and the predicted answer, and the comparison between the standard step sequence and the predicted step sequence, the evaluation results are obtained. The evaluation results are obtained by comparing the standard answer and the predicted answer, and comparing the standard step sequence and the predicted step sequence, including: Calculate a recall-oriented summary evaluation metric based on the standard answer and the predicted answer; Based on the comparison between the standard answer and the predicted answer, and the comparison between the standard step sequence and the predicted step sequence, the relevance score and task completion rate of the question-answering model under test are calculated; wherein, the relevance score represents the degree of relevance between the predicted answer and the question in the user query question-answering pair, and the task completion rate represents the proportion of the number of steps that appear in both the predicted step sequence and the standard step sequence to the total number of steps in the standard step sequence.

2. The question-answering model evaluation method according to claim 1, characterized in that, Before inputting historical question-and-answer pairs, multiple instruction documents, questions from user query question-and-answer pairs, and the first prompt word from the evaluation dataset into the question-and-answer model to be tested, and obtaining the predicted answer and prediction step sequence of the questions in the user query question-and-answer pairs output by the question-and-answer model according to the first prompt word, the process further includes: Obtain the user's query text; Retrieve multiple instruction documents obtained based on the user's query text; Input multiple instruction documents, user query text, and second prompt words into the first large model to obtain the historical question-and-answer pairs and the questions in the user query question-and-answer pairs after the first large model analyzes multiple instruction documents and user query text according to the instructions in the second prompt words. Input the questions and first prompt words from multiple instruction documents, historical question-and-answer pairs, and user query question-and-answer pairs into the first large model to obtain the standard answer and standard step sequence of the questions in the user query question-and-answer pairs after the first large model analyzes multiple instruction documents and historical question-and-answer pairs according to the instructions in the first prompt words. The evaluation dataset is constructed based on the historical question-and-answer pairs, user query question-and-answer pairs, standard step sequences, and multiple instruction documents.

3. The question-answering model evaluation method according to claim 2, characterized in that, Obtaining the user query text includes: Retrieve all query content entered by the user from the user query pool; Input all the query content and the third prompt word into the second large model to obtain the user query text filtered from all the query content after the second large model analyzes all the query content according to the instructions of the third prompt word; The third prompt word is used to instruct the second model to filter out query content containing the user's task completion requirements from all query content as the user's query text.

4. The question-answering model evaluation method according to claim 2, characterized in that, After constructing the evaluation dataset based on the historical question-and-answer pairs, user query question-and-answer pairs, standard step sequences, and multiple instruction documents, the method further includes: Receive external input for revisions of standard answers and / or standard step sequences in historical question-and-answer pairs and user query question-and-answer pairs.

5. The question-answering model evaluation method according to claim 1, characterized in that, Based on the comparison between the standard answer and the predicted answer, and the comparison between the standard step sequence and the predicted step sequence, the relevance score and task completion rate representing the question-answering model's answers are calculated, including: The historical question-and-answer pairs, the questions in the user query question-and-answer pairs, the standard answers, the predicted answers, the standard step sequences, the predicted step sequences, and the fourth prompt word are input into the third major model. The third major model then analyzes the historical question-and-answer pairs, the questions in the user query question-and-answer pairs, the standard answers, the predicted answers, the standard step sequences, and the predicted step sequences according to the instructions in the fourth prompt word, and outputs the relevance score and task completion rate.

6. A question-answering model evaluation device, characterized in that, include: The answer output unit is used to input the historical question-and-answer pairs, multiple instruction documents, questions in user query question-and-answer pairs, and the first prompt word from the evaluation dataset into the question-and-answer model under test, and obtain the predicted answer and prediction step sequence of the questions in the user query question-and-answer pairs inferred by the question-and-answer model under test according to the first prompt word. The first prompt word is used to instruct the question-and-answer model under test to analyze the multiple instruction documents and the historical question-and-answer pairs in order to infer and output the predicted answer and prediction step sequence of the questions in the user query question-and-answer pairs. The evaluation dataset includes: historical question-and-answer pairs, user query question-and-answer pairs, standard step sequences corresponding to the standard answers in the user query question-and-answer pairs, and multiple instruction documents. The multiple instruction documents are obtained based on user query text retrieval. The questions in the historical question-and-answer pairs and user query question-and-answer pairs are generated by the first model based on the multiple instruction documents. The standard answers and standard step sequences are generated by the first model based on the questions in the historical question-and-answer pairs, multiple instruction documents, and user query question-and-answer pairs. The answer comparison unit is used to compare the standard answer and the predicted answer, and the standard step sequence and the predicted step sequence to obtain the evaluation result. The answer comparison unit is specifically used for: Calculate a recall-oriented summary evaluation metric based on the standard answer and the predicted answer; Based on the comparison between the standard answer and the predicted answer, and the comparison between the standard step sequence and the predicted step sequence, the relevance score and task completion rate of the question-answering model under test are calculated; wherein, the relevance score represents the degree of relevance between the predicted answer and the question in the user query question-answering pair, and the task completion rate represents the proportion of the number of steps that appear in both the predicted step sequence and the standard step sequence to the total number of steps in the standard step sequence.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the question-answering model evaluation method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the question-answering model evaluation method as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the question-answering model evaluation method as described in any one of claims 1 to 5.

Images