Question generation and evaluation method and system based on anti-fact logic modification

Abstract

The invention relates to the technical field of question generation, in particular to a question generation and evaluation method and system based on anti-fact logic modification, and the method comprises the steps: obtaining a reading understanding data set needing logical reasoning, and carrying out the annotation of the reading understanding data set, and obtaining a training set and a test set; training a pre-constructed large language model by using the training set; the questions in the test set are split into discussion points and prerequisites, anti-fact logic modification is carried out on the prerequisites based on the trained large language model, and the questions are generated; and performing self-scoring on whether the generated question conforms to logic or not based on a large language model. According to the large language model based on anti-fact logic modification, question generation can be completed more quickly in a more labor-saving manner, and automatic evaluation can be performed.

Description

Technical Field

[0001] The present invention relates to the technical field of question generation, and more particularly to a question generation and evaluation method and system based on counterfactual logic modification. Background Art

[0002] Counterfactual logic modification is an important topic in the field of counterfactual reasoning capabilities of large language models, which aims to cultivate counterfactual thinking processes in large language models. To this end, large language models need to have reasoning and logic modification capabilities to effectively change argument texts, maintain predetermined logical relationships, and complete self-evaluation. In addition to its significant theoretical value and ability to improve large language models, counterfactual logic modification also makes a significant contribution to the logical assurance of self-evaluation generated content.

[0003] The current language model's counterfactual logical reasoning process is internally inconsistent, and is quantified using the accuracy and pass rate of discrimination tasks such as multiple-choice questions as the main evaluation indicators. This approach misunderstands and misuses the discrimination of logical relations, and the evaluation method oversimplifies the goal of discovering essential and subtle flaws in complex reasoning. At the same time, current research on counterfactual reasoning pays little attention to the logical consistency or fidelity of modality. The expected model should be able to faithfully solve real-world reasoning problems and consider counterfactuals. And the evaluation should be conducted from different angles, including mathematical reasoning, common sense reasoning, logical reasoning, and reasoning in specific fields. However, most existing evaluation methods ignore the comprehensive evaluation of the reasoning process of large language models and do not measure whether the model understands the intermediate reasoning process.

[0004] However, the formulation of current science questions requires manpower, and AI-generated questions cannot automatically measure quality, so it is necessary to propose a science question generation method based on counterfactual logic modification and evaluation. Summary of the invention

[0005] In view of this, the present invention provides a question generation and evaluation method based on counterfactual logic modification, which can complete question generation and perform automated evaluation more quickly and with less effort.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a method for generating and evaluating questions based on counterfactual logic modification, comprising the following steps:

[0008] Obtain a reading comprehension dataset that requires logical reasoning, annotate it, and obtain a training set and a test set;

[0009] Using the training set to train a pre-built large language model;

[0010] Splitting the questions in the test set into arguments and premises, performing counterfactual logic modification on the premises based on the trained large language model, and generating questions;

[0011] Based on the large language model, the generated questions are evaluated for logical consistency.

[0012] Furthermore, after obtaining the reading comprehension dataset, given a question with context, question, options and correct answer, the question is annotated to obtain the wrong options and the corresponding modified context to form data points as a training sample.

[0013] Furthermore, the process of annotating the topic includes:

[0014] Determine the internal logical relationship of the problem;

[0015] Selecting an incorrect option edits the context by deleting, adding, or replacing spans of text in the statement to arrive at a revised argument.

[0016] Furthermore, the questions in the test set are divided into arguments and premises, and counterfactual logic modification of the premises is performed based on the trained large language model to generate questions, including:

[0017] The questions in the test set are divided into an argument and two premises, Premise 1 and Premise 2. The argument comes from the original context, Premise 1 comes from the correct option, and Premise 2 comes from the annotation. The argument and Premise 1 are related to each other through a logical relationship, forming a situation with a logical relationship R, state 1.

[0018] Use the given premise Premise2 and logical relation R to modify the statements in the argument Argument to obtain the modified argument Argument', forming the second state state2 with the logical relation R;

[0019] Switch state1 to state2, and use the modified argument 'Argument' in state2 as an option in the multiple-choice question.

[0020] Furthermore, the logical relationship R includes four types, namely:

[0021] R1: The premise provides necessary assumptions for the argument; R2: The premise provides sufficient assumptions for the argument; R3: The premise strengthens the argument; R4: The premise weakens the argument.

[0022] Furthermore, the self-scoring of whether the generated questions are logical based on the large language model includes:

[0023] One method is formed by using the premise Premise 1 and the argument Argument that satisfies the logical relation R, i.e. Pr(r|p1,a); the other group requires that the premise Premise 2 and the modified argument Argument' satisfy the same logical relation R, i.e. Pr(r|p2,a') is close to 1; the relationship between the premise Premise 2 and the argument Argument should not satisfy the logical relation R, i.e. Pr(r|p2,a) tends to 0;

[0024] Construct a logical concept map based on Premise 1, Argument, Premise 2, the modified Argument', logical relations, and Pr(r|p1,a), Pr(r|p2,a'), and Pr(r|p2,a);

[0025] Based on the large language model, Pr(r|p1,a), Pr(r|p2,a′) and Pr(r|p2,a) in the logical concept graph are estimated to obtain the self-assessment score.

[0026] Furthermore, when performing self-scoring, some binary classification tasks are designed for the large language model to simplify the probability that the argument and premise satisfy the same logical relationship to c(r|p1,a), c(r|p2,a′) and c(r|p2,a)∈{0,1};

[0027] The calculation formula for the overall logic modification score of the question based on the large language model is:

[0028] s=c(r|p1,a)×c(r|p2,a′)-c(r|p2,a)×c(r|p2,a′).

[0029] Furthermore, before the self-evaluation of whether the generated questions are logical based on the large language model, the following is also included:

[0030] Split the logical reasoning evaluation scenario into several discrimination tasks in advance, and pre-train the large language model through the logical concept graphs of these tasks;

[0031] The pre-trained large language model calculates the rating of complex reasoning tasks based on the structure of the current logical concept graph, that is, calculates the final self-assessment score.

[0032] In a second aspect, the present invention provides a system for generating and evaluating questions based on counterfactual logic modification, which is applicable to the above-mentioned method for generating and evaluating questions based on counterfactual logic modification, and the system comprises:

[0033] The large language model training module is used to obtain a reading comprehension data set that requires logical reasoning, annotate it, obtain a training set and a test set; and use the training set to train the pre-built large language model;

[0034] A question generation module, used to split the questions in the test set into arguments and premises, modify the premises with counterfactual logic based on the trained large language model, and generate questions;

[0035] The scoring module is used to self-score whether the generated questions are logical based on the large language model.

[0036] Furthermore, the topic generation module includes:

[0037] The splitting unit is used to split the questions in the test set into an argument and two premises Premise1 and Premise2, where the argument comes from the original context, the premise Premise1 comes from the correct option, and the premise Premise2 comes from the annotation; the argument Argument and the premise Premise1 are related to each other through a logical relationship, forming a situation state1 with a logical relationship R;

[0038] The premise reverse modification unit uses the given premise Premise2 and the logical relationship R to modify the statements in the argument Argument, and obtains the modified argument Argument', forming the second state state2 with the logical relationship R;

[0039] The question generating unit is used for switching the situation 1 state1 to the situation 2 state2, and taking the modified argument Argument' in the situation 2 state2 as an option in the multiple-choice question.

[0040] It can be seen from the above technical solutions that, compared with the prior art, the present invention has the following beneficial effects:

[0041] (1) Compared with the prior art methods that rely on synthetic reasoning, there may be problems with the large language model misunderstanding and misusing the logical relationships in reasoning. The present invention trains the large language model to modify counterfactual logic so that it can better adapt to the task of counterfactual logic modification, and can skillfully modify the argument text with high accuracy in processing logical relationships, so as to achieve the ability to establish satisfactory specified logical relationships.

[0042] (2) Compared with the existing methods that rely on the accuracy and informativeness of answers, the present invention can enable large language models to perform faster and more effective reasoning evaluations on complex reasoning tasks that are not easy to obtain standard or manual test answers through self-evaluation scores. It can quantitatively measure the evolution of information in sentences and evaluate whether the generated questions are logical, so as to ensure that the generated questions follow specific logical relationships, and enable the model to perform high-precision evaluations.

[0043] (3) The present invention utilizes a well-trained large language model for automatic question generation and self-evaluation, which is highly efficient and saves human resources, and can complete question generation and automatic evaluation more quickly and labor-savingly. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0045] Figure 1 A flowchart of a method for generating and evaluating a question based on counterfactual logic modification provided by the present invention;

[0046] Figure 2 A schematic diagram of reverse modification of the premise provided by the present invention;

[0047] Figure 3 A conceptual diagram of counterfactual logic provided by the present invention;

[0048] Figure 4 A schematic diagram of an annotation example provided by the present invention;

[0049] Figure 5 A schematic diagram of an example of generating a question provided by the present invention;

[0050] Figure 6 Schematic diagram of the evaluation results of the tests conducted on LLaMA and LLaMA2 provided by the present invention. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] like Figure 1As shown, the embodiment of the present invention discloses a method for generating and evaluating questions based on counterfactual logic modification, comprising the following steps:

[0053] S1. Obtain a reading comprehension dataset that requires logical reasoning, annotate it, and obtain a training set and a test set;

[0054] Use the training set to train a pre-built large language model;

[0055] S2. Split the questions in the test set into arguments and premises, modify the premises’ counterfactual logic based on the trained large language model, and generate questions;

[0056] S3. Based on the large language model, self-evaluate whether the generated questions are logical.

[0057] The above steps are further explained below.

[0058] S1. Select a reading comprehension dataset that requires logical reasoning, such as ReClor, as the source data. Given a question with context (argument text or background), question, options and correct answer, annotate the question, and obtain the wrong options and the corresponding modified context to form data points as a training sample.

[0059] The specific process of annotating the topic includes:

[0060] Determine the internal logical relationship of the problem;

[0061] Select an incorrect option and edit the context by deleting, adding, or replacing the text span in the statement to obtain the revised argument. As long as the statement partition is preserved, the number of versions and the length of the edited text span are not limited. Specifically, select one from the incorrect options as the target option, modify the sentences in the context related to the target option node so that the target option is established; check whether the target option in the existing context is correct, if correct, the annotation is completed; if incorrect, iteratively modify the node sentences related to the last modified node until the target option is established. In the logical reasoning graph, replace the node of the original correct option with the target option, and modify the logical relationship.

[0062] The questions and comments are then post-processed to correspond to each data point: Argument comes from the original context, Premise1 comes from the correct option, Premise2 and Argument' come from the comments.

[0063] Finally, the large language model is trained using the annotated training set.

[0064] S2. Split the questions in the test set into arguments and premises, and modify the premises’ counterfactual logic based on the trained large language model to generate questions. Figure 2 As shown, specifically including:

[0065] The questions in the test set are divided into an argument and two premises Premise1 and Premise2. The large language model needs to modify the statements in the argument so that the logical relationship R switches from state1 to state2.

[0066] Among them, the argument Argument comes from the original context, the premise Premise 1 comes from the correct option, and the premise Premise 2 comes from the annotation; the argument Argument and the premise Premise 1 are related to each other through a logical relationship, forming a situation state1 with a logical relationship R; the logical relationship R includes four types, namely: R1: the premise provides necessary assumptions for the argument; R2: the premise provides sufficient assumptions for the argument; R3: the premise strengthens the argument; R4: the premise weakens the argument.

[0067] The additional premise Premise2 will disrupt the logical relation R. The large language model uses the given premise Premise2 and the logical relation R to modify the sentence in the argument Argument, and obtains the modified argument Argument', forming the second situation state2 where the logical relation is R;

[0068] Switch state1 to state2, and use the modified argument 'Argument' in state2 as an option in the multiple-choice question.

[0069] S3: Based on the large language model, the generated questions are evaluated for logical consistency, including:

[0070] One method is formed by using premise p1 (Premise1) and argument a (Argument) that satisfies the logical relationship r (i.e. R), namely Method 1: Pr(r|p1,a); the other group requires premise P2 (Premise2) and the modified argument a' (Argument') to satisfy the same logical relationship r, i.e. Pr(r|p2,a') is close to 1; on the contrary, the relationship between premise Premise2 and argument Argument should not satisfy the logical relationship r, i.e. Pr(r|p2,a) tends to 0;

[0071] Based on the premise Premise1, the argument Argument, the premise Premise2, the modified argument Argument', the logical relationship, and Pr(r|p1,a), Pr(r|p2,a') and Pr(r|p2,a), we construct Figure 3 The logical concept diagram shown;

[0072] After that, the self-evaluation score is decomposed to obtain the self-evaluation score based on the complex logical reasoning ability demonstrated by the current large language model. Based on the large language model, Pr(r|p1,a), Pr(r|p2,a′) and Pr(r|p2,a) in the logical concept graph are estimated.

[0073] Specifically: design some binary classification tasks for the large language model to simplify the probability that the argument and premise satisfy the same logical relationship to c(r|p1,a), c(r|p2,a′) and c(r|p2,a)∈{0,1}, as shown in Table 1 below, where “no” is 0 and “yes” is 1.

[0074] Table 1

[0075]

[0076] The formula for calculating the overall logic modification score for the question is:

[0077] s=c(r|p1,a)×c(r|p2,a′)-c(r|p2,a)×c(r|p2,a′).

[0078] For a question, multiple evaluations are often required, that is, multiple calculations of the s value (the number of calculations depends on the complexity of the question) to form a batch. Finally, all the s values ​​in this batch are summed up and divided by the size of the batch, and the average value obtained is used as the final self-assessment score.

[0079] In one embodiment, before the self-scoring of whether the generated question is logical based on the large language model, the method further includes:

[0080] The logical reasoning evaluation scenario is pre-divided into several discrimination tasks equivalent to the original scenario, and the logical concept graphs of these tasks (such as Figure 3 As shown) Pre-train the large language model;

[0081] The pre-trained large language model calculates the rating of complex reasoning tasks based on the structure of the current logical concept graph, that is, calculates the final self-assessment score.

[0082] The present invention also provides a system for generating and evaluating questions based on counterfactual logic modification, which is applicable to the above-mentioned method for generating and evaluating questions based on counterfactual logic modification, and the system comprises:

[0083] The large language model training module is used to obtain the reading comprehension data set that requires logical reasoning, annotate it, and obtain the training set and test set; use the training set to train the pre-built large language model;

[0084] The question generation module is used to split the questions in the test set into arguments and premises, and to modify the premises’ counterfactual logic based on the trained large language model to generate questions.

[0085] The scoring module is used to self-score whether the generated questions are logical based on the large language model.

[0086] Specifically, the topic generation module includes:

[0087] The splitting unit is used to split the questions in the test set into an argument and two premises Premise1 and Premise2, where the argument comes from the original context, the premise Premise1 comes from the correct option, and the premise Premise2 comes from the annotation; the argument Argument and the premise Premise1 are related to each other through a logical relationship, forming a situation state1 with a logical relationship R;

[0088] The premise reverse modification unit uses the given premise Premise2 and the logical relationship R to modify the statements in the argument Argument, and obtains the modified argument Argument', forming the second state state2 with the logical relationship R;

[0089] The question generating unit is used for switching the situation 1 state1 to the situation 2 state2, and taking the modified argument Argument' in the situation 2 state2 as an option in the multiple-choice question.

[0090] The method of the present invention is further described below with reference to a specific example.

[0091] 1. Model training:

[0092] (1) Construction of the CloMos dataset:

[0093] The 6128 questions in the ReClor dataset were annotated by relevant experts to construct the CloMos dataset for training large language models. Figure 4 As shown, this is an example of annotation. After the annotation is completed, the large language model is trained. Here, GPT-4 is selected as the experimental object.

[0094] (2) Reverse modification of the premise

[0095] Select 50 questions, split them into arguments and premises, take the knowledge points to be tested as premises, and take the wrong knowledge points as premises, and let the large language model perform counterfactual logical modifications to the premises, such as Figure 5 The following is an example of generating a question.

[0096] It is obvious that the revised statement 3 can appear as an incorrect option in the multiple-choice question.

[0097] 2. Self-assessment:

[0098] To align the evaluation of large language models with human evaluation, 50 samples were selected from the test set and checked to see how well the model self-assessment scores matched the human evaluation.

[0099] Specifically, GPT-4 was used to implement SES. Experts were recruited to evaluate the 50 selected modified arguments, with a score of 1 for good answers and 0 for bad answers. The Cohen's Kappa coefficient between annotators was κ = 0.6785, indicating the consistency of human opinions. The same set of modified arguments was then evaluated again using the large language model, and the Cohen's Kappa coefficient was found to be κ = 0.4391, which shows that the self-evaluation scores are consistent with human evaluations, so more effective modifications can be made using self-evaluation scores.

[0100] First, we evaluated the large language models GPT-3.5-Turbo and GPT-4 to generate new topics. We also recruited two experts in related fields to simultaneously test their performance as humans. The results are shown in Table 2 below:

[0101] Table 2

[0102]

[0103]

[0104] NA: necessary assumption; SA: sufficient assumption; S: strengthen; W: weaken. The values ​​in the table are the self-assessment scores. The closer to 0, the worse; the closer to 1, the better. The values ​​are in the interval [0,1].

[0105] In the results, we can see that the overall performance of human experts on the CloMo task is 0.542, which shows that this is a very challenging task. In terms of the four logic-related performances, strengthening and weakening are relatively easy, while the other two are more challenging for humans. In addition, GPT-4's performance is slightly worse than that of humans, but it also shows strong counterfactual logical reasoning ability. GPT-3.5-Turbo's performance is not as good as the previous two.

[0106] Afterwards, we tested it on LLaMA and LLaMA2, splitting the CloMo dataset into 60% / 20% / 20% as training / development / test datasets, and fine-tuned the model using CloMo’s training data. We compared the results with the inference-only setting. The evaluation results are shown in Figure 2. Figure 6 as shown.

[0107] Figure 6 The yellow in the middle represents the result of fine-tuning, and the orange represents the result of inference only. CoT, or Chain-of-thoughtsetting: first provide a question, then remind the model to think step by step in the prompt; Few-shot: give four demonstrations in the prompt (randomly selected from the CLoMo training set); then provide a new question to the large language model; Zero-shot: directly provide the language model with the problem to be solved.

[0108] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0109] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for question generation and evaluation based on counterfactual logic modification, characterized in that: The following steps are involved: Obtain a reading comprehension dataset that requires logical reasoning, annotate it, and obtain a training set and a test set; Using the training set to train a pre-built large language model; Splitting the questions in the test set into arguments and premises, performing counterfactual logic modification on the premises based on the trained large language model, and generating questions; Based on the large language model, the generated questions are evaluated for logical consistency.

2. The method for question generation and evaluation based on counterfactual logic modification according to claim 1 is characterized in that: After obtaining the reading comprehension dataset, given a question with context, questions, options and correct answers, the question is annotated to obtain the wrong options and the corresponding modified context to form data points as a training sample.

3. The method for question generation and evaluation based on counterfactual logic modification according to claim 2 is characterized in that: The process of annotating a topic involves: Determine the internal logical relationship of the problem; Selecting an incorrect option edits the context by deleting, adding, or replacing spans of text in the statement to arrive at a revised argument.

4. The method for question generation and evaluation based on counterfactual logic modification according to claim 1, characterized in that: The questions in the test set are divided into arguments and premises, and the counterfactual logic of the premises is modified based on the trained large language model to generate questions, including: The questions in the test set are divided into an argument and two premises, Premise 1 and Premise 2. The argument comes from the original context, Premise 1 comes from the correct option, and Premise 2 comes from the annotation. The argument and Premise 1 are related to each other through a logical relationship, forming a situation with a logical relationship R, state 1. Use the given premise Premise2 and logical relation R to modify the statements in the argument Argument to obtain the modified argument Argument', forming the second state state2 with the logical relation R; Switch state1 to state2, and use the modified argument 'Argument' in state2 as an option in the multiple-choice question.

5. The method for question generation and evaluation based on counterfactual logic modification according to claim 4 is characterized in that: There are four types of logical relations R: R1: The premise provides necessary assumptions for the argument; R2: The premise provides sufficient assumptions for the argument; R3: The premise strengthens the argument; R4: The premise weakens the argument.

6. The method for question generation and evaluation based on counterfactual logic modification according to claim 5, characterized in that: The self-scoring of whether the generated questions are logical based on the large language model includes: One method is formed by using the premise Premise 1 and the argument Argument that satisfies the logical relation R, i.e. Pr(r|p1,a); the other group requires that the premise Premise 2 and the modified argument Argument' satisfy the same logical relation R, i.e. Pr(r|p2,a') is close to 1; the relationship between the premise Premise 2 and the argument Argument should not satisfy the logical relation R, i.e. Pr(r|p2,a) tends to 0; Construct a logical concept map based on Premise 1, Argument, Premise 2, the modified Argument', logical relations, and Pr(r|p1,a), Pr(r|p2,a'), and Pr(r|p2,a); Based on the large language model, Pr(r|p1,a), Pr(r|p2,a′) and Pr(r|p2,a) in the logical concept graph are estimated to obtain the self-assessment score.

7. The method for question generation and evaluation based on counterfactual logic modification according to claim 6, characterized in that: When performing self-scoring, design some binary classification tasks for the large language model to simplify the probability that the argument and premise satisfy the same logical relationship to c(r|p1,a), c(r|p2,a′) and c(r|p2,a)∈{0,1}; The calculation formula for the overall logic modification score of the question based on the large language model is: s=c(r|p1,a)×c(r|p2,a′)-c(r|p2,a)×c(r|p2,a′).

8. The method for question generation and evaluation based on counterfactual logic modification according to claim 7, characterized in that: Before the self-evaluation of whether the generated questions are logical based on the large language model, it also includes: Split the logical reasoning evaluation scenario into several discrimination tasks in advance, and pre-train the large language model through the logical concept graphs of these tasks; The pre-trained large language model calculates the rating of complex reasoning tasks based on the structure of the current logical concept graph, that is, calculates the final self-assessment score.

9. A question generation and evaluation system based on counterfactual logic modification, characterized in that: The method for generating and evaluating a question based on counterfactual logic modification as claimed in any one of claims 1 to 8 comprises: The large language model training module is used to obtain a reading comprehension data set that requires logical reasoning, annotate it, obtain a training set and a test set; and use the training set to train the pre-built large language model; A question generation module, used to split the questions in the test set into arguments and premises, modify the premises with counterfactual logic based on the trained large language model, and generate questions; The scoring module is used to self-score whether the generated questions are logical based on the large language model.

10. The question generation and evaluation system based on counterfactual logic modification according to claim 8, characterized in that: The topic generation module comprises: The splitting unit is used to split the questions in the test set into an argument and two premises Premise1 and Premise2, where the argument comes from the original context, the premise Premise1 comes from the correct option, and the premise Premise2 comes from the annotation; the argument Argument and the premise Premise1 are related to each other through a logical relationship, forming a situation state1 with a logical relationship R; The premise reverse modification unit uses the given premise Premise2 and the logical relation R to modify the statements in the argument Argument, and obtains the modified argument Argument', forming the second state state2 with the logical relation R; The question generating unit is used for switching the situation 1 state1 to the situation 2 state2, and taking the modified argument Argument' in the situation 2 state2 as an option in the multiple-choice question.

Images