Large model knowledge distillation method based on anti-fact data enhancement and multi-view thinking chain

Abstract

The invention discloses a large model knowledge distillation method based on anti-fact data enhancement and a multi-view thinking chain. According to the method, firstly, key features of text semantics are obtained through the subject terms and the grammar analysis tree, high-quality anti-fact data are automatically generated through the large language model, then the diversity of the anti-fact data is enhanced through the multi-view thinking chain, and the reasoning ability of the small language model is improved. Compared with the prior art, the large-model anti-fact distillation framework provided by the invention can enhance the robustness and generalization ability of small-model reasoning, and has excellent performance on small language models with different architectures and data sets with different distributions.

Description

Technical Field

[0001] The present invention belongs to the field of large model knowledge distillation, and specifically relates to how to efficiently transfer the reasoning capability of a large language model to a small language model. Background Art

[0002] Large model knowledge distillation is the process of migrating the knowledge and reasoning capabilities of large models to small models. It is of great significance in promoting the popularization of artificial intelligence and resource optimization. First, through distillation, small models can retain the core knowledge and complex reasoning capabilities of large models, so that high-performance intelligent applications can be achieved in environments with limited computing resources. This not only reduces the computing and storage costs of the model, but also enables AI applications to be more widely deployed in edge devices and mobile terminals, improving the universality of the model in actual scenarios.

[0003] However, in the existing technology, it is difficult to efficiently transfer the knowledge and reasoning capabilities of a large model to a small model. Currently, the performance of a small model after knowledge distillation is usually difficult to meet the expected requirements, which is mainly related to the way of knowledge distillation. Therefore, it is urgent to provide a method that can more effectively distill the capabilities of a large model into a small model. Summary of the invention

[0004] The purpose of the present invention is to solve the defects of large model knowledge distillation in the prior art and provide a large model knowledge distillation method based on counterfactual data enhancement and multi-perspective thinking chain.

[0005] The technical solutions specifically adopted in the present invention are as follows:

[0006] In a first aspect, the present invention provides a large model knowledge distillation method based on counterfactual data enhancement and multi-perspective thinking chain, which comprises the following steps:

[0007] S1: Input the question-answering text consisting of questions, options, and answers into the large language model, and combine the large language model and syntactic analysis to identify phrases related to causal features from the question text;

[0008] S2: Replace the phrases related to the causal features in the question text with masked characters, and modify the answer as the task label at the same time. Then, drive the large language model with prompts with examples to complete all the masked characters in the question text based on the modified answer, thereby generating a new question-answer text that is highly similar to the original question-answer text but with the task label modified as a counterfactual example.

[0009] S3: For different question-answer texts, S1 to S2 are repeated continuously to obtain a data set to be screened consisting of a series of counterfactual examples; the data set to be screened is filtered using the self-consistency of the large language model. For each counterfactual example in the data set to be screened, the large language model generates a specified number of reasoning paths. The counterfactual examples whose path ratios of reasoning results consistent with the actual task labels are not less than a threshold are retained in the new data set, and the remaining counterfactual examples are discarded;

[0010] S4: For each question and answer text in the new dataset, a multi-perspective thinking chain is generated for each question and answer text through the large language model, and the multi-perspective thinking chain includes a positive thinking chain for the correct answer option and a negative thinking chain for each incorrect answer option; finally, the question and answer text is expanded into a dataset containing multi-perspective thinking chains and used to train the small language model.

[0011] As a preferred embodiment of the first aspect, in S1, the method for identifying phrases related to causal features from the question text is as follows:

[0012] S11: Use the context learning capability of the large language model driven by prompts to obtain the key words in the question text;

[0013] S12: Using a syntactic analysis tool to obtain all noun phrases in the question text, and removing personal pronouns and possessive pronouns therein, the remaining noun phrases together with the subject words are taken as phrases related to the causal feature.

[0014] As a preferred embodiment of the first aspect, in S2, each phrase related to the causal feature in the question text is replaced by the characters “[MASK]”.

[0015] As a preferred embodiment of the first aspect, in S2, the answer serving as the task label is modified to any other option in the options of the original question and answer text except the correct answer option.

[0016] As a preferred embodiment of the above-mentioned first aspect, in S2, the prompt with examples includes examples and instructions, wherein the example consists of an exemplary question text, options, expected answers and question completion results, wherein phrases related to causal features in the question text are replaced with masked characters in advance; and the instructions are task introductions input into the large language model, which are used to prompt the large language model to complete the masked characters in the context of given options, so that the correct answer corresponding to the completed question text in the given options is consistent with the expected answer.

[0017] As a preferred embodiment of the first aspect, in S3, the method of filtering the to-be-screened data set by utilizing the self-consistency of the large language model to generate a new data set is as follows:

[0018] For each counterfactual example in the data set to be screened, the questions and options in the corresponding new question and answer text are input into the large language model, and the large language model repeats multiple rounds of reasoning. Each reasoning of the large language model will generate a reasoning path and then select an answer option from the options, and judge whether the answer option selected in each reasoning result is consistent with the task label in the new question and answer text; if the consistency rate of all reasoning rounds is not lower than the preset threshold, the counterfactual example will be retained in the new data set, otherwise the counterfactual example will be eliminated.

[0019] As a preferred embodiment of the first aspect, in S3, when the to-be-screened data set is filtered using the self-consistency of the large language model, each counterfactual example is repeatedly inferred at least 5 times by the large language model.

[0020] As a preferred embodiment of the first aspect, the consistency rate threshold for determining whether the counterfactual example is retained in the new data set is 60%.

[0021] As a preferred embodiment of the above-mentioned first aspect, in S4, when a positive thinking chain is generated for each question and answer text through the large language model, it is necessary to drive the large language model through prompts to output a thinking chain for the current question to obtain a correct answer by reasoning; when a negative thinking chain is generated for each question and answer text through the large language model, it is necessary to drive the large language model through prompts to output a thinking chain for the current question to obtain each incorrect answer option in the options that is not a correct answer by reasoning.

[0022] In the second aspect, the present invention provides a small language model training method, which utilizes the large language model knowledge distillation method based on counterfactual data enhancement and multi-perspective thinking chain as described in any of the schemes in the first aspect above, generates the new data set as training data, and trains the small language model with the assistance of the multi-perspective thinking chain so that it can correctly implement question-answering reasoning.

[0023] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0024] The present invention provides a large model knowledge distillation method based on counterfactual data enhancement and multi-view thinking chain. The method first uses subject words and grammatical analysis trees to obtain key features of text semantics, and uses a large language model to automatically generate high-quality counterfactual data; then uses multi-view thinking chains to enhance the diversity of training data and improve the reasoning ability of small language models. Compared with the prior art, the large model counterfactual distillation framework proposed in the present invention can enhance the robustness and generalization ability of small model reasoning, and has achieved excellent performance on small language models of different architectures and data sets with different distributions. The present invention significantly outperforms the baseline on the same distribution data set, and the reasoning accuracy is improved by an average of 11.43%. On three data sets with different distributions, the reasoning accuracy of the present invention is improved by an average of 14.9% relative to the baseline. In various parameter scales (from 120M to 770M) and model structures (from decoder only to encoder-decoder) of the small model, the present invention always shows excellent performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 The overall architecture diagram of the large model knowledge distillation method based on counterfactual data enhancement and multi-perspective thinking chain provided by the present invention;

[0026] Figure 2 This is a diagram showing the experimental effect of an embodiment of the present invention on the out-of-distribution dataset CSQA;

[0027] Figure 3 This is a diagram showing the experimental results of an embodiment of the present invention on the out-of-distribution dataset QuaRel;

[0028] Figure 4 This is a diagram showing the experimental effect of an embodiment of the present invention on the out-of-distribution data set QASC;

[0029] Figure 5 It is a diagram showing the experimental effect of an embodiment of the present invention on the out-of-distribution dataset ARC. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0031] like Figure 1 As shown, in one embodiment of the present invention, a large model knowledge distillation method based on counterfactual data enhancement and multi-perspective thinking chain is provided, and its steps are shown in S1 to S4:

[0032] S1: The question-answering text consisting of questions, options and answers is input into the large language model (LLM), and the large language model and syntax analysis are combined to identify phrases related to causal features from the question text.

[0033] Through the above step S1, phrases related to the causal feature can be identified, and then the causal relationship in the original text of the question text can be destroyed through the subsequent MASK operation. In order to realize the identification of phrases related to the causal feature, the present invention proposes a method involving subject words and syntactic analysis to extract phrases related to the causal feature in the text, and the specific method is as follows:

[0034] S11: Prompt drives the large language model to use its context learning ability to obtain the topic words in the question text.

[0035] S12: Using a syntactic analysis tool to obtain all noun phrases in the question text, and removing personal pronouns and possessive pronouns therein, the remaining noun phrases together with the subject words are taken as phrases related to the causal feature.

[0036] It should be noted that noun phrases are used to introduce, describe and connect concepts in a text, thereby enhancing the expressiveness and accuracy of the text. Therefore, noun phrases are important elements that constitute causal features. The present invention uses a syntactic analysis tool to obtain noun phrases in a text, but in order to retain the elements in the original text as much as possible, personal pronouns and possessive pronouns are retained.

[0037] S2: Replace the phrases related to the causal features in the question text with mask characters, and modify the answer as the task label. Then, drive the large language model with prompts with examples to complete all the mask characters in the question text based on the modified answer, thereby generating a new question-answer text that is highly similar to the original question-answer text but with the task label modified as a counterfactual example.

[0038] In the above step S2, since it is necessary to drive the large language model through prompts with examples, it is necessary to pre-build prompts for contextual learning of the large model. In an embodiment of the present invention, in the above step S2, the prompts with examples include examples and instructions, wherein the example consists of an exemplary question text, options, expected answers, and question completion results, wherein the phrases related to causal features in the question text are replaced with masked characters in advance; and the above instructions are task introductions input into the large language model, which are used to prompt the large language model to complete the masked characters in the context of a given option, so that the correct answer corresponding to the completed question text in the given option is consistent with the expected answer. Therefore, the prompt finally constructed consists of five parts: question, option, expected answer, completion text, and instruction.

[0039] In an embodiment of the present invention, in the above step S2, each phrase related to the causal feature in the question text is replaced with the character "[MASK]". In addition, the answer used as the task label can be modified to any other option in the options of the original question and answer text except the correct answer option. Therefore, after replacing the phrase related to the causal feature in the question text with the mask character, it can be spliced ​​together with the constructed prompt and input into the large language model. Since the answer used as the task label has been replaced, the large language model will complete the question text after the Mask operation in accordance with the practice in the example, that is, according to the current answer used as the task label, the text of each character "[MASK]" position is replaced, so that the correct answer of the completed question text can match the current answer used as the task label. The new question and answer text thus generated can be used as a counterfactual example.

[0040] For example, see Figure 1 As shown in the above S1 step, the original question-answering text consists of questions, options and answers, where the question is "What might be the result if one is convicted of obstructing justice?", there are four options, namely "A: injustice; B: criminal charges; C: fear; D: going to jail", and the answer is "D: going to jail". After extracting phrases related to causal features in the text by combining the subject word and syntactic analysis in S1, the subject word "obstructing justice" and two other noun phrases "the result" and "one" are obtained. Therefore, after replacing these three phrases with mask characters, the mask text "What might be [MASK] if [MASK] is convicted of [MASK]?" is obtained. Then, the prompt with examples is constructed, which includes five parts: question text, options, expected answers, question completion results, and prompts, as follows:

[0041] Question Text: Question:Aside from [MASK] what does your [MASK] need?

[0042] Options: A:bone B:charm C:lots of attention D:walked

[0043] Expected answer: lots of attention

[0044] Question completion results:Aside from water and nourishment what does your dog need?

[0045] Tip: Multiple choice questions consist of questions, options and answers. Based on the above example, please complete the [MASK] part of the question to make it a multiple-choice question with smooth semantics and clear logic.

[0046] Therefore, after the above mask text and the prompt with examples are concatenated and input into the large language model, the following question completion results will be obtained in the following format:

[0047] Question Text: What might be [MASK] if [MASK] is convicted of [MASK]?

[0048] Options:A:injustice B:criminal charges C:fear D:going to jail

[0049] Expected answer: injustice

[0050] Question completion result:What might be a concern for society if an innocent person is convicted of a crime?

[0051] The question completion result can be reassembled with the options and expected answers to form a new question-answer text and added to the dataset of counterfactual examples.

[0052] S3: For different question-answer texts, S1 to S2 are repeated continuously to obtain a data set to be screened consisting of a series of counterfactual examples; the self-consistency of the large language model is used to filter the data set to be screened. For each counterfactual example in the data set to be screened, the large language model generates a specified number of reasoning paths. The counterfactual examples whose proportion of paths consistent with the actual task labels is not less than the threshold are retained in the new data set, and the remaining counterfactual examples are discarded.

[0053] It should be noted that the above process of generating a reasoning path for each counterfactual example by the large language model only requires inputting the question text and options in this counterfactual example into the large language model, and the large language model itself generates a reasoning path to make a judgment, that is, to select the correct answer it believes from all options.

[0054] In order to obtain high-quality counterfactual data, the present invention designs an evaluation strategy that aims to select the most promising counterfactual examples to eliminate potential errors. This strategy is used to verify whether the generated completed questions match the new answers. The large language model has self-consistency, that is, its correct answer selection for a question is generally consistent. Based on the self-consistency principle of the large language model, the present invention can generate a series of reasoning paths for each counterfactual example, and then judge the degree of consistency of its answers, and then screen the counterfactual examples.

[0055] Therefore, in the embodiment of the present invention, in the above step S3, the method of filtering the to-be-screened data set by utilizing the self-consistency of the large language model to generate a new data set is as follows:

[0056] For each counterfactual example in the dataset to be screened, the questions and options in the corresponding new question and answer text are input into the large language model, and the large language model repeats multiple rounds of reasoning. Each reasoning of the large language model will generate a reasoning path and then select an answer option from the options, and judge whether the answer option selected in each reasoning result is consistent with the task label in the new question and answer text (i.e., the expected answer mentioned above). If the consistency rate of all reasoning rounds is not lower than the preset threshold, the counterfactual example will be retained in the new dataset, otherwise the counterfactual example will be removed.

[0057] After each reasoning, the answer option can be compared with the task label in the new question and answer text (i.e., the expected answer mentioned above). If they are consistent, they are considered to be correct answers, and if they are inconsistent, they are considered to be wrong answers. Therefore, the consistency rate of all reasoning rounds can be regarded as the correct answer rate. In an embodiment of the present invention, when the self-consistency of the large language model is used to filter the data set to be screened, the number of reasoning rounds repeated by the large language model for each counterfactual example can be adjusted according to actual needs. In addition, the consistency rate threshold used to determine whether the counterfactual example is retained in the new data set can be adjusted according to actual needs. In general, each counterfactual example is repeatedly reasoned by the large language model at least 5 times, which should not be too few. The higher the consistency rate threshold, the stricter the retention standard for the counterfactual example, the fewer counterfactual examples are finally retained, and the higher the quality; conversely, the lower the threshold, the looser the retention standard for the counterfactual example, the more counterfactual examples are finally retained, and the worse the quality. In an embodiment of the present invention, the number of reasoning rounds can be set to 5 times, and the consistency rate threshold can be set to 60%.

[0058] For example, see Figure 1 As shown in the figure, the completion result of the above question "What might be the result if one is convicted of obstructing justice?" corresponds to the following options: "A: injustice B: criminal charges C: fear D: going to jail", and the expected answer (i.e., the correct answer) is: injustice. For this new question and answer text, the question and options can be input into the large language model, and the large language model can output the answer options it thinks, and then determine whether the answer options are consistent with the expected answers, which is considered as one round. After executing 5 rounds in total, if the statistical consistency rate exceeds 60%, that is, the number of times the answer options are consistent with the expected answers exceeds 3 times, the new question and answer text can be retained in the new data set, otherwise the new question and answer text needs to be deleted.

[0059] S4: For each question and answer text in the new dataset, a multi-perspective thinking chain is generated for each question and answer text through the large language model, and the multi-perspective thinking chain includes a positive thinking chain for the correct answer option and a negative thinking chain for each incorrect answer option; finally, the question and answer text is expanded into a dataset containing multi-perspective thinking chains and used to train the small language model.

[0060] In an embodiment of the present invention, in the above step S4, when a positive thinking chain is generated for each question and answer text through the large language model, it is necessary to drive the large language model through prompts to output the thinking chain for the current question to obtain the correct answer; when a negative thinking chain is generated for each question and answer text through the large language model, it is necessary to drive the large language model through prompts to output the thinking chain for the current question to obtain each incorrect answer option in the options that is not the correct answer.

[0061] The multi-perspective thinking chain generated for each question-answering text consists of two parts: a positive thinking chain generated for the correct option to support the option, and a negative thinking chain generated for the wrong option to represent refutational evidence, both of which are directly generated by the large language model through context learning. The goal of the positive thinking chain is to explore the relevant information of the correct answer, while the negative thinking chain is more focused on generating negative reasoning paths for each option (except the correct answer). This method can help small models distinguish and eliminate inappropriate options among multiple choices, enabling them to select the correct answer more specifically.

[0062] Continuing with the above new Q&A text as an example, see Figure 1As shown in the figure, in the new question-answering text, the completion result of the above question "What might be the result if one is convicted of obstructing justice?" corresponds to the following options: "A: injustice B: criminal charges C: fear D: going to jail", and the expected answer (i.e., the correct answer) is: injustice. For this new question-answering text, the large language model can output the reason why it thinks option A is the correct answer as a positive thinking chain: "If an innocent person is convicted of a crime, one of the main concerns for society would be the issue of injustice. This canerode public trust in the legal system and have far-reaching consequences for both the individual wrongfully convicted and society as a whole. Therefore, the answer is A.", and at the same time output the reason why it thinks options B, C, and D are not the correct answers as a negative thinking chain. Taking the negative thinking chain B as an example, its output content is "The question is asking about a concern for society if an innocent person is convicted of a crime. Criminal charges would not be a concern in this scenario because the person has already been convicted. Therefore, the answer is not B." Therefore, the present invention can add positive thinking chains and negative thinking chains to the new question-answer text to form a sample for training a small language model. This method can help the small model distinguish and exclude insufficient options from multiple options, so that they can choose the correct answer more specifically.

[0063] The new data set generated by the above-mentioned large language model knowledge distillation method can be used as training data to train the small language model, so that the small language model can be trained with the assistance of a multi-perspective thinking chain so that it can correctly implement question-answering reasoning.

[0064] The following example demonstrates the effect of the new dataset generated by the large language model knowledge distillation method on the training of a small language model using different datasets.

[0065] Example

[0066] This example verifies the effect on four multi-step reasoning datasets: CommonsenseQA (CSQA), QuaRel, ARC, and QASC. The ARC dataset is divided into two parts: a challenge set and a simple set. In the experiment of this example, these two subsets are combined for model training. Since the test labels of the CSQA and QASC datasets are not public, the official development set is used as the test set. The experimental results are shown in Figure 2. Figure 2 , Figure 3 , Figure 4 , Figure 5 As shown in Table 1

[0067] Table 1

[0068]

[0069]

[0070] Table 1 summarizes the accuracy comparison of small models using the proposed method with existing FT and FT-CoT methods. The method of the present invention outperforms FT-CoT in models of different scales with parameters ranging from 120 million to 770 million. Specifically, the method of the present invention achieves performance improvements ranging from 4.17% to 23.22% on these scales, with an average improvement of 11.43%. In the QASC and ARC datasets, the distillation performance of small parameter models of the present invention surpasses that of large parameter models trained using FT-CoT. For example, in the ARC dataset, the present invention achieves 40.42% performance on GPT2-base, which is 3.27% higher than GPT2-medium trained using FT-CoT. In addition, the present invention is 8.57% higher on GPT2-medium than GPT2-large trained using FT-CoT.

[0071] The above is only a preferred implementation case of the present invention and does not limit the present invention in any form. Although the implementation process of the present invention is described in detail above, for those familiar with the art, they can still modify the technical solutions recorded in the above examples, or replace some of the technical features therein with equivalents. All modifications, equivalent replacements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A large model knowledge distillation method based on counterfactual data enhancement and multi-perspective thinking chain, characterized in that: The following steps are involved: S1: Input the question-answering text consisting of questions, options, and answers into the large language model, and combine the large language model and syntactic analysis to identify phrases related to causal features from the question text; S2: Replace the phrases related to the causal features in the question text with masked characters, and modify the answer as the task label at the same time. Then, drive the large language model with prompts with examples to complete all the masked characters in the question text based on the modified answer, thereby generating a new question-answer text that is highly similar to the original question-answer text but with the task label modified as a counterfactual example. S3: For different question-answer texts, S1 to S2 are repeated continuously to obtain a data set to be screened consisting of a series of counterfactual examples; the data set to be screened is filtered using the self-consistency of the large language model. For each counterfactual example in the data set to be screened, the large language model generates a specified number of reasoning paths. The counterfactual examples whose path ratios of reasoning results consistent with the actual task labels are not less than a threshold are retained in the new data set, and the remaining counterfactual examples are discarded; S4: For each question and answer text in the new dataset, a multi-perspective thinking chain is generated for each question and answer text through the large language model, and the multi-perspective thinking chain includes a positive thinking chain for the correct answer option and a negative thinking chain for each incorrect answer option; finally, the question and answer text is expanded into a dataset containing multi-perspective thinking chains and used to train the small language model.

2. The large language model knowledge distillation method based on counterfactual data enhancement and multi-perspective thinking chain as claimed in claim 1 is characterized in that: In S1, the method for identifying phrases related to causal features from the question text is as follows: S11: Use the context learning capability of the large language model driven by prompts to obtain the key words in the question text; S12: Using a syntactic analysis tool to obtain all noun phrases in the question text, and removing personal pronouns and possessive pronouns therein, the remaining noun phrases together with the subject words are taken as phrases related to the causal feature.

3. The large language model knowledge distillation method based on counterfactual data enhancement and multi-perspective thinking chain as claimed in claim 1 is characterized in that: In S2, each phrase related to the causal feature in the question text is replaced by the characters "[MASK]".

4. The large language model knowledge distillation method based on counterfactual data enhancement and multi-perspective thinking chain as claimed in claim 1, characterized in that: In S2, the answer as the task label is modified to any other option in the options of the original question and answer text except the correct answer option.

5. The large language model knowledge distillation method based on counterfactual data enhancement and multi-perspective thinking chain as claimed in claim 1, characterized in that: In S2, the prompt with examples includes examples and instructions, wherein the example consists of an exemplary question text, options, expected answers, and question completion results, wherein phrases related to causal features in the question text are replaced with masked characters in advance; the instructions are task introductions input to the large language model, which are used to prompt the large language model to complete the masked characters in the context of a given option, so that the correct answer corresponding to the completed question text in the given option is consistent with the expected answer.

6. The large language model knowledge distillation method based on counterfactual data enhancement and multi-perspective thinking chain as claimed in claim 1, characterized in that: In S3, the method of filtering the to-be-screened data set by utilizing the self-consistency of the large language model to generate a new data set is as follows: For each counterfactual example in the data set to be screened, the questions and options in the corresponding new question and answer text are input into the large language model, and the large language model repeats multiple rounds of reasoning. Each reasoning of the large language model will generate a reasoning path and then select an answer option from the options, and judge whether the answer option selected in each reasoning result is consistent with the task label in the new question and answer text; if the consistency rate of all reasoning rounds is not lower than the preset threshold, the counterfactual example will be retained in the new data set, otherwise the counterfactual example will be eliminated.

7. The large language model knowledge distillation method based on counterfactual data enhancement and multi-perspective thinking chain according to claim 6 is characterized in that: In S3, when the to-be-screened data set is filtered using the self-consistency of the large language model, each counterfactual example is repeatedly inferred at least 5 times by the large language model.

8. The large language model knowledge distillation method based on counterfactual data enhancement and multi-perspective thinking chain according to claim 7 is characterized in that: The consistency threshold for determining whether a counterfactual example is retained in the new dataset is 60%.

9. The large language model knowledge distillation method based on counterfactual data enhancement and multi-perspective thinking chain as claimed in claim 1, characterized in that: In S4, when a positive thinking chain is generated for each question and answer text through the large language model, it is necessary to drive the large language model through prompts to output the thinking chain for the current question to obtain the correct answer; when a negative thinking chain is generated for each question and answer text through the large language model, it is necessary to drive the large language model through prompts to output the thinking chain for the current question to obtain each incorrect answer option in the options that is not the correct answer.

10. A small language model training method, characterized in that: By using the large language model knowledge distillation method based on counterfactual data enhancement and multi-perspective thinking chain as described in any one of claims 1 to 9, the new data set is generated as training data, and the small language model is trained with the assistance of the multi-perspective thinking chain so that it can correctly implement question-answering reasoning.

Images