A self-correcting intelligent teaching assistance method based on instruction-guided large language model

Abstract

The present invention provides a self-correcting intelligent teaching assistance method based on an instruction-guided large language model, comprising: obtaining massive amounts of teaching-related data to perform distributed training on the large language model; extracting a portion of the labeled teaching data set as an instruction set through random sampling, and using the instruction set to fine-tune the large language model; inputting each sample question and corresponding related knowledge in the instruction set into the fine-tuned large language model multiple times, and outputting multiple candidate answers corresponding to the sample questions; then manually sorting the questions and candidate answers to train a sorting model, which is used to score the answers generated by the fine-tuned large language model; finally, updating the parameters of the fine-tuned large language model through a policy gradient reinforcement learning algorithm, thereby obtaining a large language model that provides relatively accurate answers. The present invention constructs an intelligent teaching assistance method based on a large language model, which can accurately understand teacher and student questions and ensure the accuracy of the generated answers.

Description

Technical Field

[0001] The present invention belongs to the technical field of teaching assistance, and in particular relates to a self-correcting intelligent teaching assistance method based on an instruction-guided large language model. Background Art

[0002] Intelligent teaching assistance refers to the use of artificial intelligence technologies, such as natural language processing, machine learning, knowledge graphs, etc., to provide users with personalized, intelligent, and efficient teaching assistance services, such as intelligent question and answer, intelligent homework grading, etc. Intelligent teaching assistance can effectively improve teaching quality and learning efficiency, and meet users' personalized learning needs.

[0003] Currently, a number of AI-based intelligent teaching assistance systems and platforms have emerged, such as Siri, Google Assistant, and Baidu Smart Mini-Programs. These systems and platforms are based on rules and semantic search. They first understand a question through semantic similarity search, find the most similar question, and then return the stored standard answer to the user.

[0004] Existing teaching aid systems require a lot of manual intervention and the preparation of question and answer libraries in advance, which leads to the following problems:

[0005] 1. As the times change, the knowledge content faced by students and teachers is also changing, so the question and answer database also needs to change.

[0006] 2. There may be semantic misunderstandings or inaccuracies in the questions asked by teachers and students. Because human expressions are always changing, two completely different statements may have the same meaning. However, if the question is misunderstood, the retrieved answers will definitely be wrong, which will mislead teachers and students.

[0007] 3. The answers to questions raised by teachers and students are mostly manually compiled and often templated. It is often the case that the answers and questions do not completely match. For example, there may be a "small question, big answer" situation, that is, the question itself is a small question, but the answer provides all the relevant knowledge. In the end, teachers and students still need to make decisions, which greatly reduces the role of assistance.

[0008] In addition, existing language models have a vast amount of knowledge reserves, but they cannot match human needs. This is reflected in two aspects: first, they cannot understand human expressions well; second, the generated responses are unfriendly, often containing question-related content rather than accurate and specific content. Summary of the Invention

[0009] In response to the above-mentioned deficiencies in the prior art, the present invention provides a self-correcting intelligent teaching assistance method based on an instruction-guided large language model. The present invention solves four problems: first, it has the knowledge background of all mankind; second, it can accurately understand the questions asked by teachers and students and generate answers specific to the questions; third, it ensures the accuracy of the generated answers; and fourth, it can automatically learn and has the ability to automatically upgrade.

[0010] The present invention provides a self-correcting intelligent teaching auxiliary method based on an instruction-guided large language model, comprising:

[0011] Obtain massive amounts of teaching-related data for distributed training of large language models;

[0012] Collect sample questions, question-related knowledge, and sample answers through the teaching platform to obtain a teaching dataset with labeled information;

[0013] Through random sampling, some sample questions and their corresponding relevant knowledge and sample answers are extracted from the annotated teaching dataset as an instruction set, and the instruction set is used to fine-tune the large language model;

[0014] The sample questions and corresponding related knowledge in the instruction set are input into the fine-tuned large language model multiple times, and multiple candidate answers corresponding to the sample questions are output;

[0015] By manually sorting candidate answers to sample questions, we obtain a training set with sorted candidate answers for the sample questions. We then train a sorting model, sortModel, based on the training set. The sorting model is used to score the answers generated by the fine-tuned large language model.

[0016] The remaining part of the annotated teaching dataset after removing the instruction set is used as the fine-tuning dataset to further fine-tune the answering ability of the fine-tuned large language model; the sample questions of the fine-tuning dataset are input into the fine-tuned large language model and the test answers are output, and then the sample questions and test answers are input into the sorting model sortModel. The sorting model sortModel gives a score for each test answer. Finally, the parameters of the fine-tuned large language model are updated through the policy gradient reinforcement learning algorithm, thereby obtaining a large language model with more accurate answers.

[0017] Furthermore, it also includes:

[0018] After the large language model has been used for a period of time, users are provided with feedback on the answers output by the large language model.

[0019] Iteratively train sortModel based on user feedback to obtain a stronger sorting model sortModel_update;

[0020] Based on the more accurate scoring ability of the sorting model sortModel_update, the reinforcement learning algorithm is used again to update the parameters of the large language model, enabling the large language model to complete the capability upgrade.

[0021] Furthermore, the relevant knowledge of the sample problem input during fine-tuning of the large language model is the knowledge points needed to answer this problem.

[0022] Furthermore, the sample questions are selected from the data retained during the teaching process. When selecting sample questions, each grade and subject should be covered to ensure the comprehensiveness of the data source and the randomness of the selection;

[0023] The sample answers are manually labeled by professional teaching staff based on specific sample questions, and the relevant knowledge of the sample questions should also be marked.

[0024] Furthermore, based on the ranking model's scoring capabilities, the parameters of the large language model are updated using a policy gradient reinforcement learning algorithm. The fine-tuning dataset used as input for reinforcement learning is the remaining data from the teaching dataset, excluding the instruction set used for initial fine-tuning. This remaining data also includes sample questions, relevant knowledge, and sample answers.

[0025] Furthermore, when the large language model is provided to teachers and students for use, it obtains real-time questions raised by teachers and students, then conducts knowledge retrieval on the Internet for the real-time questions, extracts knowledge from the retrieval results to obtain relevant knowledge of the real-time questions, and then inputs it into the large language model. Finally, the large language model gives accurate answers; based on the retrieval of relevant knowledge, the large language model has the ability to correct and will not produce illogically wrong answers.

[0026] The beneficial effects of this invention are: providing an intelligent teaching assistance method based on a large language model. By drawing on a vast knowledge base to construct a large language model, the method automatically answers questions from teachers and students. Through instructional guidance and reinforcement learning, the method assists teachers and students in obtaining answers to questions using a strong artificial intelligence model. While manual labeling of a small number of questions is required initially, the method subsequently relies solely on the large language model's inherent learning capabilities and the guidance of a ranking model for continuous evolution, achieving automated upgrades for teaching assistance tools and demonstrating cutting-edge capabilities.

[0027] In addition, in order to ensure the accuracy of knowledge, the teaching assistance system first accumulates a large amount of knowledge, then trains how to understand and answer questions asked by teachers and students, and then uses Internet information to assist with relevant knowledge, thereby ensuring the accuracy of the reply answers.

[0028] The present invention is aimed at teachers and students, can effectively improve teaching quality and learning efficiency, meet users' personalized learning needs, and promote users' acquisition and improvement of knowledge and skills. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0030] Figure 1 is a schematic flow chart of a method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the technical solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0032] The key terms appearing in the present invention are explained below.

[0033] Instruction guidance: Teachers and students express real problems, usually in the form of instructions, such as "Explain what reinforcement learning is", with the aim of allowing the model to understand human natural language expressions.

[0034] Large language model: A deep neural network model with over 10 billion parameters that can automatically generate the following text based on the previous text.

[0035] Self-correction: Adding relevant knowledge of Internet search questions to the input of the large language model and using relevant auxiliary models to understand the questions or provide answers can prevent the model from making up questions or answering non-factual information.

[0036] like Figure 1 As shown, an embodiment of the present invention provides a self-correcting intelligent teaching assistance method based on an instruction-guided large language model, comprising:

[0037] S1. Obtain massive amounts of teaching-related data for distributed training of large language models;

[0038] S2. Collect sample questions, question-related knowledge, and sample answers through the teaching platform to obtain a teaching dataset with annotated information;

[0039] S3. Randomly sample some sample questions and their corresponding knowledge and sample answers from the labeled teaching dataset as an instruction set, and use the instruction set to fine-tune the large language model.

[0040] S4: Input the sample questions and corresponding related knowledge in the instruction set into the fine-tuned large language model multiple times, and output multiple candidate answers corresponding to the sample questions;

[0041] S5. Manually sort the candidate answers to the sample questions to obtain a training set of the sample questions with the sorted candidate answers. A sorting model, sortModel, is trained based on the training set. The sorting model, sortModel, is used to score the answers generated by the fine-tuned large language model.

[0042] S6. The remaining part of the annotated teaching dataset after removing the instruction set is used as the fine-tuning dataset to further fine-tune the answering ability of the fine-tuned large language model; the sample questions of the fine-tuning dataset are input into the fine-tuned large language model and the test answers are output; the sample questions and test answers are then input into the sorting model sortModel, and the sorting model sortModel gives a score for each test answer; finally, the parameters of the fine-tuned large language model are updated through the policy gradient reinforcement learning algorithm, thereby obtaining a large language model with more accurate answers.

[0043] Optionally, as an embodiment of the present application, it also includes: after the large language model has been used for a period of time, receiving user feedback on the answers output by the large language model; iteratively training sortModel based on user feedback to obtain a stronger sorting model sortModel_update; based on the more accurate scoring ability of the sorting model sortModel_update, using the reinforcement learning algorithm again to update the parameters of the large language model, so that the large language model completes the capability upgrade.

[0044] Optionally, as an embodiment of the present application, the relevant knowledge of the sample question input when fine-tuning the large language model is the knowledge points or key steps required to answer the question.

[0045] Optionally, as an embodiment of the present application, the sample questions are screened from the data retained in the teaching process, and the sample questions are selected to cover each grade and subject to ensure the comprehensiveness of the data source and the randomness of the screening;

[0046] The sample answers are manually labeled by professional teaching staff based on specific sample questions, and the relevant knowledge of the sample questions should also be marked.

[0047] Optionally, as an embodiment of the present application, the parameters of the large language model are updated using a policy gradient reinforcement learning algorithm based on the scoring capability of the ranking model. The fine-tuning dataset input for reinforcement learning is the remaining data from the teaching dataset after removing the instruction set used for initial fine-tuning. This remaining data also includes: sample questions, question-related knowledge, and sample answers.

[0048] Optionally, as an embodiment of the present application, when the large language model is provided to teachers and students for use, real-time questions raised by teachers and students are obtained, and then knowledge retrieval is performed on the Internet for the real-time questions. Knowledge extraction is performed on the retrieval results to obtain relevant knowledge of the real-time questions, which is then input into the large language model. Finally, the large language model gives accurate answers; based on the retrieval of relevant knowledge, the large language model has the ability to correct and will not produce illogically wrong answers.

[0049] The embodiment of the present invention aligns the answers generated by the large language model with the questions asked by the teacher and students, ensuring the accuracy of the answers generated by the model, that is, the answers exactly match the questions. The specific process is as follows:

[0050] The large language model can generate answers based on questions, but the ranking model needs to further determine whether the answers exactly match the questions posed by teachers and students. Using reinforcement learning algorithms, the ranking model guides the large language model in generating more appropriate answers to questions.

[0051] If the large language model generates a poor answer, for example, giving a large answer to a small question, the resulting <question, answer> pair will be input into the ranking model and receive a low score. The large language model then updates its parameters using a gradient descent algorithm. The updated results from the large language model will result in a higher score for the ranking model, indicating that the generated answer is more appropriate for the question, effectively aligning the answer with the question.

[0052] The present invention is aimed at teachers and students, aiming to better assist teachers in their lesson preparation process, improve students' proactive learning ability, and truly assist the entire teaching process. To facilitate understanding of the present invention, the following further describes the self-correcting intelligent teaching assistance method provided by the present invention, in conjunction with the process of questioning and answering questions between teachers and students using the intelligent teaching assistance system in an embodiment.

[0053] Specifically, the self-correcting intelligent teaching auxiliary method includes:

[0054] Massive Data Acquisition: Using web crawler technology, we scrape vast amounts of data from the internet, including code, books, articles, and conversations, to create a dataset D0 containing over a trillion tokens. This data includes historical events, translations between different languages, common knowledge, and math, physics, and chemistry questions and answers. This knowledge must be categorized into questions and answers, forming a massive record of "sample questions, relevant knowledge, sample answers" to form the dataset D0 required by this invention.

[0055] Pre-training the large language model M0: Using the massive data from dataset D0, a distributed autoregressive large language model is trained. The goal is to enable the large language model M0 to memorize a vast amount of world knowledge using the large number of parameters in dataset D0. The pre-trained large language model M0 has some ability to answer questions, but it cannot yet fully understand instructions from teachers and students or answer their questions completely and correctly.

[0056] Therefore, it is necessary to fine-tune the large language model through instruction guidance: optimize the large language model so that it can better understand the true meaning expressed by teachers and students, and generate answers that better match the questions. The specific method is as follows.

[0057] First, we collect a large number of questions D_q from teachers and students. We then have annotators write down the corresponding sample answers and related knowledge, resulting in multiple sets of teaching data: <sample questions, related knowledge, sample answers>. Related knowledge is used to help the model more accurately answer questions from teachers and students. Relevant knowledge refers to the knowledge points or key steps required to answer a question. For example, the knowledge points required to understand the question "What is a neural network?" are its superordinate concept, "neurons." The key steps are "explain the concept - explain the classification - list application examples."

[0058] Taking online teaching, which is currently more popular, as an example, sample questions and sample answers can be obtained from the following aspects.

[0059] The method for obtaining sample questions D_q is to collect the real-time questions asked by students and the teachers' answers and explanation ideas on the online live teaching platform; to collect and obtain the historical questions asked by students in each subject and the teachers' answers on the teaching platforms such as "Xuexi Tong" and "Chaoxing" used by the school. In addition, data such as pre-high school entrance examination Q&A questions and daily small test questions on the teaching platform can also be collected and obtained.

[0060] When collecting and acquiring data, it's important to ensure that the sample questions cover a wide range of subjects. Furthermore, for subjects with strict reasoning requirements, such as mathematics, physics, and chemistry, the sample questions should account for a larger proportion than for subjects in the humanities. When selecting sample questions, it's important to consider all grade levels and all subjects.

[0061] A random subset of records from the teaching dataset is used as an instruction set. Using the "sample question, relevant knowledge, sample answer" structure in this instruction set, the large language model MO is iteratively trained to obtain a fine-tuned large language model M1. These sample questions are then fed into the large language model M1 multiple times. For each sample question, the large language model M1 generates multiple candidate answers, resulting in an unlabeled test set D2: <sample question 1, candidate answer 1>, <sample question 1, candidate answer 2>, and so on. This step helps the large language model M1 better understand the expressions of both teachers and students.

[0062] Taking the test set D2, professional teaching staff sorted the candidate answers for each sample question to obtain a training set D3. For example, the candidate answers for sample question 1 are sorted as follows: <sample question 1, candidate answer 3, candidate answer 5, candidate answer 1, ...>. The sorting model sortModel is then trained on this training set D3. The sorting model input is the concatenated question and answer, that is, the input to the sorting model is <question, answer> pairs, such as <sample question 1, candidate answer 3>, <sample question 1, candidate answer 5>, ... <sample question 1, candidate answer n>. The output of the sorting model is a score, which can be positive or negative. A higher positive score indicates that the candidate answer output by the large language model M1 is better, while a negative score indicates that the large language model M1's answer is poor and should continue learning to generate a better answer.

[0063] Manual labeling of ranking can be done by students or teachers; the ranking standard is the degree of quality of the multiple answers generated by the model to a certain question as judged by teachers and students. The result of manual ranking is the order of quality of the answers.

[0064] In addition, when sorting the answers for each question, multiple people are required to generate enough samples to avoid manually labeled answers that reflect personal preferences.

[0065] Compared to scoring feedback, the advantage of using a ranking model to evaluate the answers generated by a large language model is that it can automatically analyze the relative quality of answers by training the ranking model, avoiding the influence of individual annotations. For example, when the large language model M1 is fed the question "How do you explain neural networks to middle school students?", it generates five candidate answers: A, B, C, D, and E. Assume that each answer is given a maximum score of 10.

[0066] The scores of annotator 1 are: A, 1 point, B, 6 points, C, 8 points, D, 5 points; E, 3 points. The annotation ranking is: C>B>D>E>A;

[0067] The scores of annotator 2 are: A, 2 points, B, 7 points, C, 9 points, D, 4 points, E, 3 points. The order of annotations is C>B>D>E>A.

[0068] Therefore, even if the scores given by two annotators are different, the relative ranking results of the answers are consistent, which does not affect the selection of the best answer.

[0069] Using a policy gradient reinforcement learning algorithm, the sorting model sortModel guides the fine-tuned large language model M1 to learn how to better generate answers to matching questions. The input data for the reinforcement learning module is the remaining sample questions in the teaching dataset, excluding the sample questions used in instruction set D1. These remaining sample questions have no manual annotations and are input into the sorting model sortModel as the fine-tuning dataset DQ.

[0070] During the reinforcement learning process, this invention provides an internet search function to search the sample questions in the fine-tuning dataset DQ. Knowledge is then extracted from the top K search results as relevant knowledge for these sample questions, which is then fed into the large language model M1 along with the sample questions. This step prevents the large language model from fabricating answers that contradict the facts and ensures that the answers it outputs are correct and logical.

[0071] When using reinforcement learning to update the large language model M1, the score of the sorting model sortModel and the loss function of the large language model M1 are combined according to the weights, the fine-tuning data set is input into the sorting model sortModel and the large language model M1, and the large language model M1 is gradient updated. In this embodiment, the large language model M1 after reinforcement learning update is defined as the large language model M2. At this time, the large language model M2 can understand the questions of teachers and students very accurately and generate answers that match the questions. At this time, the large language model M2 can be encapsulated into an interactive teaching auxiliary system to assist teachers and students in the teaching and learning process. The present invention has strong practicality and innovation. The interactive teaching auxiliary system in the present invention is realized by pre-training the large language model. It does not require a lot of manual intervention, can realize autonomous capability iterative upgrades, and can use Internet information to assist in answer generation, so that the generated answers are both accurate and meet the needs of teachers and students.

[0072] Thus, the method provided by the present invention can realize a purely intelligent teaching assistance system, and its capabilities can evolve autonomously through reinforcement learning, thereby providing long-term high-quality services for teachers and students. Specifically, it includes:

[0073] During the use of the large language model M2, the intelligent teaching assistance system background will collect statistics on the questions asked by teachers and students, the corresponding answers provided by the large language model M2, and the feedback from teachers and students on the answers, such as the data set D4 of <actual questions, responses, and satisfaction level>.

[0074] Automatic model capability upgrade: Dataset D5 is generated by preprocessing dataset D4. This preprocessing includes manually merging substantially identical real-world questions and ranking all responses to the same question by satisfaction. The sorting model sortModel is iteratively trained using dataset D5 to produce a stronger guidance model, sortModel_update. This stronger guidance model, sortModel_update, then guides the large language model M2 through reinforcement learning. Regular updates to the large language model M2 allow it to better understand questions and provide more accurate responses.

[0075] Specifically, when training the sorting model sortModel using dataset D5, the input is still a concatenation of question and answer pairs, in the format of <question, answer> pairs, for example, <actual question 1, response answer 3>, <actual question 1, response answer 5>, and so on. Through this iterative training, the sorting model sortModel outputs a score that matches the satisfaction level of the feedback, resulting in an upgraded sorting model, sortModel_update.

[0076] It should be noted that in the embodiments of the present invention, the reinforcement learning algorithm can be the Proximal Policy Optimization (PPO) algorithm or other policy gradient optimization methods. The large language model can have a Transformer-Decoder structure, an Encoder-Decoder structure, or even other deep neural network variants. The ranking model can have the same topic structure as the large language model, using a Transformer-Decoder or Encoder-Decoder deep neural network structure, except that a fully connected neural network with a dimension of 1 is added to the last layer.

[0077] In the present invention, although the ranking model plays a guiding role in the training of the entire large language model, only a ranking model with sufficient knowledge reserves can play an accurate evaluation role. Therefore, the structure of the ranking model of the present invention needs to be as large as the structure of the large language model, that is, the ranking model itself also needs enough network parameters to support the massive knowledge reserve.

[0078] Although the present invention has been described in detail with reference to the accompanying drawings and in combination with preferred embodiments, the present invention is not limited thereto. Without departing from the spirit and essence of the present invention, a person of ordinary skill in the art may make various equivalent modifications or substitutions to the embodiments of the present invention, and such modifications or substitutions shall be within the scope of the present invention. Any person skilled in the art who is familiar with the present invention may easily conceive of changes or substitutions within the technical scope disclosed in the present invention, and such changes or substitutions shall be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A self-correcting intelligent teaching assistance method based on instruction-guided large language model, characterized in that: include: Obtain massive amounts of teaching-related data for distributed training of large language models; Collect sample questions, question-related knowledge, and sample answers through the teaching platform to obtain a teaching dataset with labeled information; Through random sampling, some sample questions and their corresponding relevant knowledge and sample answers are extracted from the annotated teaching dataset as an instruction set, and the instruction set is used to fine-tune the large language model; The sample questions and corresponding related knowledge in the instruction set are input into the fine-tuned large language model multiple times, and multiple candidate answers corresponding to the sample questions are output; By manually sorting candidate answers to sample questions, we obtain a training set with sorted candidate answers for the sample questions. We then train a sorting model, sortModel, based on the training set. The sorting model is used to score the answers generated by the fine-tuned large language model. The remaining portion of the labeled teaching dataset after removing the instruction set is used as the fine-tuning dataset to further fine-tune the answering ability of the fine-tuned large language model. Sample questions from the fine-tuning dataset are input to the fine-tuned large language model and the output test answers are obtained. The sample questions and test answers are then input into the sorting model sortModel, which gives a score for each test answer. Finally, the parameters of the fine-tuned large language model are updated through the policy gradient reinforcement learning algorithm to obtain a large language model with more accurate answers.

2. The method according to claim 1, characterized in that Also includes: After the large language model has been used for a period of time, users are provided with feedback on the answers output by the large language model. Iteratively train sortModel based on user feedback to obtain a stronger sorting model sortModel_update; Based on the more accurate scoring ability of the sorting model sortModel_update, the reinforcement learning algorithm is used again to update the parameters of the large language model, enabling the large language model to complete the capability upgrade.

3. The method according to claim 1, characterized in that When fine-tuning a large language model, the relevant knowledge of the sample question is input as the knowledge points or key steps required to answer the problem.

4. The method according to claim 1, wherein The sample questions are selected from the data retained during the teaching process. When selecting sample questions, each grade and subject should be covered to ensure the comprehensiveness of the data source and the randomness of the selection; The sample answers are manually labeled by professional teaching staff based on specific sample questions, and the relevant knowledge of the sample questions should also be marked.

5. The method according to claim 1, wherein Based on the scoring ability of the ranking model, the parameters of the large language model are updated through the policy gradient reinforcement learning algorithm. The fine-tuning dataset input for reinforcement learning is the remaining data of the teaching dataset after removing the instruction set used for the initial fine-tuning. This remaining data also includes: sample questions, question-related knowledge, and sample answers.

6. The method according to claim 1, characterized in that When the big language model is provided to teachers and students for use, it obtains real-time questions raised by teachers and students, then conducts knowledge retrieval on the Internet for the real-time questions, extracts knowledge from the retrieval results to obtain relevant knowledge for the real-time questions, and then inputs it into the big language model. Finally, the big language model gives accurate answers. Based on the retrieval of relevant knowledge, the big language model has the ability to correct errors and will not produce illogically wrong answers.

Images