Problem generation method based on large language model

By automatically labeling and processing copy words in large language models, the input layer of the problem generation model is optimized, and the problems of high data costs and poor problem generation in automatic question-and-answer tasks are solved, achieving more efficient problem generation and accuracy.

CN119088917APending Publication Date: 2024-12-06ZHEJIANG GONGSHANG UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411098663.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-12
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The existing large language models require a large amount of high-quality question-and-answer data for training in automatic question-and-answer tasks, resulting in high labor and time costs. The text paragraphs are rich in information, but different systems can raise unnecessary questions, resulting in the pre-training effect of problem generation. Bad, waste of computing power.

Method used

By obtaining text data to form a training set of paragraph-question-answer, automatically labeling copy words, using copy word classification model and post-processing technology, adding copy word features to the input layer of the problem generation model, optimizing the problem generation model to reduce the 'pair' More phenomena, improve the value and accuracy of problem generation.

Benefits of technology

It reduces the difficulty of training the problem generation model, reduces the waste of computing power, improves the value and accuracy of problem generation, and reduces labor and time costs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119088917A_ABST
    Figure CN119088917A_ABST
Patent Text Reader

Abstract

The invention relates to a question generation method based on a large language model, which comprises the following steps: S1, acquiring text data, and forming a training set containing paragraphs, questions and answers; s2, copying words on the training set are automatically labeled, specifically, words in paragraphs are traversed, and if a certain word is not in a stop word list and appears in a problem, the word is labeled as a copying word; s3, obtaining a duplicated word or non-duplicated word probability of each segmented word in the paragraph through a duplicated word classification model; s4, post-processing an output result of the duplicated word classification model; and S5, based on the question generation model, adding duplicated word features in an input layer, and generating a high-value question. The invention relates to the technical field of large language models. According to the method, duplicated word features are fused in the input layer of the question generation model, the words duplicated from the original paragraph text contained in the generation result are limited, the duplicating accuracy can be improved, more critical information can be duplicated, generation of correct but unimportant questions is reduced, and waste of computing power is reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of large language models, and in particular to a question generation method based on a large language model. Background Art

[0002] The existing large language models have the ability of Chain of Thought (COT), which can improve the performance of a series of downstream tasks such as mathematical reasoning, agent communication, and tool use by introducing the thinking process of the large model during the generation process.

[0003] Automatic question answering tasks provide direct and accurate answers to questions raised by users. They have been applied to various fields such as online shopping, online medical care, and online education, providing users with more convenient and efficient services. They have a wide range of application scenarios in all walks of life in human society. However, automatic question answering tasks (QA) require a large number of high-quality question and answer pairs related to business scenarios as corpus for training. Therefore, the scale and quality of question and answer data are the key to the final performance of the automatic question and answer system, which brings high labor and time costs.

[0004] In current technology, text paragraphs contain a large amount of information. Even for the same paragraph, different computer systems can ask many questions. Even if all these questions are correct, some questions are more important and some are less important. Unimportant questions have a poor pre-training effect on question generation and waste computing power. Summary of the invention

[0005] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a question generation method based on a large language model to solve the above technical problems.

[0006] The above technical objectives of the present invention are achieved through the following technical solutions:

[0007] The question generation method based on the large language model includes the following steps:

[0008] S1. Obtain text data and form a training set containing paragraphs, questions and answers;

[0009] S2. Automatically label duplicate words in the training set. The specific method is as follows: traverse the words in the paragraph. If a word is not in the stop word list and appears in the question, it is labeled as a duplicate word.

[0010] S3, obtaining the probability of each word in the paragraph being a copied word or a non-copied word through the copied word classification model;

[0011] S4, post-processing the output results of the duplicate word classification model;

[0012] S5. Based on the question generation model, add duplicate word features to the input layer to generate high-value questions.

[0013] In a preferred example, the present invention can be further configured as follows: the stop word list is a set of manually annotated stop words. In order to save storage space and improve search efficiency, the staff filters out certain characters or words before or after processing natural language data. These characters or words are manually input and non-automatically generated. The generated stop words will form a stop word list.

[0014] In a preferred example, the present invention can be further configured as follows: the duplicate word classification model consists of an embedding vector module, a multi-layer bidirectional self-attention encoding module and a duplicate word classification module.

[0015] In a preferred example, the present invention can be further configured as follows: the output result of the duplicate word classification model is post-processed, and the specific method is:

[0016] 1) The staff lowers the probability threshold for phrases that may be duplicate words, and then raises the threshold for unqualified features;

[0017] 2) Setting a threshold for duplicate word classification. If the result of the duplicate word classification model analysis is greater than the threshold, the phrase is a duplicate word, otherwise it is not a duplicate word;

[0018] The post-processed duplicate word classification results will be added as features to the input of the question generation model to optimize the question generation model and improve the accuracy of the question generation model.

[0019] In a preferred example, the present invention can be further configured as follows: the question generation model includes an embedding vector module containing duplicate word features, a multi-layer sequence-to-sequence self-attention encoder module, and a question generation module. In the embedding vector module containing duplicate word features, the duplicate word feature embedding vector is added to the word segmentation embedding, position embedding, fragment embedding, and task embedding. The encoding of the duplicate word is set to 1, and the encoding of the non-duplicate word is set to 0.

[0020] In a preferred example, the present invention can be further configured as follows: the question generation model includes a pre-training module, and the training method of the pre-training module is:

[0021] Question generation and answer extraction are integrated into a unified framework. The target document is encoded by the BART encoder to obtain the semantic vector of the document, and the semantic vector is sent to the decoder for question generation. The decoder terminal vector and the document semantic vector are cross-attentioned to obtain the initialization vector of the answer prediction module, and the answer position of the question in the original text is obtained after prediction. The loss generated by the answer prediction and the loss generated by the original question generation task of the BART model are jointly back-propagated through the algorithm to update the model parameters.

[0022] In a preferred example, the present invention can be further configured as follows: the BART encoder is a bidirectional encoder that encodes the input sentence from two directions to obtain more context information;

[0023] The decoder is an autoregressive decoder. During the training process, the decoding module is initialized and combined with the start mark of the sentence to generate the first word. Then the first word and the hidden state of the previous historical moment are added to the input of the next step, and the cycle is repeated until the end mark is generated. The decoder fuses all the input vectors to output the next vector.

[0024] In a preferred example, the present invention can be further configured as follows: the evaluation method of the question generation method is:

[0025] First, the training data is divided into equal parts and the data is randomly divided into two sets of documents. Half of the documents are used to train the question-answering data generation model, while the other half of the documents are used to generate synthetic data to fine-tune the MRC model; the fine-tuned MRC model is then evaluated on the development set, where the evaluation results are used as an alternative measure of the quality of the synthetic data.

[0026] In summary, the present invention includes at least one of the following beneficial technical effects:

[0027] The present invention integrates duplicate word features in the input layer of the question generation model, which can alleviate the "one-to-many" phenomenon faced by the question generation task model, limit the solution space of the model to a certain extent, reduce the difficulty of fitting the model during training, help to improve the value of question generation, and reduce waste of computing power. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0029] Figure 1 It is a flow chart of the question generation method based on a large language model of the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0031] It should be noted that, although the functional modules are divided in the system schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first / S100", "second / S200", etc. in the specification, claims and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0032] Reference to an "embodiment" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiment may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0033] Embodiment 1:

[0034] Reference Figure 1 The present invention proposes a question generation method based on a large language model, comprising the following steps:

[0035] S1. Obtain text data to form a training set containing paragraphs, questions and answers;

[0036] Text data can be obtained from existing data sets as text data according to application requirements. For example, if the application requirement is to use a large language model to solve math problems, the original sample data can be obtained from the GSM8K data set, so that the trained question-answering model can generate corresponding analysis and answers based on the input math problem text.

[0037] S2. Automatically label duplicate words in the training set. The specific method is as follows: traverse the words in the paragraph. If a word is not in the stop word list and appears in the question, it is labeled as a duplicate word.

[0038] The stop word list is a set of manually annotated stop words. In order to save storage space and improve search efficiency, staff filter out certain characters or words before or after processing natural language data. These characters or words are manually input and not automatically generated. The generated stop words will form a stop word list.

[0039] S3, obtaining the probability of each word in the paragraph being a copied word or a non-copied word through the copied word classification model;

[0040] The duplicate word classification model consists of an embedding vector module, a multi-layer bidirectional self-attention encoding module and a duplicate word classification module.

[0041] S4. Post-process the output results of the duplicate word classification model. The specific method of post-processing is:

[0042] 1) The staff lowers the probability threshold for phrases that may be duplicate words, and then raises the threshold for unqualified features;

[0043] 2) Setting a threshold for duplicate word classification. If the result of the duplicate word classification model analysis is greater than the threshold, the phrase is a duplicate word, otherwise it is not a duplicate word;

[0044] The post-processed duplicate word classification results will be added as features to the input of the question generation model to optimize the question generation model and improve the accuracy of the question generation model.

[0045] S5. Based on the question generation model, add duplicate word features to the input layer to generate high-value questions.

[0046] The question generation model includes an embedding vector module containing duplicate word features, a multi-layer sequence-to-sequence self-attention encoder module, and a question generation module. In the embedding vector module containing duplicate word features, the duplicate word feature embedding vector is added to the word segmentation embedding, position embedding, fragment embedding, and task embedding. The encoding of the duplicate word is set to 1, and the encoding of the non-duplicate word is set to 0.

[0047] Embodiment 2:

[0048] Reference Figure 1 This embodiment proposes a question generation method based on a large language model that is superior to Embodiment 1, including the following steps:

[0049] S1. Obtain text data to form a training set containing paragraphs, questions and answers;

[0050] Text data can be obtained from existing data sets as text data according to application requirements. For example, if the application requirement is to use a large language model to solve math problems, the original sample data can be obtained from the GSM8K data set, so that the trained question-answering model can generate corresponding analysis and answers based on the input math problem text.

[0051] S2. Automatically label duplicate words in the training set. The specific method is as follows: traverse the words in the paragraph. If a word is not in the stop word list and appears in the question, it is labeled as a duplicate word.

[0052] The stop word list is a set of manually annotated stop words. In order to save storage space and improve search efficiency, staff filter out certain characters or words before or after processing natural language data. These characters or words are manually input and not automatically generated. The generated stop words will form a stop word list.

[0053] S3, obtaining the probability of each word in the paragraph being a copied word or a non-copied word through the copied word classification model;

[0054] The duplicate word classification model consists of an embedding vector module, a multi-layer bidirectional self-attention encoding module and a duplicate word classification module.

[0055] S4. Post-process the output results of the duplicate word classification model. The specific method of post-processing is:

[0056] 1) The staff lowers the probability threshold for phrases that may be duplicate words, and then raises the threshold for unqualified features;

[0057] 2) Setting a threshold for duplicate word classification. If the result of the duplicate word classification model analysis is greater than the threshold, the phrase is a duplicate word, otherwise it is not a duplicate word;

[0058] The post-processed duplicate word classification results will be added as features to the input of the question generation model to optimize the question generation model and improve the accuracy of the question generation model.

[0059] S5. Based on the question generation model, add duplicate word features to the input layer to generate high-value questions.

[0060] The question generation model includes an embedding vector module containing duplicate word features, a multi-layer sequence-to-sequence self-attention encoder module, and a question generation module. In the embedding vector module containing duplicate word features, the duplicate word feature embedding vector is added to the word segmentation embedding, position embedding, fragment embedding, and task embedding. The encoding of the duplicate word is set to 1, and the encoding of the non-duplicate word is set to 0.

[0061] The question generation model includes a pre-training module, and the training method of the pre-training module is:

[0062] Question generation and answer extraction are integrated into a unified framework. The target document is encoded by the BART encoder to obtain the semantic vector of the document, and the semantic vector is sent to the decoder for question generation. The decoder terminal vector and the document semantic vector are cross-attentioned to obtain the initialization vector of the answer prediction module, and the answer position of the question in the original text is obtained after prediction. The loss generated by the answer prediction and the loss generated by the original question generation task of the BART model are jointly back-propagated through the algorithm to update the model parameters.

[0063] Based on the commonly used pre-trained model for text generation, this method uses multi-task learning to fully combine the two problems of question generation and answer extraction to improve the performance of the question and answer data generation model.

[0064] The BART encoder is a bidirectional encoder that encodes the input sentence from two directions to obtain more context information;

[0065] The decoder is an autoregressive decoder. During the training process, the decoding module is initialized and combined with the start mark of the sentence to generate the first word. Then the first word and the hidden state of the previous historical moment are added to the input of the next step, and the cycle is repeated until the end mark is generated. The decoder fuses all the input vectors to output the next vector.

[0066] This embodiment trains the question generation model based on the pre-training module to improve its question generation accuracy.

[0067] Embodiment 3:

[0068] Reference Figure 1 This embodiment proposes a question generation method based on a large language model that is superior to Embodiment 1 and Embodiment 2, including the following steps:

[0069] S1. Obtain text data and form a training set containing paragraphs, questions and answers;

[0070] Text data can be obtained from existing data sets as text data according to application requirements. For example, if the application requirement is to use a large language model to solve math problems, the original sample data can be obtained from the GSM8K data set, so that the trained question-answering model can generate corresponding analysis and answers based on the input math problem text.

[0071] S2. Automatically label duplicate words in the training set. The specific method is as follows: traverse the words in the paragraph. If a word is not in the stop word list and appears in the question, it is labeled as a duplicate word.

[0072] The stop word list is a set of manually annotated stop words. In order to save storage space and improve search efficiency, staff filter out certain characters or words before or after processing natural language data. These characters or words are manually input and not automatically generated. The generated stop words will form a stop word list.

[0073] S3, obtaining the probability of each word in the paragraph being a copied word or a non-copied word through the copied word classification model;

[0074] The duplicate word classification model consists of an embedding vector module, a multi-layer bidirectional self-attention encoding module and a duplicate word classification module.

[0075] S4. Post-process the output results of the duplicate word classification model. The specific method of post-processing is:

[0076] 1) The staff lowers the probability threshold for phrases that may be duplicate words, and then raises the threshold for unqualified features;

[0077] 2) Setting a threshold for duplicate word classification. If the result of the duplicate word classification model analysis is greater than the threshold, the phrase is a duplicate word, otherwise it is not a duplicate word;

[0078] The post-processed duplicate word classification results will be added as features to the input of the question generation model to optimize the question generation model and improve the accuracy of the question generation model.

[0079] S5. Based on the question generation model, add duplicate word features to the input layer to generate high-value questions.

[0080] The question generation model includes an embedding vector module containing duplicate word features, a multi-layer sequence-to-sequence self-attention encoder module, and a question generation module. In the embedding vector module containing duplicate word features, the duplicate word feature embedding vector is added to the word segmentation embedding, position embedding, fragment embedding, and task embedding. The encoding of the duplicate word is set to 1, and the encoding of the non-duplicate word is set to 0.

[0081] The question generation model includes a pre-training module, and the training method of the pre-training module is:

[0082] Question generation and answer extraction are integrated into a unified framework. The target document is encoded by the BART encoder to obtain the semantic vector of the document, and the semantic vector is sent to the decoder for question generation. The decoder terminal vector and the document semantic vector are cross-attentioned to obtain the initialization vector of the answer prediction module, and the answer position of the question in the original text is obtained after prediction. The loss generated by the answer prediction and the loss generated by the original question generation task of the BART model are jointly back-propagated through the algorithm to update the model parameters.

[0083] Based on the commonly used pre-trained model for text generation, this method uses multi-task learning to fully combine the two problems of question generation and answer extraction to improve the performance of the question and answer data generation model.

[0084] The BART encoder is a bidirectional encoder that encodes the input sentence from two directions to obtain more context information;

[0085] The decoder is an autoregressive decoder. During the training process, the decoding module is initialized and combined with the start mark of the sentence to generate the first word. Then the first word and the hidden state of the previous historical moment are added to the input of the next step, and the cycle is repeated until the end mark is generated. The decoder fuses all the input vectors to output the next vector.

[0086] The evaluation method of the problem generation method is:

[0087] First, the training data is divided into equal parts, and the data is randomly divided into two sets of documents. Half of the documents are used to train the question-answering data generation model, while the other half of the documents are used to generate synthetic data to fine-tune the MRC model; the fine-tuned MRC model is then evaluated on the development set, where the evaluation results are used as an alternative measure of the quality of the synthetic data. The data set is divided to avoid leakage and overfitting between the data seen during training and generation, thereby testing the generalization ability of the model.

[0088] This embodiment proposes an evaluation method for the question generation method based on a large language model described in Embodiment 1 or 2 to test the generalization ability of the question generation method based on a large language model.

[0089] The implementation principle of the above embodiment is:

[0090] S1. Obtain text data and form a training set containing paragraphs, questions and answers;

[0091] S2. Automatically label duplicate words in the training set. The specific method is as follows: traverse the words in the paragraph. If a word is not in the stop word list and appears in the question, it is labeled as a duplicate word.

[0092] S3, obtaining the probability of each word in the paragraph being a copied word or a non-copied word through the copied word classification model;

[0093] S4, post-processing the output results of the duplicate word classification model;

[0094] S5. Based on the question generation model, add duplicate word features to the input layer to generate high-value questions.

[0095] The contents not described in detail in this specification belong to the prior art known to professional and technical personnel in this field.

[0096] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. Question generation method based on large language model, Features: The following steps are involved: S1. Obtain text data to form a training set containing paragraphs, questions and answers; S2. Automatically label duplicate words in the training set. The specific method is as follows: traverse the words in the paragraph. If a word is not in the stop word list and appears in the question, it is labeled as a duplicate word. S3, obtaining the probability of each word in the paragraph being a copied word or a non-copied word through the copied word classification model; S4, post-processing the output results of the duplicate word classification model; S5. Based on the question generation model, add duplicate word features to the input layer to generate high-value questions.

2. The method for generating questions based on a large language model according to claim 1, Features: The stop word list is a set of manually annotated stop words. In order to save storage space and improve search efficiency, staff filter out certain characters or words before or after processing natural language data. These characters or words are manually input and not automatically generated. The generated stop words will form a stop word list.

3. The method for generating questions based on a large language model according to claim 2, Features: The duplicate word classification model consists of an embedding vector module, a multi-layer bidirectional self-attention encoding module and a duplicate word classification module.

4. The method for generating questions based on a large language model according to claim 3, Features: The output result of the duplicate word classification model is post-processed, and the specific method is as follows: 1) The staff lowers the probability threshold for phrases that may be duplicate words, and then raises the threshold for unqualified features; 2) Setting a threshold for duplicate word classification. If the result of the duplicate word classification model analysis is greater than the threshold, the phrase is a duplicate word, otherwise it is not a duplicate word; The post-processed duplicate word classification results will be added as features to the input of the question generation model to optimize the question generation model and improve the accuracy of the question generation model.

5. The method for generating questions based on a large language model according to claim 4, Features: The question generation model includes an embedding vector module containing duplicate word features, a multi-layer sequence-to-sequence self-attention encoder module, and a question generation module. In the embedding vector module containing duplicate word features, the duplicate word feature embedding vector is added to the word segmentation embedding, position embedding, fragment embedding, and task embedding. The encoding of the duplicate word is set to 1, and the encoding of the non-duplicate word is set to 0.

6. The method for generating questions based on a large language model according to claim 5, Features: The question generation model includes a pre-training module, and the training method of the pre-training module is: Question generation and answer extraction are integrated into a unified framework. The target document is encoded by the BART encoder to obtain the semantic vector of the document, and the semantic vector is sent to the decoder for question generation. The decoder terminal vector and the document semantic vector are cross-attentioned to obtain the initialization vector of the answer prediction module, and the answer position of the question in the original text is obtained after prediction. The loss generated by the answer prediction and the loss generated by the original question generation task of the BART model are jointly back-propagated through the algorithm to update the model parameters.

7. The method for generating questions based on a large language model according to claim 6, Features: The BART encoder is a bidirectional encoder that encodes the input sentence from two directions to obtain more context information; The decoder is an autoregressive decoder. During the training process, the decoding module is initialized and combined with the start mark of the sentence to generate the first word. Then the first word and the hidden state of the previous historical moment are added to the input of the next step, and the cycle is repeated until the end mark is generated. The decoder fuses all the input vectors to output the next vector.

8. The method for generating questions based on a large language model according to claim 7, Features: The evaluation method of the problem generation method is: First, the training data is divided into equal parts and the data is randomly divided into two sets of documents. Half of the documents are used to train the question-answering data generation model, while the other half of the documents are used to generate synthetic data to fine-tune the MRC model; the fine-tuned MRC model is then evaluated on the development set, where the evaluation results are used as an alternative measure of the quality of the synthetic data.