Intelligent test question generation method and system based on large language model

Abstract

The application relates to the technical field of intelligent question setting, and provides a test question intelligent generation method and system based on a large language model. The method comprises the following steps: extracting keywords based on historical textbook data; extracting the relationship between the keywords based on the keywords, and outputting in the form of triples to construct a knowledge graph; performing a breadth search in the knowledge graph according to the entity part in the specified triple, finding all triples in which the entity part appears, constructing a first triple set, calculating the correlation score between each triple in the first triple set and the specified triple, if the correlation score is greater than a set threshold, adding the triple to a second triple set, and adding the correlation score to a score set; selecting the triple corresponding to the score according to a plurality of scores to form a relevant subgraph; and using a large language model to generate a question and a corresponding answer based on the relevant subgraph, a preset question type and a test question prompt.

Description

Technical Field

[0001] This invention relates to the field of intelligent question generation technology, and in particular to a method and system for intelligent question generation based on a large language model. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Traditional question-setting methods often require teachers to spend a significant amount of time designing, compiling, selecting, and organizing questions. This process is not only time-consuming and labor-intensive, but also prone to errors and easily influenced by the teacher's personal subjectivity and biases, making it difficult to guarantee the fairness and impartiality of the exam papers. With the help of artificial intelligence technology, factors such as knowledge points, question types, and difficulty levels can be automatically analyzed to quickly generate exam papers that meet the requirements. This not only greatly reduces the workload of teachers but also ensures the objectivity and fairness of the question-setting process.

[0004] Rule-based question generation, or automatic question generation, involves designing heuristic rules to transform descriptive text into relevant questions. While research on rule-based question generation methods has demonstrated the possibility of automated question generation, it has significant limitations. First, due to the complexity of language, it is difficult to generalize rules applicable to all questions, and text rules vary considerably across different domains. Therefore, rule-based question generation algorithms have poor portability and are difficult to extend.

[0005] While neural network-based question generation methods have made significant progress compared to rule-based methods, they still have several drawbacks. First, neural network models typically require extensive training to achieve high performance. This prolonged training not only increases computational costs but may also limit the model's applicability in practical applications. Second, the performance of neural network models is highly dependent on the quality and diversity of the training data. If the training data is biased or insufficient, the questions generated by the model may exhibit similar patterns or problems, thus reducing the diversity and effectiveness of the questions. Summary of the Invention

[0006] To address the technical problems mentioned above, this invention provides a method and system for intelligent test question generation based on a large language model. This invention can generate high-quality test questions and answers, while also possessing strong portability and versatility.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] The first aspect of this invention provides a method for intelligent test question generation based on a large language model.

[0009] A test question intelligent generation method based on a large language model includes:

[0010] Based on historical textbook data, keywords were extracted, including time, people, and ideas;

[0011] Based on the keywords, the relationships between the keywords are extracted and output in the form of triples to construct a knowledge graph.

[0012] Based on the entity portion of the specified triples, a breadth-first search is performed in the knowledge graph to find all triples in which the entity portion appears, constructing a first triplet set. The relevance score between each triple in the first triplet set and the specified triplet is calculated. If the relevance score is greater than a set threshold, the triplet is added to a second triplet set, and the relevance score is added to a score set. After the breadth-first search, the scores in the score set are sorted in descending order to obtain a sorted index sequence. If the length of the second triplet set is greater than a set value, the set is truncated to the latest index set, and several indices are randomly selected from it; otherwise, the first few indices are directly truncated. Triples are selected according to the indices to form a relevance subgraph.

[0013] Based on the relevant subgraphs, preset question types, and question prompts, a large language model is used to generate questions and corresponding answers.

[0014] Furthermore, the relevance score is the average of the character relevance score and the semantic relevance score; the character relevance score is the Jaccard coefficient between the concatenated text of the two triples; the semantic relevance score is calculated by encoding the head and tail entities of each triple using a language model, calculating the cosine distance between the encodings, and taking the maximum cosine distance as the semantic relevance score.

[0015] Furthermore, the process of obtaining the relevant subgraph uses the following formula:

[0016]

[0017] Among them, K i Indicates its keywords, head i This indicates its head entity, tail i Represents its tail entity, rel i This represents the relationship between the head and tail entities, T represents the specified triple, B represents the number of layers in the breadth-first search on the knowledge graph, N represents the number of triples in the relevant subgraph, θ represents the threshold for the relevance score, and KG represents the knowledge graph.

[0018] Furthermore, the process of generating questions and corresponding answers based on the relevant subgraphs, preset question types, and question prompts using a large language model employs the following formula:

[0019] (question, answer) = LLM(L, D, P Q (KG)

[0020] Where LLM stands for Large Language Model, L represents the Relevance Subgraph, D represents the Item Type, and P represents the Item Type. Q This indicates a prompt for the test question, and KG represents a knowledge graph.

[0021] Furthermore, during the breadth-first search, if the entity part of a triple in the second triple set does not appear in the existing entity set, then the entity part of that triple is added to the existing entity set.

[0022] Furthermore, before constructing the knowledge graph, the process includes: collecting historical textbooks, converting them from PDF to TXT format to retain the effective text; and based on the effective text, segmenting the text according to the paragraph relationships to convert the effective text into paragraphs containing a number of words.

[0023] Furthermore, the process of preserving valid text includes removing sidebars, chapter exercises, and illustrations from historical textbooks.

[0024] A second aspect of the present invention provides a test question intelligent generation system based on a large language model.

[0025] A test question intelligent generation system based on a large language model includes:

[0026] The keyword extraction module is configured to extract keywords, including time, people, and ideas, based on historical textbook data.

[0027] The knowledge graph construction module is configured to: extract the relationships between the keywords based on the keywords and output them in the form of triples to construct a knowledge graph;

[0028] The relevant subgraph generation module is configured to: perform a breadth-first search in the knowledge graph based on the entity part in the specified triplet, find all triplets in which the entity part appears, construct a first triplet set, calculate the relevance score between each triplet in the first triplet set and the specified triplet, and if the relevance score is greater than a set threshold, add the triplet to a second triplet set and add the relevance score to a score set; after the breadth-first search, sort the scores in the score set in descending order to obtain the sorted index sequence; if the length of the second triplet set is greater than a set value, truncate the set to the latest index set and randomly select several indices from it; otherwise, directly truncate the first few indices; select the corresponding triplets according to the indexes to form a relevant subgraph;

[0029] The intelligent generation module is configured to generate questions and corresponding answers based on the relevant subgraphs, preset question types, and question prompts, using a large language model.

[0030] A third aspect of the present invention provides a computer-readable storage medium.

[0031] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the intelligent test item generation method based on a large language model as described in the first aspect above.

[0032] A fourth aspect of the present invention provides a computer device.

[0033] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the intelligent test item generation method based on a large language model as described in the first aspect above.

[0034] Compared with the prior art, the beneficial effects of the present invention are:

[0035] This invention improves the effectiveness and authenticity of triplet data by focusing on keywords, and constructs a historical knowledge graph.

[0036] This invention expands the data of the triples in the prompts by increasing the number of triples, adding subject keywords, and using context learning algorithms, thereby generating high-quality test questions and answers.

[0037] This invention provides a test question intelligent generation system based on a large language model, which features clear module division and strong portability and versatility. Attached Figure Description

[0038] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0039] Figure 1 This is a network diagram of the intelligent test question generation method based on a large language model shown in this invention;

[0040] Figure 2 This is the triplet extraction prompt shown in this invention;

[0041] Figure 3 This is the prompt for generating multiple-choice questions as shown in this invention;

[0042] Figure 4 This is the automatic evaluation score of the algorithm shown in this invention;

[0043] Figure 5 This is the human evaluation score of the algorithm shown in this invention;

[0044] Figure 6 This is an example of a multiple-choice question as illustrated in this invention. Detailed Implementation

[0045] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0046] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0047] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0048] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and systems according to various embodiments of this disclosure. It should be noted that each block in a flowchart or block diagram may represent a module, segment, or portion of code, which may include one or more executable instructions for implementing the logical functions specified in the various embodiments. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutively represented blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, may be implemented using a dedicated hardware-based system that performs the specified functions or operations, or using a combination of dedicated hardware and computer instructions.

[0049] Example 1

[0050] like Figure 1As shown, this embodiment provides a method for intelligent test question generation based on a large language model. This embodiment uses the application of this method to a server as an example for illustration. It is understood that this method can also be applied to terminals, and can also be applied to systems including terminals, servers, and other components, and can be implemented through interaction between the terminal and the server. The server can be an independent physical server, a server cluster composed of multiple physical servers, or a distributed system. It can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network servers, cloud communication, middleware services, domain name services, CDN security services, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein. In this embodiment, the method includes the following steps:

[0051] Based on historical textbook data, keywords were extracted, including time, people, and ideas;

[0052] Based on the keywords, the relationships between the keywords are extracted and output in the form of triples to construct a knowledge graph.

[0053] Based on the entity portion of the specified triples, a breadth-first search is performed in the knowledge graph to find all triples in which the entity portion appears, constructing a first triplet set. The relevance score between each triple in the first triplet set and the specified triplet is calculated. If the relevance score is greater than a set threshold, the triplet is added to a second triplet set, and the relevance score is added to a score set. After the breadth-first search, the scores in the score set are sorted in descending order to obtain a sorted index sequence. If the length of the second triplet set is greater than a set value, the set is truncated to the latest index set, and several indices are randomly selected from it; otherwise, the first few indices are directly truncated. Triples are selected according to the indices to form a relevance subgraph.

[0054] Based on the relevant subgraphs, preset question types, and question prompts, a large language model is used to generate questions and corresponding answers.

[0055] This invention utilizes knowledge graphs and prompting technology to generate high-quality test questions. The invention mainly consists of two parts: (1) a knowledge graph construction algorithm: using historical textbook data as raw data and relying on a large language model, keyword extraction and triple extraction strategies are implemented based on the characteristics of historical data to transform unstructured text into structured triple data and construct a historical knowledge graph. (2) a test question generation algorithm: based on the aforementioned knowledge graph, appropriate prompting strategies are designed, and questions and answers are automatically generated through a large language model. Compared to traditional methods, this invention has strong portability and versatility. The framework of this invention is as follows: Figure 1 As shown.

[0056] 1. Knowledge Graph Construction

[0057] As the foundation for the test question generation task, this part attempts to construct a historical knowledge graph to organize historical knowledge in a structured form. Using history textbook data as the raw data and relying on a large language model, keyword extraction and triple extraction strategies are designed based on the characteristics of historical data. First, keywords are extracted: time, people, and ideas. Then, triples are extracted based on the keywords, thereby transforming unstructured text into structured triple data and constructing a historical knowledge graph.

[0058] (1) Triple Extraction Algorithm

[0059] For input text I, design a suitable prompting strategy to guide the LLM in extracting all possible triples.

[0060]

[0061] Where J is the number of triples extracted from the input text I. For the j-th triple, where head j ,tail j These are the head entity and the tail entity, rel. j This represents the relationship between head and tail entities. An LLM can be any language model, such as ChatGPT or ERINE. T This indicates a prompt.

[0062] The complexity of history makes accurate relationship extraction from historical texts difficult. First, history's complexity stems from the numerous factors involved; historical events are often influenced by a combination of political, economic, cultural, and other factors. Second, history's complexity is also reflected in the intricate nature of its causal relationships. The causal relationships between historical events are rarely simple linear relationships, but rather interwoven into a complex network. Therefore, how to extract high-quality triplet data from historical texts is the focus of this section.

[0063] (2) Keyword-guided triple extraction algorithm

[0064] In the process of triple extraction, the complexity of historical knowledge leads to problems such as scattered focus and semantic ambiguity in the extracted triples. Furthermore, some triples lack key information, rendering their descriptions meaningless and failing to meet the accuracy requirements of historical knowledge. This invention proposes a novel triple extraction algorithm. First, it guides the large model to focus on the keywords in the text, and then extracts relationships based on these keywords. This makes the extracted relationship results more focused, while the text keywords can augment the triple data, improving its effectiveness and refinement. Considering the characteristics of history, three keywords—time, figures, and ideas—are selected as the focus of triple extraction. First, time is the cornerstone of history. The chronological nature of history requires that the time factor be considered when understanding and analyzing historical events. Time is both the basic coordinate of historical events and a key factor in analyzing and understanding them. Second, figures are the main body of history. By studying historical figures, we can gain a deeper understanding of their historical background, social status, personal qualities, and the impact of their thoughts and behaviors on history. Finally, ideas are the soul of history. By studying historical perspectives, we can understand the values, beliefs, and moral concepts of people in different historical periods, thereby gaining insight into the development and evolution of history.

[0065] Contextual learning has proven to be an effective method for improving the performance of large language models. Contextual learning involves selecting a small number of labeled samples and designing task-related instructions to form a prompt template, which guides the generation of corresponding results. This invention provides several samples in the prompt to guide the large language model to better understand the triple extraction task and output structure-normalized results. The triple extraction algorithm is as follows:

[0066]

[0067] The algorithm mainly includes two formulas: keyword extraction formula (2) and triplet extraction formula (3). Input I is the result of data preprocessing, a text of 50-200 characters. K I This refers to keywords related to the time, people, and ideas of input I. The algorithm's logical idea is a two-step process, implemented as the task input for LLM. The task definition in the hint is to first extract keywords related to time, people, and ideas from the text, then extract relations based on the keywords, and output them in the form of triples. The hint provides two examples, each containing an input, a task definition, and an output. T is the set of triples, and head... i Represents the head entity, tail i Represents the tail entity, relation i This indicates the relationship between the head entity and the tail entity. (P) K and P T Indicates as Figure 2The prompt shown.

[0068] (3) Data Acquisition and Preprocessing

[0069] To ensure the systematic nature of the data, a complete set of standardized high school history textbooks was collected as initial data, including two compulsory textbooks and three elective textbooks. Data preprocessing mainly included three parts: format conversion, effective text extraction, and text segmentation. Format conversion involved converting the history textbooks from PDF to TXT format, retaining only the text data and removing distracting data such as images to facilitate direct text processing. Effective text extraction involved retaining the main text of the textbooks while removing distracting data such as sidebars, chapter exercises, and illustrations, greatly improving the effectiveness of the data. Text segmentation involved converting the effective text into paragraphs containing 50-200 words based on the paragraph relationships in the textbook text, facilitating subsequent triple extraction.

[0070] 2. Question Generation Algorithm

[0071] Based on the aforementioned knowledge graph, a suitable prompting strategy is designed to automatically generate questions and answers using a large language model. For triples, LLM and carefully designed prompts are used to achieve automatic question generation. The algorithm is defined as follows:

[0072] (question,answer)=LLM(T,D,P Q ), T=(head, rel, tail) (5)

[0073] Where T is a triple, head is its head entity, tail is its tail entity, and rel is the relationship between its head and tail entities. D refers to the question type; this invention provides three question types: multiple choice, short answer, and material-based questions. P Q This is the prompt used to generate the question. The output is a tuple containing the question and answer.

[0074] Observation of the test questions generated by the algorithm revealed poor performance, primarily due to three issues. First, insufficient data for some triples led to semantic ambiguity, resulting in semantically ambiguous generated test questions that made it impossible to determine the correct answer. Second, the algorithm generated test questions based on single triples, resulting in a focus on only a single knowledge point and insufficient diversity. Finally, the non-standard output structure of the large language model hindered the extraction of questions and answers.

[0075] (1) Advanced Question Generation Algorithm

[0076] To address these issues with the algorithm, this invention attempts to improve it in three aspects. First, historical keywords are added to the triples. By supplementing key information such as time, people, and ideas, the effectiveness of the triples is greatly improved. Second, the number of triples is increased. Instead of generating questions based on a single triple, data augmentation is performed using a knowledge graph, searching for related subgraphs of the triples in the knowledge graph as input. Increasing the number of triples improves the data's robustness to interference and ensures the diversity of test questions. Finally, to ensure that the large model obtains structured results, the concept of context learning is introduced during prompt construction. Multiple samples are added to help the LLM better understand the task of generating test questions and output standardized results. The formal definition of the algorithm is as follows:

[0077] (question, answer) = LLM(L, D, P Q (6) KG

[0078] Where L refers to the relevant subgraph, D refers to the question type, including three types: multiple choice, short answer, and material-based questions. P Q This is a prompt for generating test questions; the specific prompts for multiple-choice questions are as follows: Figure 3 As shown, KG is a knowledge graph, and its output is a pair consisting of a question and an answer.

[0079] (2) Calculation of relevance score

[0080] When calculating the relevance score between triples, instead of considering all components of the triple, only the entity part of the triple is considered. This is because, during the question generation process, relevant triples should ideally be centrally located to avoid some triples becoming distracting data due to their overly broad scope. This can be achieved by considering only the entity elements. At the same time, focusing only on the core entity part can improve the algorithm's computational speed and save computational resources.

[0081] The relevance score consists of two parts: character relevance score and semantic relevance score. The average of these two scores is taken as the final score. When calculating the character relevance score, the Jaccard coefficient between the concatenated text of the first and last elements of two triples is calculated. When calculating the semantic relevance score, a pre-trained language model is used to encode the [first entity, last entity] of each triple. Then, the cosine distance between the encodings is calculated, and the maximum value is taken as the final output.

[0082] (3) Obtaining related subgraphs

[0083] The goal of the relevant subgraph acquisition algorithm is to find the most relevant subgraphs in a knowledge graph based on a given triple. The algorithm is defined as follows:

[0084]

[0085] Where T represents the triple to be queried, K represents its keyword, head represents its head entity, tail represents its tail entity, and rel represents the relationship between its head and tail entities. Output L is the set of relation triples; for the j-th triple, K... j Indicates its keywords, head j This indicates its head entity, tail j Represents its tail entity, rel j This represents the relationship between the head and tail entities. B represents the number of layers in the breadth-first search on the knowledge graph, N represents the number of triples in the relevant subgraph, θ represents the threshold for the relevance score, and KG represents the knowledge graph.

[0086] A breadth-first search is performed on the Keyword Group (KG) based on the entities. When traversing each entity, the first step is to use the word-contain function to find all triples of entities appearing in the KG, regardless of whether they are keywords, head entities, tail entities, or relations, resulting in a set C of triples. Then, each triple t in C is traversed. i Calculate t and t i The correlation score between them i If the score is greater than the threshold θ, then t i Add s to result set P1 i Add the entity to the score set S. Check if its head and tail appear in the entity set; if not, add it. After traversing each level of entities, update the values ​​of start and end. After the breadth-first search, sort S in descending order of relevance score and save the index sequence (the index of the score in the score set before sorting) to I. If the length of the result set P1 is greater than 2N, randomly select N indices from the top 2N indices; otherwise, directly use the top N indices. Select triples based on the indices to form a relevance subgraph. By introducing randomness, the algorithm improves the scalability between triples while ensuring the relevance of relation triples.

[0087] This invention constructs a historical knowledge graph. During the construction of the historical graph, a novel approach of keyword extraction and triple extraction is proposed. By centering on keywords, the effectiveness and authenticity of the triple data are improved.

[0088] This invention designs an intelligent test question generation algorithm based on knowledge graphs and large language models. The core of the algorithm lies in the design and construction of prompts. Based on relation triples, the algorithm expands the prompts by increasing the number of triples, adding subject keywords, and employing context learning algorithms, thereby improving algorithm performance. A key feature of this invention is the use of large language models to achieve automatic test question generation, providing a new approach to intelligent test question generation.

[0089] The experiments in this invention used ERNIE Bot as a large language model. The data for constructing the knowledge graph came from a publicly available high school history textbook, which contains 423,000 words. After data preprocessing, the text consisted of 137,000 words and contained 1,647 data entries. After relation extraction, a total of 7,601 triplet data entries were extracted.

[0090] To test the effectiveness of the algorithm on different types of questions, we collected several sets of historical exam papers from the internet, including 100 multiple-choice questions, 100 short-answer questions, and 100 essay questions. We then extracted triples from the questions to generate new questions.

[0091] This invention uses EAS (Embedding Average Score), Distinct, F1, and BLEU to automatically evaluate the algorithm's performance. EAS is a word vector-based evaluation metric that primarily focuses on the similarity of the semantic distribution between the generated and reference questions. BLEU focuses on accuracy, calculating the ratio of the number of n-grams in the generated question to the number of n-grams in the reference question, using BLEU-1 as the evaluation metric. Distinct is used to evaluate the diversity of the generated questions, calculating the proportion of unique n-grams in the generated text to the total number of n-grams, using Distinct-1 as the evaluation metric. F1 is a comprehensive evaluation metric whose core idea is to comprehensively consider both accuracy and recall to find a balance between the two key metrics. The automatic evaluation metrics range from 0 to 1.

[0092] This invention uses fluency, relevance, and diversity to manually evaluate the algorithm's performance. Fluency assesses the reasonableness and coherence of the language used in the test questions. Relevance evaluates the correlation between the generated test questions and the reference test questions, as well as the historical knowledge points they involve. Diversity evaluates the scope and number of historical knowledge points covered in the test questions. Manual evaluation uses a tiered rating system, assigning five rating levels: 1, 2, 3, 4, and 5, representing very poor, poor, average, good, and very good, respectively.

[0093] The algorithm proposed in this invention is compared with the following baselines:

[0094] The One_none input contains only one triple, consisting of a head entity, a relation, and a tail entity. The input and task definition are included during prompt construction.

[0095] The One_Key input includes a triplet and adds historical keywords. The prompt construction includes both the input and the task definition.

[0096] The One_ICL input includes a triplet with added historical keywords. The prompt construction includes the input, task definition, and sample. Samples are provided based on context learning, with each sample containing the input, task definition, and output.

[0097] The Many_None input is a related subgraph of triples. The input and task definitions are included during the prompt construction.

[0098] The Many_Key input is a related subgraph of triples, with historical keywords added. The prompt construction includes both the input and task definition.

[0099] Figure 4 The automatic evaluation results are for the baseline algorithm and the final algorithm Many-ICL. Figure 5 The results show the manual evaluation of the baseline algorithm and the final algorithm, Many-ICL. In both automatic and manual evaluations, the Many-ICL algorithm achieved optimal results for various subject types and evaluation metrics, demonstrating the effectiveness of the algorithm proposed in this invention.

[0100] The data augmentation strategies proposed in this invention, including increasing the number of triples, adding keywords, and employing contextual learning, all improve the algorithm's performance to some extent, demonstrating the effectiveness of the data augmentation strategies. (Observation) Figure 6 The multiple-choice question examples show that by adding subject-specific keywords, the effective knowledge in the triples is supplemented, improving the effectiveness and accuracy of the generated questions compared to algorithms without keywords. By employing contextual learning, the key content tested in the questions and the format of the options are more standardized, highlighting the key points more clearly. Increasing the number of triples greatly enhances the diversity of the generated questions, particularly evident in the options.

[0101] Example 2

[0102] This embodiment provides a test question intelligent generation system based on a large language model.

[0103] A test question intelligent generation system based on a large language model includes:

[0104] The keyword extraction module is configured to extract keywords, including time, people, and ideas, based on historical textbook data.

[0105] The knowledge graph construction module is configured to: extract the relationships between the keywords based on the keywords and output them in the form of triples to construct a knowledge graph;

[0106] The relevant subgraph generation module is configured to: perform a breadth-first search in the knowledge graph based on the entity part in the specified triplet, find all triplets in which the entity part appears, construct a first triplet set, calculate the relevance score between each triplet in the first triplet set and the specified triplet, and if the relevance score is greater than a set threshold, add the triplet to a second triplet set and add the relevance score to a score set; after the breadth-first search, sort the scores in the score set in descending order to obtain the sorted index sequence; if the length of the second triplet set is greater than a set value, truncate the set to the latest index set and randomly select several indices from it; otherwise, directly truncate the first few indices; select the corresponding triplets according to the indexes to form a relevant subgraph;

[0107] The intelligent generation module is configured to generate questions and corresponding answers based on the relevant subgraphs, preset question types, and question prompts, using a large language model.

[0108] It should be noted that the keyword extraction module, knowledge graph construction module, related subgraph generation module, and intelligent generation module described above are the same examples and application scenarios implemented in the steps of Embodiment 1, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer-executable instructions.

[0109] Example 3

[0110] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the intelligent test question generation method based on a large language model as described in Embodiment 1 above.

[0111] Example 4

[0112] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the intelligent test question generation method based on a large language model as described in Embodiment 1 above.

[0113] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0114] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0115] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0116] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0117] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0118] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for intelligent test item generation based on a large language model, characterized in that, include: Based on historical textbook data, keywords were extracted, including time, people, and ideas; Based on the keywords, the relationships between the keywords are extracted and output in the form of triples to construct a knowledge graph. Based on the entity portion of the specified triples, a breadth-first search is performed in the knowledge graph to find all triples in which the entity portion appears, constructing a first triplet set. The relevance score between each triple in the first triplet set and the specified triplet is calculated. If the relevance score is greater than a set threshold, the triplet is added to a second triplet set, and the relevance score is added to a score set. After the breadth-first search, the scores in the score set are sorted in descending order to obtain a sorted index sequence. If the length of the second triplet set is greater than a set value, the set is truncated to the latest index set, and several indices are randomly selected from it; otherwise, the first few indices are directly truncated. Triples are selected according to the indices to form a relevance subgraph. Based on the relevant subgraphs, preset question types, and question prompts, a large language model is used to generate questions and corresponding answers. The relevance score is the average of the character relevance score and the semantic relevance score; the character relevance score is the Jaccard coefficient between the concatenated text of the beginning and end of two triples; the semantic relevance score is calculated by encoding the beginning and end entities of each triple using a language model, calculating the cosine distance between the encodings, and taking the maximum cosine distance as the semantic relevance score. The process of obtaining the relevant subgraph uses the following formula: in, Indicate its keywords, Indicates its head entity, Indicates its tail entity, This represents the relationship between the head and tail entities, T represents the specified triple, B represents the number of layers in the breadth-first search on the knowledge graph, N represents the number of triples in the relevant subgraph, θ represents the threshold for the relevance score, and KG represents the knowledge graph.

2. The intelligent test question generation method based on a large language model according to claim 1, characterized in that, The process of generating questions and corresponding answers based on the relevant subgraphs, preset question types, and question prompts, using a large language model, employs the following formula: in, Represents the large language model, L represents the relevant subgraph, D represents the question type, and P represents the relevant subgraph. Q This indicates a prompt for the test question, and KG represents a knowledge graph.

3. The intelligent test question generation method based on a large language model according to claim 1, characterized in that, During the breadth-first search, if the entity part of a triple in the second triple set does not appear in the existing entity set, then the entity part of that triple is added to the existing entity set.

4. The intelligent test question generation method based on a large language model according to claim 1, characterized in that, Before constructing the knowledge graph, the process also includes: collecting historical textbooks, converting them from PDF to TXT format to retain the effective text; and based on the effective text, segmenting the text according to the paragraph relationships to convert the effective text into paragraphs containing several words.

5. The intelligent test question generation method based on a large language model according to claim 4, characterized in that, The process of preserving valid text includes removing sidebars, chapter exercises, and illustrations from historical textbooks.

6. A test question intelligent generation system based on a large language model, characterized in that, include: The keyword extraction module is configured to extract keywords based on historical textbook data, including time, people, and ideas. The knowledge graph construction module is configured to: extract the relationships between the keywords based on the keywords and output them in the form of triples to construct a knowledge graph; The relevant subgraph generation module is configured to: perform a breadth-first search in the knowledge graph based on the entity part in the specified triplet, find all triplets in which the entity part appears, construct a first triplet set, calculate the relevance score between each triplet in the first triplet set and the specified triplet, and if the relevance score is greater than a set threshold, add the triplet to a second triplet set and add the relevance score to a score set; after the breadth-first search, sort the scores in the score set in descending order to obtain the sorted index sequence; if the length of the second triplet set is greater than a set value, truncate the set to the latest index set and randomly select several indices from it; otherwise, directly truncate the first few indices; select the corresponding triplets according to the indexes to form a relevant subgraph; The intelligent generation module is configured to generate questions and corresponding answers based on the relevant subgraphs, preset question types, and question prompts, using a large language model. The relevance score is the average of the character relevance score and the semantic relevance score; the character relevance score is the Jaccard coefficient between the concatenated text of the beginning and end of two triples; the semantic relevance score is calculated by encoding the beginning and end entities of each triple using a language model, calculating the cosine distance between the encodings, and taking the maximum cosine distance as the semantic relevance score. The process of obtaining the relevant subgraph uses the following formula: in, Indicate its keywords, Indicates its head entity, Indicates its tail entity, This represents the relationship between the head and tail entities, T represents the specified triple, B represents the number of layers in the breadth-first search on the knowledge graph, N represents the number of triples in the relevant subgraph, θ represents the threshold for the relevance score, and KG represents the knowledge graph.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the intelligent test question generation method based on a large language model as described in any one of claims 1-5.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the intelligent test question generation method based on a large language model as described in any one of claims 1-5.

Images