A hierarchical automatic generation method of textbook knowledge graph based on large language model

Abstract

The present invention provides a method for automatically generating a hierarchical textbook knowledge graph based on a large language model, which belongs to the field of knowledge graph technology. By customizing and fine-tuning the Qwen2.5 model, the natural language understanding of the textbook is better performed, the logical relationship between knowledge points is accurately extracted, and the attribute description of each knowledge point is summarized, so that the generated knowledge graph can clearly present the knowledge points involved in the textbook, the relationship between knowledge points, and the attributes of the knowledge points; by designing a hierarchical extraction process to achieve coarse and fine granularity control, the directory structure and segmentation strategy are used to gradually extract directories, entities, relationships, and attributes, ensuring that knowledge points that can be directly used by teachers in teaching can be directly extracted, avoiding the problem of extracting named entities that are not related to teaching when extracting knowledge points; by designing detailed and strict extraction instructions, the extraction range is constrained to ensure strong semantic binding between attributes and entities, so that the model can better understand the task requirements and thus reduce deviations in the extraction process.

Description

Technical Field

[0001] The present invention relates to the field of knowledge graph technology, and in particular to a method for automatically generating hierarchical teaching material knowledge graphs based on a large language model. Background Art

[0002] The market size of the smart education industry is growing rapidly. According to a report from the China Research Institute of Industry and Information Technology, the scale of China's smart education market will exceed 900 billion yuan by 2025, with an annual compound growth rate of about 21%, showing broad development space.

[0003] With the rapid growth of the smart education market, higher requirements are placed on the in-depth mining and intelligent application of educational knowledge. It has become a development trend to present knowledge in the field of education in a structured form through knowledge graph extraction solutions to provide strong support for the digitalization of education.

[0004] Large-scale pre-trained language models such as GPT-4 have achieved remarkable results in knowledge graph extraction, promoting a new paradigm for AI applications. However, due to their openness and hallucinations, large models still face many challenges in their implementation. This is especially true for the construction of textbook knowledge graphs, where three core issues remain:

[0005] (1) Dynamic hierarchical adaptability is low and the cost of domain migration is high. Existing methods rely on subject templates and are difficult to model the heterogeneous hierarchical structure of interdisciplinary textbooks. Zero-shot solutions require manual design of prompt words, and adapting to new textbooks takes an average of a long time.

[0006] (2) The existing technology based on natural language understanding is called named entity recognition when extracting knowledge point entities. It will recognize all nouns in a sentence, but not every noun is a key course knowledge point. Therefore, the granularity of the extraction is very critical.

[0007] (3) In the process of automatic generation of knowledge graphs, most of them can only extract knowledge point entities from the textbook directory structure without understanding the textbook content. There are obvious deficiencies in the extraction of more fine-grained knowledge points, and it is difficult to deeply explore the detailed knowledge point information below the directory level. Summary of the Invention

[0008] In response to the problems existing in the prior art, the present invention provides a method for automatically generating hierarchical textbook knowledge graphs based on a large language model. By fine-tuning the Qwen2.5 model and designing hierarchical extraction processes and extraction instructions, high-precision automatic generation of textbook knowledge graphs can be achieved.

[0009] The present invention provides a method for automatically generating a hierarchical teaching material knowledge graph based on a large language model, comprising:

[0010] Step 1: Use the general large model to extract knowledge points from the sample textbook text and conduct manual review. The extracted knowledge points are used as annotation information for the sample textbook text to obtain training data;

[0011] Step 2: Select the Qwen2.5 model as the textbook knowledge graph extraction model, and use the LLaMA-Factory tool to customize and fine-tune the Qwen2.5 model based on the training data;

[0012] Step 3: Design a hierarchical extraction process and extraction instructions, and extract the textbook knowledge graph of the textbook text to be processed based on the hierarchical extraction process and extraction instructions and the Qwen2.5 model. The hierarchical extraction process includes directory structure extraction, directory hierarchical relationship extraction, text segmentation and directory correspondence, knowledge point entity extraction in the directory, relationship extraction between knowledge point entities, and knowledge point entity attribute extraction.

[0013] Optionally, in step 2, the LLaMA-Factory tool is used to customize and fine-tune the Qwen2.5 model in combination with the training data, including:

[0014] The LoRA fine-tuning method is used, lora_target is set to all, SFT is used for supervised fine-tuning, the template is glm4, the maximum truncation length of the textbook text is set to 4000, 16 working processes are set for parallel processing in the data preprocessing part, the learning rate is set to 1.0e-4 in the training part, the training batch size of each device is set to 1, the gradient of 8 batches is accumulated before updating, the cosine learning rate scheduler is used to adjust the learning rate, and the learning rate warm-up ratio is 0.1. In the evaluation part, 10% of the data is divided from the training data set as the validation set, the batch size of each device during evaluation is set to 1, and an evaluation is set every 500 training steps. The evaluation indicators are calculated using the validation set, the output directory, logging interval and checkpoint save steps are set to monitor the training status in real time and save key training results.

[0015] Optionally, in step three, extracting the textbook knowledge graph of the textbook text to be processed based on the hierarchical extraction process and extraction instructions and the Qwen2.5 model includes:

[0016] Directory structure extraction: Taking the textbook text to be processed as input, the Qwen2.5 model is used to extract the directory, and list derivation is used to extract all directory knowledge point entities and store them in a list;

[0017] Extracting directory hierarchical relationships: Filter the directory list to remove empty directories, identify all chapter directories based on their format, identify the subdirectories contained in the chapter directory based on the common features between the chapter directory and other directories, traverse all remaining directories, and add the hierarchical relationships between all directories to the relationship set;

[0018] Text segmentation corresponds to catalog: divide the textbook text according to the catalog hierarchy, and assign each minimum level catalog to the corresponding text segment;

[0019] Knowledge point entity extraction in the catalog: Design a text knowledge point entity extraction instruction and call the Qwen2.5 model to extract knowledge point entities in the text fragment according to the requirements of the text knowledge point entity extraction instruction. The text knowledge point entity extraction instruction specifies the knowledge point type, constraints and output format of the text knowledge point entity extraction.

[0020] Extracting relationships between knowledge point entities: Designing instructions for extracting relationships between knowledge point entities, and calling the Qwen2.5 model to extract relationships between knowledge point entities according to the requirements of the instructions. Generate relationship triples based on the extraction results of the relationships between knowledge point entities, where the instructions for extracting relationships between knowledge point entities specify the relationship type, constraints, and output format.

[0021] Knowledge point entity attribute extraction: Design knowledge point entity attribute extraction instructions, call the Qwen2.5 model to extract knowledge point entity attributes according to the knowledge point entity attribute instructions, and generate attribute triples based on the knowledge point entity attribute extraction results, wherein the knowledge point entity attribute extraction instructions specify the attribute type, constraints and output format.

[0022] After adopting the above technical solution, the present invention has at least the following beneficial effects:

[0023] (1) By training and customizing the Qwen2.5 model, the model is better adapted to the specific task of extracting knowledge graphs from textbooks, can better understand the natural language of textbooks, and improves the performance of the model in educational knowledge extraction tasks.

[0024] (2) By designing a hierarchical extraction process to achieve coarse and fine granularity control, we use the textbook directory structure and segmentation strategy to gradually extract directories, entities, relationships, and attributes, ensuring that the extraction process is more accurate and granular, and can accurately extract knowledge points in the textbook rather than named entities in natural language understanding.

[0025] (3) By designing detailed and strict extraction instructions, the fine-tuned Qwen2.5 model can better understand the requirements of educational knowledge tasks, accurately extract the logical relationship between knowledge points, and summarize the attribute description of each knowledge point, so that the entire knowledge graph can clearly display the knowledge points involved in the course, the relationship between knowledge points, and the description of knowledge points, which is convenient for teachers to teach and students to self-study. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0027] Figure 1 A flowchart of a method for automatically generating a hierarchical textbook knowledge graph based on a large language model according to an embodiment of the present disclosure;

[0028] Figure 2 The confusion matrix of the prediction values ​​of the five models for the experimental textbook text, (a) is the confusion matrix of the prediction values ​​of the kimi model for the experimental textbook text, (b) is the confusion matrix of the prediction values ​​of the Wenxin Yiyan model for the experimental textbook text, (c) is the confusion matrix of the prediction values ​​of the Tongyi Qianwen model for the experimental textbook text, (d) is the confusion matrix of the prediction values ​​of the DeepSeek model for the experimental textbook text, and (e) is the confusion matrix of the prediction values ​​of the fine-tuned Qwen2.5 model for the experimental textbook text;

[0029] Figure 3 The ROC curves of the five models for the experimental teaching material text. DETAILED DESCRIPTION

[0030] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0031] refer to Figure 1 The present disclosure provides a method for automatically generating a hierarchical textbook knowledge graph based on a large language model, including:

[0032] Step 1: Use the general large model to extract knowledge points from the sample textbook text and conduct manual review. The extracted knowledge points are used as annotation information for the sample textbook text to obtain training data;

[0033] In this embodiment, knowledge points are extracted from the textbook as training data for training the textbook knowledge graph extraction model. This data is highly relevant to the actual knowledge extraction task, enabling the model to better learn the specific patterns and features of the textbook;

[0034] Step 2: Select the Qwen2.5 model as the textbook knowledge graph extraction model, and use the LLaMA-Factory tool to customize and fine-tune the Qwen2.5 model based on the training data;

[0035] In this embodiment, in order to make the Qwen2.5 model better adapt to the specific tasks and language styles in the education field and improve the model's performance in educational knowledge extraction tasks, the Qwen2.5 model is customized and fine-tuned;

[0036] Specifically, the LoRA fine-tuning method is adopted to adapt to specific tasks by adding low-rank matrices to the pre-trained model instead of directly fine-tuning all parameters. The advantage is that it does not significantly increase the number of model parameters, and can effectively adjust the model to adapt to specific domain knowledge extraction tasks; setting lora_target to all, low-rank adaptation is performed on all target layers of the model, so that the model can better learn knowledge and patterns in specific fields without increasing too much computational burden; using SFT for supervised fine-tuning, it provides many convenient functions and configuration options to effectively optimize model performance; using the template glm4, which defines the format and processing method of the data, including text preprocessing, adding special tags, etc.; setting the maximum truncation length of the textbook text to 4000, and text exceeding this length will be truncated to control the input data scale to improve training efficiency; in the data preprocessing part, 16 working processes are set for parallel processing to speed up data preprocessing; in the training part, the learning rate is set to 1.0e-4, the training batch size of each device is set to 1, the gradient of 8 batches is accumulated before updating, the cosine learning rate scheduler is used to adjust the learning rate, and the learning rate warm-up ratio is 0.1 to ensure that the model converges stably during fine-tuning, avoid overfitting, and improve the model's understanding and extraction capabilities of text in specific fields; in the evaluation part, 10% of the data is divided from the training data set as the validation set, the batch size of each device during evaluation is set to 1, and an evaluation is set every 500 training steps. The evaluation indicators are calculated using the validation set, and the output directory, logging interval, and checkpoint save steps are set to monitor the training status in real time and save key training results;

[0037] Step 3: Design a hierarchical extraction process and extraction instructions, and extract the textbook knowledge graph of the to-be-processed textbook text based on the hierarchical extraction process, extraction instructions, and the Qwen2.5 model. Among them, the hierarchical extraction process includes directory structure extraction, directory hierarchical relationship extraction, text segmentation and directory correspondence, knowledge point entity extraction within the directory, relationship extraction between knowledge point entities, and attribute extraction of knowledge point entities;

[0038] In this embodiment, fine and coarse granularity control is achieved through hierarchical extraction. Through detailed and strict extraction instructions, the model can better understand the task requirements and reduce deviations during the extraction process. The specific process is as follows:

[0039] (1) Directory structure extraction

[0040] Purpose: Extract directory nodes from the original input text to provide a basic framework for subsequent hierarchical extraction;

[0041] Method: Taking the to-be-processed textbook text as input, use the Qwen2.5 model to extract the directory, and use list comprehension to extract all directory knowledge point entities and store them in a list;

[0042] (2) Directory hierarchical relationship extraction

[0043] Purpose: Identify the inclusion relationships between directories, construct the hierarchical structure between chapter and section directories and their subordinate directories, and provide a basis for the hierarchical organization of the knowledge graph;

[0044] Method: Filter the directory list to remove empty directories. Identify all chapter and section directories according to the format of chapter and section directories. Identify the subordinate directories included in the chapter and section directories according to the common characteristics of chapter and section directories and other directories. Traverse all the remaining directories and add the hierarchical relationships between all directories to the relationship set. For example, by judging whether the directory string starts with "Chapter" and contains the character "Chapter", find all chapter and section directories, extract the chapter prefix from the chapter and section directories, such as "Chapter 3". For each chapter and section directory, extract the digital part of the chapter number, and find the directory starting with this number plus "." and with a length of 2 after splitting (i.e., subordinate directory 1). Traverse other directories in the same way. For each directory, find the next-level directory starting with the prefix of this directory (excluding itself), and add the inclusion relationship between them to the relationship set;

[0045] (3) Text segmentation and directory correspondence

[0046] Purpose: Divide the input textbook text into multiple segments according to the extracted directory structure, so that each text segment corresponds to a directory node, providing a local text basis for subsequent entity, relationship, and attribute extraction;

[0047] Method: Divide the textbook text according to the directory hierarchy, find the position of each directory node in the text, traverse all directories, and based on the position of the current directory and the position of the next directory, cut out the text fragment corresponding to the current directory from the text and store it in a dictionary with the directory as the key and the corresponding text fragment as the value, until each minimum-level directory is assigned to the corresponding text fragment;

[0048] (4) Knowledge point entity extraction in the catalog

[0049] Purpose: Extract coarse-grained knowledge point entities from the text fragments corresponding to each directory to provide core node information for the knowledge graph;

[0050] Methods: We designed a text knowledge point entity extraction instruction, explicitly requiring the model to extract knowledge point entities that are fully defined and described in the text. We also provided examples to constrain the model's extraction standards and scope. We also specified output format requirements to ensure that the model returned standardized results. We then called the Qwen2.5 model to extract knowledge point entities from text fragments according to the requirements of the text knowledge point entity extraction instruction.

[0051] For example, instruction=(

[0052] f"Extract coarse knowledge point entities based on the following text."

[0053] f"These knowledge points should summarize higher-level content rather than specific details."

[0054] f"Knowledge point examples: Coarse-grained knowledge points: Sequence structure, basic steps of programming. Non-coarse-grained knowledge points (not extracted): Finiteness, single branch, loop conditions, characteristics, etc.."

[0055] Note: 1. Do not extract simple proper nouns or phrases. The knowledge point entities to be extracted should have strong definitions and descriptions in the text, usually concepts, theories, methods, etc.

[0056] f"2. Output format: (knowledge point entity 1, knowledge point entity 2, ...)"

[0057] f"3. Output according to the output format, do not output additional instructions" )

[0059] (5) Extracting relationships between knowledge point entities

[0060] Purpose: Identify the relationships between knowledge point entities (such as "include", "predecessor", "successor", and "parallel") to provide rich semantic connections for knowledge graph construction;

[0061] Methods: We designed an instruction for extracting relations between knowledge point entities, explicitly requiring the model to extract relations between knowledge point entities and specifying the relation type and output format. At the same time, we emphasized that only previously extracted knowledge point entities and specified relation types could be used to ensure the accuracy and standardization of the extraction results. The Qwen2.5 model was called to extract relations between knowledge point entities according to the requirements of the instruction, and relation triples were generated based on the relations extracted between knowledge point entities.

[0062] For example, instruction = (

[0063] f"Based on the following text and knowledge point entities, extract the relationship between these knowledge point entities (including, predecessor, successor, parallel),"

[0064] The output format of the relation is a triple: (knowledge point entity-relationship-knowledge point entity), for example: (sequential structure-contains-list), (function-contains-parameter)..."

[0065] Note: 1. The two knowledge point entities corresponding to the extracted triples must use the input knowledge point entities. Do not use unlisted entities. Do not use them. If there is no relationship, do not output it.

[0066] f"2. Do not extract relation types not listed. "

[0067] f"3. Output in Chinese."

[0068] f"4. Output according to the output format, do not output additional instructions." )

[0070] (6) Knowledge point entity attribute extraction

[0071] Purpose: To extract attribute information of each knowledge point entity from the text, such as definition, description, function, etc., to provide detailed attribute descriptions for the nodes in the knowledge graph and enrich the semantic information of the knowledge graph.

[0072] Method: A knowledge point entity attribute extraction instruction was designed to explicitly require the model to extract the attributes of the knowledge point entity and specify the attribute type and output format. At the same time, it was emphasized that the attribute content must be directly derived from the original text and self-generation must be eliminated to ensure the accuracy and reliability of the extraction results. The Qwen2.5 model was called to extract the knowledge point entity attributes according to the knowledge point entity attribute instruction, and attribute triples were generated according to the knowledge point entity attribute extraction results. Among them, the knowledge point entity attribute extraction instruction specified the attribute type, constraints and output format.

[0073] For example, instruction = (

[0074] f"Based on the following text and knowledge point entities, extract the attributes of these entities (attribute type: definition, description, characteristics, properties, classification, principle, function, steps, case (practical application), example (simple example)),"

[0075] f"Output format: (knowledge point entity-attribute type-attribute), for example: (feature data type-definition-feature data type refers to the composite data type other than the basic data type in Python, including lists, tuples, dictionaries, and sets.)"

[0076] Note: 1. The knowledge point entity part of the triple must use the input knowledge point entity. Do not extract attributes of other entities.

[0077] f"2. Output in Chinese."

[0078] f"3. Output according to the output format, do not output additional instructions."

[0079] 4. The extracted attributes must come from the original text. Do not generate them yourself. If the entity does not have attribute content of the corresponding attribute type in the text, do not extract it.

[0080] f"5. If the knowledge point has attributes, output all of them."

[0081] f"5. If the knowledge point you input has no attributes, you do not need to list them." )

[0083] The following specific examples are provided in conjunction with the above embodiments. It can be understood that the following specific examples are merely illustrative of the specific implementation of the above embodiments, and are not intended to limit the technical solutions of the above embodiments.

[0084] Select "Python Language Programming (Second Edition)" as the input text, select DeepSeek, Kimi, Tongyi Qianwen, Wenxin Yiyan four large models and compare them with the output triples of the fine-tuned Qwen2.5 large model. The prompt word is set to "Based on the text of Chapter 3, generate triples with inclusion relationships between knowledge point entities in the directory". Enter the text data of Chapter 3 respectively, set TP as the triples correctly output by the large model (true positive examples), FP as the value output by the model but not in the standard triples (false positive examples), FN as the value not output by the model but in the standard triples (false negative examples), and TN as the value not output by the standard triples and not output by the model (true negative examples). Figure 2 As shown in the figure, the output confusion matrices of the five models are obtained by establishing a confusion matrix. The precision (P), recall (R) and harmonic mean (F1) of the five models are calculated as evaluation indicators based on the confusion matrix. The calculation formula is as follows:

[0085]

[0086]

[0087]

[0088] The evaluation indicators of the five models are statistically analyzed, and the experimental results are shown in Table 1.

[0089] Table 1 Statistical results of evaluation indicators of five models

[0090]

[0091] As can be seen from Table 1, the recall rate of the fine-tuned Qwen2.5 model reaches 91.25%, and the precision rate of 86.96% is significantly higher than other models. The F1 value of 89.02% is the best performance, which verifies the accuracy and effectiveness of the fine-tuned Qwen2.5 model in the automatic generation of textbook knowledge graphs.

[0092] The false positive rate (FPR) and true positive rate (TPR) of the five models are further calculated based on the confusion matrix. The calculation formula is as follows:

[0093]

[0094]

[0095] Figure 3 In order to obtain the ROC curves of the five models based on the false positive rate (FPR) and true positive rate (TPR) of the five models, Figure 3 It can be seen that the ROC curve of the fine-tuned Qwen2.5 model is at the top, with the highest TPR and the lowest FPR. It performs best under all thresholds and has the best overall performance.

[0096] The embodiments of the present invention have so far described the technical solutions of the present invention in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present invention.

Claims

1. A method for automatically generating hierarchical teaching material knowledge graphs based on a large language model, characterized in that: include: Step 1: Use the general large model to extract knowledge points from the sample textbook text and conduct manual review. The extracted knowledge points are used as annotation information for the sample textbook text to obtain training data; Step 2: Select the Qwen2.5 model as the textbook knowledge graph extraction model, and use the LLaMA-Factory tool to customize and fine-tune the Qwen2.5 model based on the training data; Step 3: Design a hierarchical extraction process and extraction instructions, and extract the textbook knowledge graph of the textbook text to be processed based on the hierarchical extraction process and extraction instructions and the Qwen2.5 model, wherein the hierarchical extraction process includes directory structure extraction, directory hierarchical relationship extraction, text segmentation and directory correspondence, knowledge point entity extraction in the directory, relationship extraction between knowledge point entities, and knowledge point entity attribute extraction. The text knowledge point entity extraction instruction specifies the knowledge point type, constraint and output format of the text knowledge point entity extraction, the knowledge point entity relationship extraction instruction specifies the relationship type, constraint and output format, and the knowledge point entity attribute extraction instruction specifies the attribute type, constraint and output format; The LLaMA-Factory tool was used to customize and fine-tune the Qwen2.5 model based on the training data, including: The LoRA fine-tuning method is adopted to adapt to specific tasks by adding low-rank matrices to the pre-trained model instead of directly fine-tuning all parameters. The LoRA fine-tuning method is adopted, lora_target is set to all, SFT is used for supervised fine-tuning, the template is glm4, the maximum truncation length of the textbook text is set to 4000, 16 working processes are set for parallel processing in the data preprocessing part, the learning rate is set to 1.0e-4 in the training part, the training batch size of each device is set to 1, the gradient of 8 batches is accumulated before updating, the cosine learning rate scheduler is used to adjust the learning rate, and the learning rate warm-up ratio is 0.

1. In the evaluation part, 10% of the data is divided from the training data set as the validation set, the batch size of each device during evaluation is set to 1, and an evaluation is set every 500 training steps. The evaluation indicators are calculated using the validation set, the output directory, logging interval and checkpoint save steps are set to monitor the training status in real time and save key training results.

2. The method for automatically generating hierarchical teaching material knowledge graphs based on a large language model according to claim 1 is characterized in that: In step 3, the textbook knowledge graph of the textbook text to be processed is extracted based on the hierarchical extraction process and extraction instructions and the Qwen2.5 model, including: Directory structure extraction: Taking the textbook text to be processed as input, the Qwen2.5 model is used to extract the directory, and list derivation is used to extract all directory knowledge point entities and store them in a list; Extracting directory hierarchical relationships: Filter the directory list to remove empty directories, identify all chapter directories based on their format, identify the subdirectories contained in the chapter directory based on the common features between the chapter directory and other directories, traverse all remaining directories, and add the hierarchical relationships between all directories to the relationship set; Text segmentation corresponds to catalog: divide the textbook text according to the catalog hierarchy, and assign each minimum level catalog to the corresponding text segment; Knowledge point entity extraction in the catalog: Design text knowledge point entity extraction instructions, and call the Qwen2.5 model to extract knowledge point entities in the text fragment according to the requirements of the text knowledge point entity extraction instructions; Extraction of relations between knowledge point entities: Design instructions for extracting relations between knowledge point entities, and call the Qwen2.5 model to extract relations between knowledge point entities according to the requirements of the instructions, and generate relation triples based on the extraction results of relations between knowledge point entities. Knowledge point entity attribute extraction: Design knowledge point entity attribute extraction instructions, call the Qwen2.5 model to extract knowledge point entity attributes according to the knowledge point entity attribute instructions, and generate attribute triples based on the knowledge point entity attribute extraction results.

Images