Textbook knowledge graph hierarchical automatic generation method based on large language model

Abstract

The invention provides a hierarchical automatic generation method for a teaching material knowledge graph based on a large language model, belongs to the technical field of knowledge graphs, and aims at better performing natural language understanding on a teaching material, accurately extracting a logical relationship among knowledge points and summarizing attribute description of each knowledge point by performing customized fine tuning on a Qwen2.5 model. The generated knowledge graph can clearly present knowledge points related to the textbook, the relationship between the knowledge points and the attributes of the knowledge points; according to the method, coarse and fine granularity control is realized by designing a hierarchical extraction process, directories, entities, relationships and attributes are extracted step by step by utilizing a directory structure and a segmentation strategy, it is ensured that knowledge points which can be directly used in teaching of teachers can be directly extracted, and the problem that named entities irrelevant to teaching are extracted when the knowledge points are extracted is avoided; by designing detailed and strict extraction instructions, the extraction range is constrained to ensure strong semantic binding of attributes and entities, so that the model better understands task requirements, and deviation is reduced in the extraction process.

Description

Technical Field

[0001] The present invention relates to the technical field of knowledge graphs, and in particular to a hierarchical automatic generation method for teaching material knowledge graphs based on large language models. Background Art

[0002] With the rapid growth of the market scale of the intelligent education industry, according to the report prediction of the CRI Industry Research Institute, by 2025, the market scale of China's intelligent education will exceed 900 billion yuan, with an annual compound growth rate of about 21%, showing broad development space.

[0003] With the rapid growth of the market scale of intelligent education, higher requirements are put forward for the in-depth excavation and intelligent application of educational knowledge. Through the knowledge graph extraction scheme, presenting the knowledge in the education field in a structured form to provide strong support for education digitization has become a development trend.

[0004] Currently, large pre-trained language models such as GPT-4 have achieved amazing results in knowledge graph extraction, leading to a new paradigm of artificial intelligence applications. However, due to their openness and hallucination phenomena, large models still encounter many challenges in domain implementation, especially in the construction of teaching material knowledge graphs, and there are still three core problems: (1) Low dynamic hierarchical adaptability and high domain migration cost. Existing methods rely on subject templates and it is difficult to model the heterogeneous hierarchical structures of interdisciplinary teaching materials. Zero-shot and other schemes require manual design of prompt words, and the average time-consuming for adapting to new teaching materials is long.

[0005] (2) Existing natural language understanding-based technologies, when extracting knowledge point entities, called named entity recognition, will identify all nouns in a sentence, but not every noun is a key curriculum knowledge point. Therefore, the extraction granularity is very crucial.

[0006] (3) In the process of automatic generation of knowledge graphs, most can only extract knowledge point entities from the teaching material table of contents structure and do not understand the teaching material content, and there are obvious deficiencies in the extraction of more fine-grained knowledge points, making it difficult to deeply excavate the detailed knowledge point information below the table of contents level. Summary of the Invention

[0007] Aiming at the problems existing in the prior art, the present invention provides a hierarchical automatic generation method for teaching material knowledge graphs based on large language models, which realizes the automatic generation of highly accurate teaching material knowledge graphs by fine-tuning the Qwen2.5 model, designing a hierarchical extraction process and extraction instructions.

[0008] The present invention provides a hierarchical automatic generation method for teaching material knowledge graphs based on large language models, including: Step 1: Use a general large model to extract knowledge points from the sample textbook text and conduct manual review. Take the extracted knowledge points as the annotation information of the sample textbook text to obtain training data; Step 2: Select the Qwen2.5 model as the textbook knowledge graph extraction model, and use the LLaMA-Factory tool to perform customized fine-tuning on the Qwen2.5 model in combination with the training data; Step 3: Design a hierarchical extraction process and extraction instructions, and extract the textbook knowledge graph of the textbook text to be processed based on the hierarchical extraction process, extraction instructions, and the Qwen2.5 model. Among them, the hierarchical extraction process includes table of contents structure extraction, table of contents hierarchy relationship extraction, text segmentation and table of contents correspondence, knowledge point entity extraction within the table of contents, relationship extraction between knowledge point entities, and attribute extraction of knowledge point entities.

[0009] Optionally, in Step 2, using the LLaMA-Factory tool to perform customized fine-tuning on the Qwen2.5 model in combination with the training data includes: Adopt the LoRA fine-tuning method, set lora_target to all, use SFT for supervised fine-tuning, use the template glm4, set the maximum truncation length of the textbook text to 4000, set 16 worker processes for parallel processing in the data preprocessing part, set the learning rate to 1.0e-4 in the training part, set the training batch size for each device to 1, set to update after accumulating gradients for 8 batches, use the cosine learning rate scheduler to adjust the learning rate, and the learning rate warm-up ratio is 0.1. In the evaluation part, divide 10% of the data from the training dataset as the validation set, set the batch size for each device during evaluation to 1, set to perform an evaluation every 500 training steps, calculate the evaluation metrics using the validation set, set the output directory, log recording interval, and checkpoint saving steps to monitor the training status in real time and save key training results.

[0010] Optionally, in Step 3, extracting the textbook knowledge graph of the textbook text to be processed based on the hierarchical extraction process, extraction instructions, and the Qwen2.5 model includes: Table of contents structure extraction: Take the textbook text to be processed as the input, use the Qwen2.5 model to extract the table of contents, and use list comprehension to extract all table of contents knowledge point entities and store them in a list; Table of contents hierarchy relationship extraction: Filter the table of contents list to remove empty directories, identify all chapter and section directories according to the format of the chapter and section table of contents, identify the subdirectories included in the chapter and section directories according to the common characteristics of the chapter and section directories and other directories, traverse all the remaining directories, and add the hierarchical relationships between all directories to the relationship set; Text segmentation corresponding to the table of contents: Divide the textbook text according to the table of contents hierarchy, and assign each lowest-level table of contents to the corresponding text segment; Extraction of knowledge point entities in the table of contents: Design instructions for extracting knowledge point entities in the text, and call the Qwen 2.5 model to extract the knowledge point entities in the text segment according to the requirements of the instructions for extracting knowledge point entities in the text. Among them, the instructions for extracting knowledge point entities in the text specify the knowledge point types, constraints, and output formats for extracting knowledge point entities; Extraction of relationships between knowledge point entities: Design instructions for extracting relationships between knowledge point entities, and call the Qwen 2.5 model to extract the relationships between knowledge point entities according to the requirements of the instructions for extracting relationships between knowledge point entities. Generate relationship triples based on the extraction results of the relationships between knowledge point entities. Among them, the instructions for extracting relationships between knowledge point entities specify the relationship types, constraints, and output formats; Extraction of knowledge point entity attributes: Design instructions for extracting knowledge point entity attributes, call the Qwen 2.5 model to extract knowledge point entity attributes according to the instructions for knowledge point entity attributes, and generate attribute triples based on the extraction results of knowledge point entity attributes. Among them, the instructions for extracting knowledge point entity attributes specify the attribute types, constraints, and output formats.

[0011] After adopting the above technical solutions, the present invention has at least the following beneficial effects: (1) By training and customizing and fine-tuning the Qwen 2.5 model, the model better adapts to the specific task of extracting knowledge graphs from textbooks, can better perform natural language understanding of textbooks, and improves the performance of the model in educational knowledge extraction tasks.

[0012] (2) By designing a hierarchical extraction process to achieve fine-grained control, using the textbook table of contents structure and segmentation strategy, gradually extract the table of contents, entities, relationships, and attributes, ensuring that the extraction process is more accurate and the granularity is finer, and can accurately extract the knowledge points in the textbook, rather than the named entities in natural language understanding.

[0013] (3) By designing detailed and strict extraction instructions, the fine-tuned Qwen 2.5 model can better understand the requirements of educational knowledge tasks, accurately extract the logical relationships between knowledge points, and summarize the attribute descriptions of each knowledge point, so that the entire knowledge graph can clearly display the knowledge points involved in the course, the relationships between knowledge points, and the descriptions of knowledge points, facilitating teachers' teaching and students' self-study. Description of the drawings

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0015] Figure 1 It is a schematic flowchart of a method for hierarchically and automatically generating a teaching material knowledge graph based on a large language model provided by an embodiment of the present disclosure; Figure 2 It is a confusion matrix of the predicted values of five models for the experimental teaching material text. (a) is the confusion matrix of the predicted values of the Kimi model for the experimental teaching material text, (b) is the confusion matrix of the predicted values of the Wenxin Yiyan model for the experimental teaching material text, (c) is the confusion matrix of the predicted values of the Tongyi Qianwen model for the experimental teaching material text, (d) is the confusion matrix of the predicted values of the DeepSeek model for the experimental teaching material text, and (e) is the confusion matrix of the predicted values of the fine-tuned Qwen 2.5 model for the experimental teaching material text; Figure 3 It is the ROC curve of five models for the experimental teaching material text. Specific embodiments

[0016] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

[0017] Refer to Figure 1 , an embodiment of the present disclosure provides a method for hierarchically and automatically generating a teaching material knowledge graph based on a large language model, including: Step 1: Use a general large model to extract knowledge points from the sample teaching material text and conduct manual review, and obtain training data with the extracted knowledge points as the annotation information of the sample teaching material text; In this embodiment, knowledge points are extracted from the teaching materials as training data for training the teaching material knowledge graph extraction model. These data are highly relevant to the actual knowledge extraction task, enabling the model to better learn the specific patterns and features of the teaching materials; Step 2: Select the Qwen 2.5 model as the teaching material knowledge graph extraction model, and use the LLaMA-Factory tool to perform customized fine-tuning on the Qwen 2.5 model in combination with the training data; In this embodiment, in order to enable the Qwen2.5 model to better adapt to specific tasks and language styles in the education field and improve the model's performance in educational knowledge extraction tasks, the Qwen2.5 model is customized and fine-tuned; Specifically, the LoRA fine-tuning method is adopted. By adding low-rank matrices to the pre-trained model to adapt to specific tasks instead of directly fine-tuning all parameters, the advantage is that it does not significantly increase the number of model parameters, and at the same time can effectively adjust the model to adapt to specific domain knowledge extraction tasks; set lora_target to all to perform low-rank adaptation on all target layers of the model, so that the model can better learn specific domain knowledge and patterns without adding too much computational burden; use SFT for supervised fine-tuning, which provides many convenient functions and configuration options to effectively optimize the model performance; adopt the template glm4, which defines the data format and processing method, including text preprocessing, addition of special tokens, etc.; set the maximum truncation length of the textbook text to 4000, and texts longer than this length will be truncated to control the scale of the input data and improve the training efficiency; 16 worker processes are set in the data preprocessing part for parallel processing to speed up the data preprocessing speed; in the training part, set the learning rate to 1.0e-4, set the training batch size of each device to 1, set to update after accumulating the gradients of 8 batches, use the cosine learning rate scheduler to adjust the learning rate, and the learning rate warm-up ratio is 0.1 to ensure stable convergence of the model during fine-tuning, avoid overfitting, and improve the model's understanding and extraction ability of specific domain texts; in the evaluation part, 10% of the data is divided from the training dataset as the validation set, set the batch size of each device during evaluation to 1, set to perform an evaluation every 500 training steps, calculate the evaluation metrics using the validation set, and set the output directory, logging interval, and checkpoint saving steps to monitor the training status in real time and save key training results; Step three: Design a hierarchical extraction process and extraction instructions, and extract the textbook knowledge graph of the textbook text to be processed based on the hierarchical extraction process, extraction instructions, and the Qwen2.5 model. Among them, the hierarchical extraction process includes table of contents structure extraction, table of contents hierarchical relationship extraction, text segmentation and correspondence with the table of contents, in-table-of-contents knowledge point entity extraction, relationship extraction between knowledge point entities, and attribute extraction of knowledge point entities; In this embodiment, fine-grained control is achieved through hierarchical extraction. Through detailed and strict extraction instructions, the model can better understand the task requirements and thus reduce deviations during the extraction process. The specific process is as follows: (1) Table of contents structure extraction Purpose: Extract the table of contents nodes from the original input text to provide a basic framework for subsequent hierarchical extraction; Method: Taking the teaching material text to be processed as the input, use the Qwen2.5 model to extract the table of contents, and use list comprehension to extract all the knowledge point entities in the table of contents and store them in a list; (2) Extraction of the hierarchical relationship of the table of contents Purpose: To identify the inclusion relationship between the tables of contents, construct the hierarchical structure between the chapter and section headings and the subordinate headings, and provide a basis for the hierarchical organization of the knowledge graph; Method: Filter the table of contents list to remove empty directories, identify all chapter and section headings according to the format of the chapter and section headings, identify the subordinate headings included in the chapter and section headings according to the common characteristics of the chapter and section headings and other headings, traverse all the remaining headings, and add the hierarchical relationship between all the headings to the relationship set. For example, by judging whether the heading string starts with "Chapter" and contains the character "Chapter", find all chapter and section headings, extract the chapter prefix from the chapter and section headings, such as "Chapter 3", for each chapter and section heading, extract the digital part of the chapter number, and find the headings that start with this number plus "." and have a length of 2 after splitting (i.e., subordinate heading 1). Traverse the other headings in the same way. For each heading, find the next-level headings (excluding itself) that start with the prefix of this heading, and add the inclusion relationship between them to the relationship set; (3) Text segmentation and correspondence with the table of contents Purpose: Divide the input teaching material text into multiple segments according to the extracted table of contents structure, so that each text segment corresponds to a table of contents node, and provide a local text basis for the subsequent extraction of entities, relationships and attributes; Method: Divide the teaching material text according to the table of contents hierarchy, find the position of each table of contents node in the text, traverse all the tables of contents, and cut out the text segment corresponding to the current table of contents from the text according to the position of the current table of contents and the position of the next table of contents, and store it in a dictionary with the table of contents as the key and the corresponding text segment as the value until each smallest-level table of contents is assigned to the corresponding text segment; (4) Extraction of knowledge point entities within the table of contents Purpose: Extract the coarse-grained knowledge point entities from the text segment corresponding to each table of contents, and provide the core node information for the knowledge graph; Method: Design the extraction instruction for text knowledge point entities, clearly require the model to extract the knowledge point entities that are fully defined and described in the text, and give examples to constrain the extraction criteria and scope of the model. At the same time, specify the output format requirements to ensure that the model returns results in a standard way, and call the Qwen2.5 model to extract the knowledge point entities in the text segment according to the requirements of the text knowledge point entity extraction instruction; For example, instruction = ( f"Extract the coarse knowledge point entities according to the following text." f"These knowledge points should summarize higher-level content rather than specific details." f"Knowledge point example: Coarse-grained knowledge point: Sequential structure, the basic steps of programming. Non-coarse-grained knowledge points (not extracted): Finiteness, single-branch, loop condition, characteristics, etc." f"Note: 1. Do not extract simple proper nouns or phrases. The knowledge point entities to be extracted should have strong definitions and descriptions supported in the text, usually concepts, theories, methods, etc." f"2. Output format: (Knowledge point entity 1, Knowledge point entity 2,...)." f"3. Output according to the output format without outputting additional explanations." ) (5)Extraction of relationships between knowledge point entities Purpose: To identify the relationships (such as "contains", "predecessor", "successor", "parallel") between knowledge point entities, providing rich semantic connections for knowledge graph construction; Method: Design instructions for extracting relationships between knowledge point entities, clearly require the model to extract the relationships between knowledge point entities, and specify the relationship types and output formats. At the same time, emphasize that only the previously extracted knowledge point entities and the specified relationship types can be used to ensure the accuracy and standardization of the extraction results. Call the Qwen2.5 model to extract the relationships between knowledge point entities according to the requirements of the instructions for extracting relationships between knowledge point entities, and generate relationship triples based on the extraction results of the relationships between knowledge point entities; For example, instruction = ( f"According to the following text and knowledge point entities, extract the relationships (contains, predecessor, successor, parallel) between these knowledge point entities," f"The relationship output format is a triple: (Knowledge point entity - relationship - Knowledge point entity), for example: (Sequential structure - contains - List), (Function - contains - Parameter)..." f"Note: 1. Both knowledge point entities corresponding to the extracted triple should use the input knowledge point entities. Do not use entities that are not listed. Do not. If there is no relationship, do not output." f"2. Do not extract relationship types that are not listed." f"3. Output in Chinese." f"4. Output according to the output format without outputting additional explanations." ) (6)Extraction of knowledge point entity attributes Purpose: To extract the attribute information of each knowledge point entity from the text, such as definition, description, function, etc., providing detailed attribute descriptions for the nodes in the knowledge graph and enriching the semantic information of the knowledge graph.

[0018] Method: Design instructions for extracting the attributes of knowledge point entities, clearly requiring the model to extract the attributes of knowledge point entities, specifying the attribute types and output formats, and emphasizing that the attribute content must be directly sourced from the original text, precluding self-generation to ensure the accuracy and reliability of the extraction results. Invoke the Qwen2.5 model to extract the attributes of knowledge point entities according to the instructions for extracting the attributes of knowledge point entities, and generate attribute triples based on the extraction results of the attributes of knowledge point entities. Among them, the instructions for extracting the attributes of knowledge point entities specify the attribute types, constraints, and output formats.

[0019] For example, instruction = ( f"Extract the attributes of these entities (attribute types: definition, description, characteristics, properties, classification, principle, function, steps, case (practical application), example (simple example)) based on the following text and knowledge point entities." f"Output format: (knowledge point entity - attribute type - attribute), for example: (feature data type - definition - Feature data type refers to composite data types in Python other than basic data types, including lists, tuples, dictionaries, and sets, etc.)." f"Note: 1. The knowledge point entity part of the triple must use the input knowledge point entity, and do not extract the attributes of other entities." f"2. Output in Chinese." f"3. Output according to the output format without additional explanations." f"4. The extracted attributes must be sourced from the original text and not self-generated. If the entity does not have attribute content of the corresponding attribute type in the text, do not extract it." f"5. If the knowledge point has attributes, output them all." f"5. If the input knowledge point has no attributes, do not list them." ) The following specific examples are proposed in combination with the above embodiments. It can be understood that the following specific examples only exemplarily elaborate on the specific implementation of the above embodiments and do not limit the technical solutions of the above embodiments.

[0020] Select the input text as "Python Programming Language (Second Edition)", and compare the output triple results of four large models, DeepSeek, Kimi, Tongyi Qianwen, and Wenxin Yiyan, with the fine-tuned Qwen 2.5 large model. Set the prompt as "Based on the text in Chapter 3, generate triples with an inclusion relationship between the knowledge point entities in the table of contents", and input the text data of Chapter 3 respectively. Set TP as the triples correctly output by the large model (true positives), FP as the values output by the model but not in the standard triples (false positives), FN as the values not output by the model but in the standard triples (false negatives), and TN as the values not in the standard triples and not output by the model either (true negatives). As Figure 2 shown, the output confusion matrices of the five models are obtained by establishing a confusion matrix. Calculate the precision (P), recall (R), and harmonic mean (F1) of the five models as evaluation indicators according to the confusion matrix. The calculation formulas are as follows:

[0021]

[0022]

[0023] Conduct a statistical analysis on the evaluation indicators of the five models. The experimental results are shown in Table 1.

[0024] Table 1 Statistical Results of Evaluation Indicators of Five Models

[0025] As can be seen from Table 1, the recall rate of the fine-tuned Qwen 2.5 model reaches 91.25%, and the precision rate of 86.96% is significantly higher than that of other models. The F1 value of 89.02% shows the best performance, verifying the accuracy and effectiveness of the fine-tuned Qwen 2.5 model in the automatic generation of textbook knowledge graphs.

[0026] Further calculate the false positive rate (FPR) and true positive rate (TPR) of the five models according to the confusion matrix. The calculation formulas are as follows:

[0027]

[0028] Figure 3 To obtain the ROC curves of the five models based on the false positive rate (FPR) and true positive rate (TPR) of the five models, from Figure 3 it can be seen that the ROC curve of the fine-tuned Qwen 2.5 model is at the top, with the highest TPR and the lowest FPR, showing the best performance at all thresholds and the best comprehensive performance.

[0029] So far in the embodiments of the present invention, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A method for automatically generating hierarchical textbook knowledge graphs based on a large language model, characterized in that: include: Step 1: Use the general large model to extract knowledge points from the sample textbook text and conduct manual review, and use the extracted knowledge points as the annotation information of the sample textbook text to obtain training data; Step 2: Select the Qwen2.5 model as the textbook knowledge graph extraction model, and use the LLaMA-Factory tool combined with the training data to customize and fine-tune the Qwen2.5 model; Step three: Design a hierarchical extraction process and extraction instructions, and extract the textbook knowledge graph of the textbook text to be processed based on the hierarchical extraction process and extraction instructions and the Qwen2.5 model, wherein the hierarchical extraction process includes directory structure extraction, directory hierarchical relationship extraction, text segmentation and directory correspondence, knowledge point entity extraction in the directory, knowledge point entity relationship extraction, knowledge point entity attribute extraction.

2. The method for automatically generating a hierarchical teaching material knowledge graph based on a large language model according to claim 1 is characterized in that: In the step 2, the LLaMA-Factory tool is used to customize and fine-tune the Qwen2.5 model in combination with the training data, including: The LoRA fine-tuning method is adopted, lora_target is set to all, SFT is used for supervised fine-tuning, the template is glm4, the maximum truncation length of the textbook text is set to 4000, 16 working processes are set for parallel processing in the data preprocessing part, the learning rate is set to 1.0e-4 in the training part, the training batch size of each device is set to 1, the gradient of 8 batches is accumulated before updating, the cosine learning rate scheduler is used to adjust the learning rate, and the learning rate warm-up ratio is 0.

1. In the evaluation part, 10% of the data is divided from the training data set as the validation set, the batch size of each device during evaluation is set to 1, and an evaluation is set every 500 training steps. The evaluation indicators are calculated using the validation set, and the output directory, logging interval, and checkpoint save steps are set to monitor the training status in real time and save key training results.

3. The method for automatically generating a hierarchical teaching material knowledge graph based on a large language model according to claim 1 is characterized in that: In the step 3, the textbook knowledge graph of the textbook text to be processed is extracted based on the hierarchical extraction process and extraction instructions and the Qwen2.5 model, including: Directory structure extraction: Take the textbook text to be processed as input, use the Qwen2.5 model to extract the directory, use list derivation to extract all directory knowledge point entities and store them in a list; Extraction of directory hierarchical relationships: Filter the directory list to remove empty directories, identify all chapter directories according to the format of the chapter directory, identify the subdirectories contained in the chapter directory according to the common features between the chapter directory and other directories, traverse all remaining directories, and add the hierarchical relationships between all directories to the relationship set; Text segmentation corresponds to catalog: divide the textbook text according to the catalog level, and assign each minimum level catalog to the corresponding text segment; Knowledge point entity extraction in the directory: design text knowledge point entity extraction instructions, and call the Qwen2.5 model to extract knowledge point entities in the text fragment according to the requirements of the text knowledge point entity extraction instructions, wherein the text knowledge point entity extraction instructions specify the knowledge point type, constraints and output format of the text knowledge point entity extraction; Extraction of relations between knowledge point entities: Design an instruction for extracting relations between knowledge point entities, and call the Qwen2.5 model to extract relations between knowledge point entities according to the requirements of the instruction for extracting relations between knowledge point entities, and generate relation triples according to the extraction results of relations between knowledge point entities, wherein the instruction for extracting relations between knowledge point entities specifies the relation type, constraints and output format; Knowledge point entity attribute extraction: design knowledge point entity attribute extraction instructions, call Qwen2.5 model to extract knowledge point entity attributes according to the knowledge point entity attribute instructions, and generate attribute triples according to the knowledge point entity attribute extraction results, wherein the knowledge point entity attribute extraction instructions specify attribute types, constraints and output formats.

Images