Method and system for realizing incremental construction of knowledge graph based on Md5-hash of text block

Abstract

The invention discloses a method and system for realizing incremental construction of a knowledge graph based on Md5-hash of a text block, and the method comprises the following steps: S1, constructing an initial knowledge graph based on a target document, generating community metadata, and storing the community metadata in a vector database; s2, performing block processing on a target document, performing MD5 hash calculation on each text block, generating a unique hash value, and storing the unique hash value in a historical hash library; s3, analyzing the updated document, extracting a set of all blocks of the current document according to a specified block size, recalculating a hash value of each block, comparing the hash value with a historical hash library, filtering out text blocks of which entity relationships are extracted and a graph is generated, and extracting an updated text block set; and S4, performing incremental updating of the knowledge graph according to the changed block type, wherein the incremental updating comprises timestamp marking of entity nodes, dynamic maintenance of relation links and generation of version submission records. According to the method, the maintenance cost can be reduced, the graph updating efficiency is improved, and the knowledge graph iteration speed is increased.

Description

Technical Field

[0001] The present invention relates to the field of text extraction knowledge graphs, and in particular to a method and system for incrementally constructing a knowledge graph based on Md5-hash of text blocks. Background Art

[0002] A knowledge graph is a knowledge base that represents entities (concepts, people, and things) in the objective world and the relationships between them in a graphical form. Knowledge graphs have garnered widespread attention and applied research, and have now become a foundational technology for intelligent services such as semantic search, intelligent question-answering, and decision support. Knowledge graphs have become a crucial technology in the field of artificial intelligence (AI), helping people better understand and utilize data, thereby improving the performance of AI systems.

[0003] However, incremental updates of knowledge graphs have always been a difficult problem in the industry due to the complex structure of knowledge graphs and the optimization of incremental update algorithms. Summary of the Invention

[0004] To solve the existing problems, the present invention provides a method and system for incrementally constructing a knowledge graph based on Md5-hash of text blocks. The specific solution is as follows:

[0005] A method for incrementally constructing a knowledge graph based on Md5-hash of text blocks, comprising the following steps:

[0006] S1. Build an initial knowledge graph based on the target document, generate community metadata and store it in a vector database;

[0007] S2. Divide the extracted documents into blocks and perform MD5 hash calculation on each text block in combination with the ID of the graph knowledge base to generate a unique hash value and store it in the historical hash library;

[0008] S3. Parse the updated document, extract all block sets of the current document according to the specified block size, recalculate the hash value of each block according to the method in step S2, compare it with the historical hash library, filter out the text blocks for which entity relationships have been extracted and graphs have been generated, and extract the updated text block set;

[0009] S4. Perform incremental updates of the knowledge graph based on the change block type, including timestamp marking of entity nodes, dynamic maintenance of relationship links, and generation of version submission records.

[0010] Preferably, the community metadata in step S1 includes a calling interface find_communities for the community discovery algorithm and a community details query interface get_community_detail.

[0011] Preferably, updating the document in step S3 includes adding, modifying, and deleting the content of the document, and does not support online updating of the document.

[0012] Preferably, in step S3, the newly added, deleted, and modified text blocks are classified and processed as follows:

[0013] For newly added text blocks: calculate the hash value and store it in the historical hash library, triggering the knowledge extraction module to generate new entities and relationships;

[0014] For deleted text blocks: remove the hash value record, mark the corresponding entity as invalid, and terminate the timeliness of the association relationship;

[0015] For modified text blocks: match the old hash value in the historical hash library, update it to the current hash value, and trigger difference analysis to update the corresponding entity attributes or relationships.

[0016] Preferably, step S4 is specifically:

[0017] For newly added instance entities: create a corresponding entity in the knowledge graph, add attribute information, and set the BEGIN_TIMESTAMP attribute value of the created entity to the current timestamp; establish a relationship between this entity and other entities, and set the BEGIN_TIMESTAMP attribute value of the established relationship to the current timestamp; and establish an add relationship between the current Commit entity and the newly created entity;

[0018] For deleting instance entities: match the code entity to be deleted in the knowledge graph, query the instance entity's incoming edge and the first entity associated with the incoming edge, as well as the outgoing edge and the tail entity associated with the outgoing edge, and perform a recursive query on the last entity; then set the END_TIMESTAMP attribute value of all queried entities and relationships to the current timestamp, and create a deletion relationship between the current Commit entity and the related code entities;

[0019] For modifying instance entities: if the modification operation only involves modifying entity attributes, the attribute values of the related entities are modified; if the modification operation involves adding a new relationship between entities, a relationship between the corresponding head and tail entities is created and the relationship attribute BEGIN_TIMESTAMP is set; if the modification operation involves deleting an entity, the END_TIMESTAMP attribute of the deleted entity is set to the current timestamp, and a modification relationship is created between the current Commit entity and the updated entity.

[0020] A system for incrementally updating knowledge graph community summaries and subgraphs based on MD5-Hash, comprising:

[0021] Incremental document extraction module: This module extracts text blocks from the target document according to the user-specified segmentation method and compares them with the historical text blocks in the historical hash library to extract each updated text block. Each updated text block is then resent to the large model to obtain clear entities, attributes, and relationships. If no corresponding hash value for the text block is found, the current text block is saved to the historical hash library.

[0022] Knowledge extraction module: Uses the incremental document extractor to segment and parse the document, extracting the set of text blocks modified at the time. When extracting knowledge from the text blocks based on the LLM, it also extracts the graph structure and summarizes the elements to reduce the number of LLM calls;

[0023] Graph incremental update module: The function of this module is to connect to the corresponding knowledge graph, identify existing entities or relationships based on the corresponding updated entities or relationships, create or update the relationships between nodes and edges based on the updated entities and write them into the graph file, then verify the correctness and completeness of the entity relationships of the entire graph, and finally regenerate the knowledge graph file and re-write it into the graph database to complete the incremental update of the existing knowledge graph.

[0024] Preferably, the specific process implemented by the graph incremental update module includes:

[0025] S1', Identify entities: It is necessary to identify entities from new data sources, which involves named entity recognition (NER) technology, which is used to extract entities with specific meanings from text;

[0026] S2', create or update nodes: add the newly identified entity as a new node to the knowledge graph, or if the entity already exists in the knowledge graph, update its related node information; node updates include adding new attributes, relationships or modifying existing attribute values;

[0027] S3', Establishing Edge Relationships: In a knowledge graph, relationships between nodes are represented by edges. Based on the relationship information in the new data, edge relationships are established between new or existing nodes. This involves determining the type and direction between nodes and adding relevant attributes to the edges.

[0028] S4', edge weight adjustment: adjust the weight of existing edge relationships based on the information in the new data;

[0029] S5', Verification and Validation: After completing the update of nodes and edges, verification and validation steps are required to ensure the accuracy and consistency of the update, including checking whether the newly added entities and relationships comply with the rules and standards of the knowledge graph, and checking whether the update has caused contradictions or conflicts in the knowledge graph;

[0030] S6', storage and indexing: Store the updated knowledge graph in the corresponding database or knowledge base, and establish an index to improve query efficiency.

[0031] The present invention also discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is run, any of the above methods is executed.

[0032] The present invention also discloses a computer system, including a processor and a storage medium, wherein a computer program is stored on the storage medium, and the processor reads and runs the computer program from the storage medium to execute any of the methods described above.

[0033] The beneficial effects of the present invention are:

[0034] By modifying the data source content as new input, users can incrementally update the knowledge graph of existing documents, and record and display the newly added entity relationships after the update in the graph. Using the incremental update method, only part of the data in the knowledge graph needs to be updated without rebuilding the entire graph. This can reduce maintenance costs, improve the efficiency of graph updates, and speed up the iteration speed of the knowledge graph. It greatly reduces the redundant overhead caused by the full update of the graph, ensures the timeliness and consistency of the graph content, and enhances the scalability of the system. These positive effects help the knowledge graph better support various knowledge services and applications, and provide users with a better experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0036] Figure 1 Schematic diagram of the key steps in building a knowledge graph;

[0037] Figure 2 Perform Md5Hash on the knowledge base ID and store the schematic diagram;

[0038] Figure 3 Update the diagram for the graph node;

[0039] Figure 4 Screenshot of the code for calling the large model to generate the knowledge graph and obtain the entity, attribute, and relationship results;

[0040] Figure 5 This is a screenshot of the core code for updating entity relationships based on the neo4j graph database. DETAILED DESCRIPTION

[0041] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0042] In order to solve the existing problems, the present invention provides a method for incrementally constructing a knowledge graph based on the md5-hash of a text block, a method for incrementally constructing a knowledge graph based on the md5-hash of a text block as a key, filtering duplicate text blocks through a specific filter, and extracting optimal triples or entities based on the extracted updated text block set and using a general language large model to achieve efficient and accurate addition of new knowledge. By constructing documents + blocks to ensure the uniqueness of the extracted text blocks, the introduction of duplicate information in the knowledge graph is avoided. It allows the model to automatically update when new data arrives without the need to train from scratch, and only needs to process new or modified text blocks. It contributes to the continuous evolution and expansion of the knowledge graph, thereby maintaining the consistency and accuracy of the data.

[0043] Specifically, if Figure 1 A method for incrementally constructing a knowledge graph based on Md5-hash of text blocks includes the following steps:

[0044] S1. Build an initial knowledge graph based on the target document, generate community metadata, and store it in a vector database. The community metadata includes the call interface find_communities for the community discovery algorithm and the community details query interface get_community_detail.

[0045] S2. Divide the extracted documents into blocks and perform MD5 hash calculation on each text block in combination with the ID of the graph knowledge base to generate a unique hash value and store it in the historical hash library. Figure 2 shown.

[0046] S3. Parse the updated document and extract all the block sets of the current document according to the specified block size. Recalculate the hash value of each block according to the method in step S2 and compare it with the historical hash library. Filter out the text blocks for which entity relationships have been extracted and the graph has been generated, and extract the updated text block set. Document updates include adding, modifying, and deleting document content, and online document updates are not supported. Newly added, deleted, and modified text blocks are classified and processed as follows:

[0047] For newly added text blocks: the hash value is calculated and stored in the historical hash library, triggering the knowledge extraction module to generate new entities and relationships; that is, static analysis is performed on the local text block to obtain the entity attributes and association relationships in the knowledge graph corresponding to the newly added text block;

[0048] For deleted text blocks: remove the hash value record, mark the corresponding entity as invalid, and terminate the timeliness of the association relationship; that is, match the deleted text block to the entity in the knowledge graph based on the absolute path and element name of the deleted text block in the document;

[0049] For modified text blocks: the old hash value in the historical hash library is matched, updated to the current hash value, and a difference analysis is triggered to update the corresponding entity attributes or relationships. This means that the modified text block is first matched to the entity in the knowledge graph. Then, a static analysis is performed on the document block containing the text block. The parsed result is compared with the current knowledge graph for a difference analysis to determine the entity attributes and relationships that need to be updated in the knowledge graph corresponding to the code element.

[0050] S4. Perform incremental updates to the knowledge graph based on the change block type, including timestamp marking of entity nodes, dynamic maintenance of relationship links, and generation of version submission records. Specifically:

[0051] For newly added instance entities: create a corresponding entity in the knowledge graph, add attribute information, and set the BEGIN_TIMESTAMP attribute value of the created entity to the current timestamp; establish a relationship between this entity and other entities, and set the BEGIN_TIMESTAMP attribute value of the established relationship to the current timestamp; and establish an add relationship between the current Commit entity and the newly created entity;

[0052] For deleting instance entities: match the code entity to be deleted in the knowledge graph, query the instance entity's incoming edge and the first entity associated with the incoming edge, as well as the outgoing edge and the tail entity associated with the outgoing edge, and perform a recursive query on the last entity; then set the END_TIMESTAMP attribute value of all queried entities and relationships to the current timestamp, and create a deletion relationship between the current Commit entity and the related code entities;

[0053] For modifying instance entities: if the modification operation only involves modifying entity attributes, the attribute values of the related entities are modified; if the modification operation involves adding a new relationship between entities, a relationship between the corresponding head and tail entities is created and the relationship attribute BEGIN_TIMESTAMP is set; if the modification operation involves deleting an entity, the END_TIMESTAMP attribute of the deleted entity is set to the current timestamp, and a modification relationship is created between the current Commit entity and the updated entity.

[0054] The present invention also discloses a system for incrementally constructing a knowledge graph based on Md5-hash of text blocks, comprising:

[0055] Incremental document extraction module: This module extracts text blocks from the target document according to the user-specified segmentation method and compares them with the historical text blocks in the historical hash library to extract each updated text block. Each updated text block is resent to the large model to obtain clear entities, attributes and relationships. If the hash value of the corresponding text block is not found, the current text block is saved in the historical hash library. Figure 3 shown.

[0056] Knowledge extraction module: Use the incremental document extractor to divide and parse the document, extract the set of text blocks modified at that time, and extract the knowledge of the text blocks based on LLM. It will also extract the graph structure and summarize the elements to reduce the number of LLM calls. However, this may be a challenge to the LLM capability. In addition, the number of associated text blocks also needs to be controlled by configuration parameters to avoid overloading the context window. Call the large model to generate the knowledge graph method and obtain the entity, attribute and relationship results, such as Figure 4 shown.

[0057] Graph incremental update module: The function of this module is to connect to the corresponding knowledge graph, identify existing entities or relationships based on the corresponding updated entities or relationships, create or update the relationships between nodes and edges based on the updated entities and write them into the graph file, then verify the correctness and completeness of the entity relationships of the entire graph, and finally regenerate the knowledge graph file and re-write it into the graph database to complete the incremental update of the existing knowledge graph.

[0058] The specific process of implementing the graph incremental update module includes:

[0059] S1', Identify entities: It is necessary to identify entities from new data sources, which involves named entity recognition (NER) technology, which is used to extract entities with specific meanings from text;

[0060] S2', create or update nodes: add the newly identified entity as a new node to the knowledge graph, or if the entity already exists in the knowledge graph, update its related node information; node updates include adding new attributes, relationships or modifying existing attribute values;

[0061] S3', Establishing Edge Relationships: In a knowledge graph, relationships between nodes are represented by edges. Based on the relationship information in the new data, edge relationships are established between new or existing nodes. This involves determining the type and direction between nodes and adding relevant attributes to the edges.

[0062] S4', edge weight adjustment: adjust the weight of existing edge relationships based on the information in the new data;

[0063] S5', Verification and Validation: After completing the update of nodes and edges, verification and validation steps are required to ensure the accuracy and consistency of the update, including checking whether the newly added entities and relationships comply with the rules and standards of the knowledge graph, and checking whether the update has caused contradictions or conflicts in the knowledge graph;

[0064] S6', storage and indexing: store the updated knowledge graph in the corresponding database or knowledge base, and create an index to improve query efficiency. Figure 5 The core code for updating entity relationships based on the neo4j graph database.

[0065] By modifying the data source content as new input, users can incrementally update the knowledge graph of existing documents, and record and display the newly added entity relationships after the update in the graph. Using the incremental update method, only part of the data in the knowledge graph needs to be updated without rebuilding the entire graph. This can reduce maintenance costs, improve the efficiency of graph updates, and speed up the iteration speed of the knowledge graph. It greatly reduces the redundant overhead caused by the full update of the graph, ensures the timeliness and consistency of the graph content, and enhances the scalability of the system. These positive effects help the knowledge graph better support various knowledge services and applications, and provide users with a better experience.

[0066] The present invention also discloses a computer-readable storage medium and a computer system. The computer-readable storage medium stores a computer program, which, when executed, performs any of the methods described above. A computer system includes a processor and a storage medium, wherein the storage medium stores the computer program, and the processor reads and executes the computer program from the storage medium to perform any of the methods described above.

[0067] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of the two. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. A skilled person may implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as resulting in a departure from the scope of the present invention.

[0068] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read and write information from / to the storage medium. In an alternative, the storage medium may be integrated into the processor. The processor and storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and storage medium may reside in a user terminal as discrete components.

[0069] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0070] Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for incrementally constructing a knowledge graph based on Md5-hash of text blocks, characterized in that: The following steps are involved: S1. Build an initial knowledge graph based on the target document, generate community metadata and store it in a vector database; S2. Divide the extracted documents into blocks and perform MD5 hash calculation on each text block in combination with the ID of the graph knowledge base to generate a unique hash value and store it in the historical hash library; S3. Parse the updated document, extract all block sets of the current document according to the specified block size, recalculate the hash value of each block according to the method in step S2, compare it with the historical hash library, filter out the text blocks for which entity relationships have been extracted and graphs have been generated, and extract the updated text block set; S4. Perform incremental updates of the knowledge graph based on the change block type, including timestamp marking of entity nodes, dynamic maintenance of relationship links, and generation of version submission records.

2. The method according to claim 1, wherein: The community metadata in step S1 includes the call interface find_communities of the community discovery algorithm and the community details query interface get_community_detail.

3. The method according to claim 1, wherein: The updating of the document in step S3 includes adding, modifying, and deleting the content of the document, and does not support online updating of the document.

4. The method according to claim 3, characterized in that In step S3, the newly added, deleted, and modified text blocks are classified and processed, specifically: For newly added text blocks: calculate the hash value and store it in the historical hash library, triggering the knowledge extraction module to generate new entities and relationships; For deleted text blocks: remove the hash value record, mark the corresponding entity as invalid, and terminate the timeliness of the association relationship; For modified text blocks: match the old hash value in the historical hash library, update it to the current hash value, and trigger difference analysis to update the corresponding entity attributes or relationships.

5. The method according to claim 2, characterized in that Step S4 is specifically as follows: For newly added instance entities: create a corresponding entity in the knowledge graph, add attribute information, and set the BEGIN_TIMESTAMP attribute value of the created entity to the current timestamp; establish a relationship between this entity and other entities, and set the BEGIN_TIMESTAMP attribute value of the established relationship to the current timestamp; And establish an add relationship between the current Commit entity and the newly created entity; For deleting instance entities: match the code entity to be deleted in the knowledge graph, query the instance entity's incoming edge and the first entity associated with the incoming edge, as well as the outgoing edge and the tail entity associated with the outgoing edge, and perform a recursive query on the last entity; then set the END_TIMESTAMP attribute value of all queried entities and relationships to the current timestamp, and create a deletion relationship between the current Commit entity and the related code entities; For modifying instance entities: if the modification operation only involves modifying entity attributes, the attribute values of the related entities are modified; if the modification operation involves adding a new relationship between entities, a relationship between the corresponding head and tail entities is created and the relationship attribute BEGIN_TIMESTAMP is set; if the modification operation involves deleting an entity, the END_TIMESTAMP attribute of the deleted entity is set to the current timestamp, and a modification relationship is created between the current Commit entity and the updated entity.

6. A system based on the method according to any one of claims 1 to 5, characterized in that: include: Incremental document extraction module: This module extracts text blocks from the target document according to the user-specified segmentation method and compares them with the historical text blocks in the historical hash library to extract each updated text block. Each updated text block is then resent to the main model to obtain clear entities, attributes, and relationships. If the hash value of the corresponding text block is not found, the current text block is saved in the historical hash library; Knowledge extraction module: Uses the incremental document extractor to segment and parse the document, extracting the set of text blocks modified at the time. When extracting knowledge from the text blocks based on the LLM, it also extracts the graph structure and summarizes the elements to reduce the number of LLM calls; Graph incremental update module: The function of this module is to connect to the corresponding knowledge graph, identify existing entities or relationships based on the corresponding updated entities or relationships, create or update the relationships between nodes and edges based on the updated entities and write them into the graph file, then verify the correctness and completeness of the entity relationships of the entire graph, and finally regenerate the knowledge graph file and re-write it into the graph database to complete the incremental update of the existing knowledge graph.

7. The system according to claim 6, characterized in that The specific process of implementing the graph incremental update module includes: S1', Identify entities: It is necessary to identify entities from new data sources, which involves named entity recognition (NER) technology, which is used to extract entities with specific meanings from text; S2', create or update nodes: add the newly identified entity as a new node to the knowledge graph, or if the entity already exists in the knowledge graph, update its related node information; node updates include adding new attributes, relationships or modifying existing attribute values; S3', Establishing Edge Relationships: In a knowledge graph, relationships between nodes are represented by edges. Based on the relationship information in the new data, edge relationships are established between new or existing nodes. This involves determining the type and direction between nodes and adding relevant attributes to the edges. S4', edge weight adjustment: adjust the weight of existing edge relationships based on the information in the new data; S5', Verification and Validation: After completing the update of nodes and edges, verification and validation steps are required to ensure the accuracy and consistency of the update, including checking whether the newly added entities and relationships comply with the rules and standards of the knowledge graph, and checking whether the update has caused contradictions or conflicts in the knowledge graph; S6', storage and indexing: Store the updated knowledge graph in the corresponding database or knowledge base, and establish an index to improve query efficiency.

8. A computer-readable storage medium, characterized in that: The medium stores a computer program, and after the computer program is run, the method according to any one of claims 1 to 5 is executed.

9. A computer system, characterized in that: The method comprises a processor and a storage medium, wherein the storage medium stores a computer program, and the processor reads and runs the computer program from the storage medium to execute the method according to any one of claims 1 to 5.

Images