A dynamic data updating method based on a knowledge graph

Abstract

The application relates to the technical field of data updating, in particular to a dynamic data updating method based on a knowledge graph, which comprises the following steps: S1, receiving reported data, and performing preliminary classification on each data item; S2, assigning a unique identifier to each data item, and recording state mapping to a knowledge graph node; S3, storing the data in a temporary cache area and using a hash table for management; S4, grouping and managing the data in the cache; S5, performing batch processing of the data according to conditions; and S6, performing data mapping and incremental updating operations. Through the above steps, the application not only guarantees the accuracy of the data and the consistency of the knowledge graph, but also reduces the processing time and resource consumption. In addition, the steps of data verification, deletion of outdated data and regular backup further improve the reliability and security of the data. The application greatly improves the efficiency of processing dynamic data and ensures that the knowledge graph always maintains the latest state.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of data updating, and particularly relates to a dynamic data updating method based on a knowledge graph. BACKGROUND

[0002] The knowledge graph has important application value in data visualization and is a large semantic network connected by entities through relations, is composed of nodes and edges, wherein the nodes represent entities, and the edges represent the relations between the entities, and can effectively solve real problems with complex relations.

[0003] The traditional knowledge graph has obvious defects in data updating, mainly due to the lack of effective data classification, unique identifier allocation and efficient cache management mechanism, resulting in inaccuracy and inefficiency in the data integration and updating process, and due to the lack of automatic and intelligent data mapping and updating strategies, manual operation is not only time-consuming but also prone to errors, affecting the quality and speed of data updating, further leading to low efficiency of knowledge graph updating.

[0004] Therefore, the application provides a dynamic data updating method based on a knowledge graph. SUMMARY

[0005] The application aims to provide a dynamic data updating method capable of guaranteeing data accuracy and knowledge graph consistency, reducing processing time and resource consumption and improving dynamic data processing efficiency.

[0006] To solve the above technical problems, the application adopts the following technical scheme:

[0007] A dynamic data updating method based on a knowledge graph comprises the following steps:

[0008] S1, receiving reported data, and performing preliminary classification on each data item;

[0009] S2, allocating a unique identifier to each data item, and recording the state and associated knowledge graph node information of each data item;

[0010] S3, storing the received data in a temporary cache area, and using a hash table as a data structure for fast access and data arrangement;

[0011] S4, grouping and managing the data in the temporary cache area;

[0012] S5, setting data batch processing conditions, and triggering batch processing of the data when the conditions are met, and extracting the corresponding data set from the cache for next step processing;

[0013] S6, mapping the extracted data to the corresponding nodes of the knowledge graph, performing incremental updating operation on the data marked as new data, and updating the related part of the knowledge graph.

[0014] Preferably, a unique timestamp identifier is assigned to each data item in S2, ensuring the uniqueness and traceability of each data item.

[0015] Preferably, the temporary buffer area in S3 is provided with a capacity limit, and when the preset capacity is reached, the batch processing operation in step S5 is automatically triggered.

[0016] Preferably, during the temporary buffer process in S4, the data is grouped according to its age and update status, and further classified according to the type label of the data item, optimizing the efficiency of data management.

[0017] Preferably, the data batch processing conditions in S5 include time interval and data volume, and the data batch processing conditions are dynamically regulated by the system detected data processing demand.

[0018] Preferably, the data mapping in S6 includes node update and creation and modification of relationships in the knowledge graph.

[0019] Preferably, the incremental update operation in S6 includes a deletion operation for removing outdated data in the knowledge graph.

[0020] Preferably, the data mapping and incremental update operation in S6 includes a verification process to ensure the consistency and accuracy of the knowledge graph after data update.

[0021] Preferably, after completing the data update, the knowledge graph is backed up to prevent data loss and damage, and to ensure the possibility of data recovery.

[0022] The above technical solution provides a dynamic data updating method based on knowledge graph, which accurately classifies, assigns unique identifiers, efficiently manages buffers, and maps and updates data. This method not only ensures the accuracy of data and the consistency of the knowledge graph, but also reduces processing time and resource consumption. In addition, steps such as data verification, deletion of outdated data, and regular backup further improve the reliability and security of data, greatly improving the efficiency of processing dynamic data and ensuring that the knowledge graph always remains up-to-date. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 The method flowchart of the present application. DETAILED DESCRIPTION

[0024] In order to make the purpose and advantages of the present application more clear and explicit, the present application will be specifically described below in combination with examples. It should be understood that the following text is only used to describe one or several specific embodiments of the present application, and does not strictly limit the specific claimed protection range of the present application.

[0025] Example 1:

[0026] The technical scheme adopted by the present application is as shown in the description Figure 1 A dynamic data updating method based on a knowledge graph, comprising the following steps:

[0027] S1, receiving the reported data, classifying each data item, distinguishing "new data" and "update of existing data", the data source includes but is not limited to network interface, file upload or direct database writing, the received data exists in multiple formats such as JSON, XML or CSV file, once the data is received, the received data is parsed and formatted to identify the structure and content of the data item, each received data item will go through the preliminary classification process, and the data item will be classified according to the predefined rules and logic:

[0028] One kind is brand-new data (new data);

[0029] Another kind is data that already exists in the system and needs to be updated (update of existing data).

[0030] For each data item, check whether its unique identifier already exists in the database; if not, it is considered "new data"; if it exists, compare the timestamp to determine whether the data is an update of the existing record, to ensure accurate classification and minimize processing time and resource consumption;

[0031] S2, assign a unique identifier to each data item, record the status (new data and update data) and associated knowledge graph node information of each data item, assign a unique timestamp identifier to each data item to ensure its uniqueness and traceability, the timestamp identifier is generated by the current date and time, accurate to the nearest millisecond, to ensure the uniqueness and traceability of each data item, by using the built-in time generation function, a timestamp is automatically generated for each received data item, then, based on the previous classification results (i.e. "new data" and "update of existing data" in step S1), record the status of each data item, its status information is stored together with the data item to facilitate processing and decision-making in subsequent steps; finally, associate each data item with the corresponding knowledge graph node information and map it to a specific node in the knowledge graph, for this purpose, use the predefined mapping rules to determine which node in the knowledge graph is associated with the data item based on its attributes (keywords, classification labels), the mapping process ensures that data updates can be accurately reflected to the appropriate part of the knowledge graph.

[0032] S3, transmit the data processed in steps S1 and S2 to a temporary cache area, which is a storage space specially set up in the system memory for temporarily storing data to be processed, in order to efficiently manage and quickly access these data, the temporary cache area uses a hash table as its main data structure, the hash table uses the unique identifier (timestamp identifier) of the data item as the key (Key) to achieve fast retrieval and access of any data item, in addition, in order to prevent the temporary cache area from being overloaded and maintain system performance, the temporary cache area is provided with a capacity limit, which is set in advance according to system resources and processing capacity, when the amount of data stored in the temporary cache area reaches this preset capacity, the batch processing operation of step S5 is triggered, once the amount of data in the cache area reaches the threshold, the system will further process and update the data.

[0033] In the whole step S3, the temporary cache area and the capacity limit mechanism ensure the efficiency of data processing and the stable operation of the system, by using the hash table, the data can be quickly located and processed, and the automatically triggered batch processing mechanism effectively manages the data flow and system load, ensuring that the backlog of data is processed in time.

[0034] S4, the system groups data based on the new and old state and update state of data in the temporary cache area, new data refers to data that has not yet existed in the knowledge graph in step S1, and update data refers to data that needs to modify the data that already exists in the knowledge graph, this classification is based on the unique identifier and recorded state information assigned to each data item in step S2, and further classified according to the type label of the data item, each data item is assigned a specific type label in the source system or when received, the label describes the nature or classification of the data item, in the temporary cache area, the data is allocated to different subgroups using the type label, which uses a classification algorithm to group data items according to their type labels according to pre-set rules;

[0035] This grouping mechanism greatly optimizes the efficiency of data management, by classifying data into new data, update data, and further classified data according to type labels, data is more organized and processed and updated.

[0036] S5, trigger batch processing of data based on conditions set by time interval and data volume in cache, when the conditions are met, extract the corresponding data set from the cache for further processing;

[0037] In this step, the batch processing of data is triggered according to two main conditions: time interval and data volume in temporary cache;

[0038] By setting a threshold for time intervals, such as every certain number of hours or minutes, the system periodically checks the data in the temporary buffer area. Secondly, the system monitors the amount of data in the cache. When the amount of data reaches a set threshold, it also triggers batch processing. In addition to these two basic conditions, based on real-time monitoring of system performance indicators such as processing queue length, CPU usage, and memory usage, the system adjusts the triggering conditions dynamically according to the current data processing needs. When the system detects an increase in data processing needs, it reduces the time interval or adjusts the data volume threshold to trigger batch processing more frequently. Conversely, during periods of low load, it increases the time interval or data volume threshold to optimize resource usage.

[0039] When the triggering conditions for batch processing are met, the corresponding data set is extracted from the cache for further processing. All data items in the cache marked as new data or updated data are selected, and data subsets are grouped according to type tags.

[0040] S6, mapping the extracted data to the corresponding nodes of the knowledge graph, performing incremental update operations for data marked as new data, updating the relevant part of the knowledge graph,

[0041] From the data set extracted in step S5, select each data item. For each selected data item, use the information recorded in step S2 to determine its corresponding node in the knowledge graph, and match the features with the node in the knowledge graph. The matching process is based on pre-defined mapping rules, which associate specific types of data items to specific nodes in the knowledge graph through rules;

[0042] Features of pre-defined mapping rules:

[0043] Property matching: rules contain logic for identifying and matching key properties of data items, including keywords, classification tags, and identifiers, to determine the association of data items with which node in the knowledge graph;

[0044] Node positioning: mapping rules specify the positioning of specific nodes in the knowledge graph based on the properties of data items;

[0045] Relationship establishment: indication of establishing relationships between data items and knowledge graph nodes, especially when there are logical associations between data items;

[0046] Type conversion: ensure the format of data items is compatible with knowledge graph nodes through type conversion instructions, such as converting text dates to standard date formats;

[0047] Default operation: for data items that cannot be directly mapped, mapping rules provide default operations, such as assigning them to a general or pending classification node;

[0048] Then, incremental update operations are performed on those data items marked as new data, which means adding or modifying only the part of information that does not exist in the knowledge graph, instead of replacing or restructuring the entire node, including creating new nodes or adding new attributes and relationships in existing nodes, to ensure that the data of the knowledge graph remains up-to-date and complete.

[0049] Specifically, the data mapping in step S6 is not limited to node updates, but also includes the creation and modification of relationships in the knowledge graph. By creating and modifying relationships between nodes, the knowledge graph can more comprehensively reflect the diverse connections between entities, not only increasing the amount of information in the graph, but also improving its ability to provide insights and analysis depth.

[0050] Specifically, the incremental update operation in step S6 includes a deletion operation to remove outdated data in the knowledge graph. Over time, some information becomes inaccurate or irrelevant. By deleting this outdated data, the knowledge graph can maintain the currency and accuracy of its content, ensuring that the information accessed by users is up-to-date, and removing inaccurate or outdated information can improve the overall quality of the knowledge graph. Regularly cleaning outdated data helps to avoid information overload and makes the information in the knowledge graph more focused and relevant.

[0051] Specifically, the data mapping and updating operation in step S6 includes a verification process to ensure the consistency and accuracy of the knowledge graph after data update. Verification ensures that all new or modified data items are accurate and do not conflict with other parts of the knowledge graph. The verification process checks whether the update operation has destroyed the existing data structure and logical relationship in the knowledge graph, and by ensuring that all updates comply with predefined rules and standards, it helps to maintain data consistency.

[0052] Specifically, after completing the data update, the knowledge graph is backed up to prevent data loss or damage and ensure the possibility of data recovery.

[0053] Embodiment 2:

[0054] Enterprise market analysis knowledge graph update

[0055] In this embodiment, the enterprise uses the knowledge graph to analyze and predict market trends. The enterprise receives a large amount of data from different sources on a daily basis, including market reports, news sources, social media feeds, etc., in various formats such as JSON, XML, and CSV files.

[0056] S1, the system receives about 100GB of data from multiple network interfaces, file uploads and database writes every day, and performs preliminary classification on each data item through automated scripts, for example, a new market report is marked as "new data", while the latest financial report of an existing cooperative enterprise is marked as "update of existing data".

[0057] S2. For each data item, the system automatically generates a timestamp identifier. For example, if a new social media trend report is received on March 1, 2024 at 14:32:10, the timestamp assigned is 20240301143210. The data item is mapped to the knowledge graph, such as mapping a new market report to a specific industry node.

[0058] S3. All data items are stored in a temporary cache area with a capacity limit of 50GB. Once this limit is reached, batch processing operations are automatically triggered.

[0059] S4. Within the cache, data is grouped based on type labels. For example, all reports about "e-commerce" are grouped into one group, and all reports about "car manufacturing" are grouped into another group.

[0060] S5. The system is set to check the cache every 6 hours. During the check, if the amount of data in the cache exceeds 25GB or the set time interval is reached, the data will be extracted and processed.

[0061] S6. The system retrieves data from the cache and performs incremental updates. For example, a report on emerging markets is added to the "Emerging Markets" node of the knowledge graph. At the same time, the system deletes some outdated market analysis data to maintain the accuracy of the knowledge graph. All update operations undergo a verification process to ensure consistency and accuracy.

[0062] This embodiment enables continuous updates to its market analysis knowledge graph, maintaining the timeliness and accuracy of information, thereby better understanding market dynamics and predicting future trends.

[0063] In summary, this invention ensures data accuracy and knowledge graph consistency through precise preliminary classification, unique identifier allocation, efficient cache management, data mapping, and update strategies, while reducing processing time and resource consumption. Furthermore, steps such as data verification, deletion of outdated data, and regular backups further enhance data reliability and security. This invention significantly improves the efficiency of processing dynamic data and ensures that the knowledge graph is always up-to-date.

[0064] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. For those skilled in the art, after learning the contents described in the present invention, several equivalent changes and substitutions can be made without departing from the principle of the present invention. These equivalent changes and substitutions should also be considered to fall within the protection scope of the present invention.

Claims

1. A dynamic data update method based on knowledge graphs, characterized in that: S1. Receive the reported data and perform preliminary classification on each data item; S2. Assign a unique identifier to each data item and record the status of each data item and the associated knowledge graph node information; S3. Store the received data in a temporary cache area and use a hash table as the data structure for fast access and data organization. S4. In the temporary buffer area, group and manage the data; S5. Set data batch processing conditions. When the conditions are met, trigger the batch processing of data and retrieve the corresponding data set from the cache for the next step of processing. S6. Map the extracted data to the corresponding nodes in the knowledge graph. For data marked as new, perform an incremental update operation to update the relevant parts of the knowledge graph.

2. The dynamic data update method based on knowledge graphs according to claim 1, characterized in that: In S2, a unique timestamp identifier is assigned to each data item to ensure the uniqueness and traceability of each data item.

3. The dynamic data update method based on knowledge graphs according to claim 1, characterized in that: The temporary buffer area described in S3 has a capacity limit. When the preset capacity is reached, the batch processing operation in step S5 is automatically triggered.

4. The dynamic data update method based on knowledge graphs according to claim 1, characterized in that: During the S4 temporary caching process, data is grouped according to its age and update status, and further categorized according to the type label of the data item to optimize data management efficiency.

5. The dynamic data update method based on knowledge graphs according to claim 1, characterized in that: The data batch processing conditions described in S5 include time interval and data volume, and the data batch processing conditions are dynamically adjusted by the data processing requirements detected by the system.

6. The dynamic data update method based on knowledge graphs according to claim 1, characterized in that: The data mapping described in S6 includes node updates and the creation and modification of relationships in the knowledge graph.

7. The dynamic data update method based on knowledge graphs according to claim 1, characterized in that: The incremental update operation described in S6 includes a deletion operation, used to remove outdated data from the knowledge graph.

8. The dynamic data update method based on knowledge graphs according to claim 1, characterized in that: The data mapping and incremental update operations described in S6 include a verification process to ensure the consistency and accuracy of the knowledge graph after the data update.

9. The dynamic data update method based on knowledge graphs according to claim 1, characterized in that: After completing the data update, back up the knowledge graph to prevent data loss or damage and to ensure the possibility of data recovery.

Images