A Data Quality Improvement Method Based on Knowledge Graph

Abstract

This invention discloses a data quality improvement method based on knowledge graphs, including: preprocessing table creation statements in a database, followed by entity and relation extraction; automatically building a knowledge graph by extracting relationships between data warehouse processing flows in the database; mapping relationships between data processing and processing processes based on relation retrieval from the knowledge graph; and using the relationships in the knowledge graph to supplement data for anomaly detection of similar data features during the application of anomaly detection algorithms, thereby improving the anomaly detection effect of large datasets. This invention simply utilizes two different anomaly detection algorithm models to detect sample electricity consumption data. While the algorithms themselves have certain limitations, optimizing them by using knowledge graphs for group partitioning improves their accuracy to a certain extent. The knowledge graph effectively reflects the data processing structure, thus significantly improving the accuracy of anomaly detection.

Description

Technical Field

[0001] This invention relates to the field of big data processing technology, and in particular to a method for improving data quality based on knowledge graphs. Background Technology

[0002] In the application of big data technology for feature analysis and mining, machine learning methods still suffer from drawbacks such as poor interpretability, limited data sources, and low completeness of user pattern representation. Furthermore, as enterprise service quality continues to improve, user behavior analysis systems demand increasingly higher interpretability of their results. Existing big data analytics technologies struggle to overcome these shortcomings. The advent of the big data era has spurred rapid development in artificial intelligence (AI) technology, leading to increased demands for data processing and understanding. Knowledge graph applications are gaining traction, and as a knowledge representation of "big data + AI," knowledge graphs possess inherent advantages in terms of the completeness and interpretability of behavioral characterization, overcoming the shortcomings of existing methods. At the knowledge application level, knowledge graphs are applied to industry data, and even when dealing with concentrated data volumes, their results remain effective. The development of the knowledge graph field will continue to exhibit trends of specialization, openness, and intelligence. To better leverage the advantages of existing knowledge graphs in knowledge expression and resources, it is necessary to combine them with other technologies (information recommendation, event graphs, machine learning, deep learning, etc.).

[0003] In big data processing applications, anomaly detection is a typical application for improving data quality. However, due to the limitations of current big data technologies, systems struggle to trace the origins of outliers, and the accuracy of detection needs improvement. Analysis of domestic and international research reveals several problems with current anomaly detection methods. Supervised learning-based anomaly detection struggles to obtain sufficient anomaly data labels for training, and the inherently high number of missing values ​​in big data significantly impacts the results of supervised learning anomaly detection models. Unsupervised anomaly detection models, such as those based on clustering algorithms, require setting cluster centers or various parameters based on human experience. However, the selection of these parameters is crucial; improper selection can lead to local optima, resulting in outliers appearing close to normal data and going undetected. Summary of the Invention

[0004] The purpose of this section is to outline some aspects of embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0005] In view of the aforementioned existing problems, the present invention is proposed.

[0006] Therefore, the technical problem solved by this invention is that the accuracy of abnormal data detection is low in a big data environment.

[0007] To solve the above-mentioned technical problems, the present invention provides the following technical solution: including,

[0008] The table creation statements in the database are preprocessed before entity extraction and relation extraction are performed.

[0009] Automatically build a knowledge graph by extracting the relationships between data warehouse processing flows in the database;

[0010] Based on the relation retrieval of the knowledge graph, the data processing and processing procedures are mapped to related relationships.

[0011] In the process of applying anomaly detection algorithms, the relationships in the knowledge graph are used to provide data supplementation for anomaly detection of similar data features, thereby improving the anomaly detection effect of big data.

[0012] As a preferred embodiment of the knowledge graph-based data quality improvement method of the present invention, the preprocessing includes:

[0013] The table creation process of each table is extracted using regular expressions, and then analyzed sequentially.

[0014] At the same time, the regular expression is used to remove unnecessary comments and statements to avoid interference;

[0015] The regular expression rules are briefly described as the statements between "create table" and ";", which extract the generation process of a single intermediate table for subsequent analysis.

[0016] As a preferred embodiment of the knowledge graph-based data quality improvement method described in this invention, it further includes:

[0017] After obtaining the table creation statements for each table separately, the SQL statements are analyzed.

[0018] Based on the logical relationships found in the analysis results, the subordinate relationships between field entities and table entities, as well as the generation relationship between the original table and this table, are extracted.

[0019] As a preferred embodiment of the knowledge graph-based data quality improvement method described in this invention, the establishment of the knowledge graph includes:

[0020] Using graph structures to apply knowledge triples<key,value> Stored in the form of;

[0021] Entities are stored in the form of nodes;

[0022] The relationships between entities are stored in the form of edges between nodes, thus efficiently displaying the knowledge information of the knowledge graph.

[0023] As a preferred embodiment of the knowledge graph-based data quality improvement method described in this invention, the relationship retrieval based on the knowledge graph includes:

[0024] Based on the effectiveness of graph databases, Neo4j is used to store knowledge.

[0025] Neo4j is a NoSQL database, and the categories and attributes of Neo4j nodes and edges are stored separately.

[0026] Neo4j also includes full transaction support, powerful graph search capabilities, and a certain degree of horizontal scalability.

[0027] The full transaction support means that Neo4j requires each data modification action to be completed within a transaction, which helps to ensure data consistency.

[0028] The powerful graph search capability means that Neo4j has clients that support a range of popular languages, including Cypher, a language specifically designed for graph databases, making it easy for users to operate and search the Neo4j database.

[0029] The aforementioned horizontal scaling capability indicates that Neo4j can improve performance through read-write splitting via Read Replica.

[0030] As a preferred embodiment of the knowledge graph-based data quality improvement method described in this invention, the association mapping includes:

[0031] The node retrieval of the knowledge graph is used to simulate the data processing and manipulation process, and the logical relationships of data processing are mapped to the logical relationships between nodes.

[0032] As a preferred embodiment of the knowledge graph-based data quality improvement method of the present invention, the anomaly detection algorithm includes employing a method based on the Local Anomaly Factor (LOF), the specific algorithm definition of which is as follows:

[0033] For a given data point p, the k-distance k-distance(p) of p can determine the neighbor members of p in the dataset. The condition that k-distance(p) is equal to the Euclidean distance d(p, o) between data point p and data point o needs to satisfy two requirements.

[0034] As a preferred embodiment of the knowledge graph-based data quality improvement method described in this invention, the two requirements include:

[0035] The dataset contains at least k data points, excluding p. The condition d(p, o) <= d(p, o) is satisfied.

[0036] The dataset contains at most k-1 data points, excluding p. Satisfying d(p, o) <d(p,o)。

[0037] As a preferred embodiment of the knowledge graph-based data quality improvement method described in this invention, the calculation of the k-distance k-distance(p) of p includes,

[0038] Calculate the distance from p to all data points in the dataset;

[0039] Choose k distinct minimum distances;

[0040] The maximum value among the k minimum distances is the k-distance of p.

[0041] As a preferred embodiment of the knowledge graph-based data quality improvement method described in this invention, it further includes:

[0042] Let N be the set of all neighboring data points whose distance to p is less than or equal to the k-distance (k-distance(p)) to p. k-d (p);

[0043] The reachability distance from object p to object q is defined as: reach-dk(p, q) = max{kd(q), d(p, q)};

[0044] The reachability distance between data points p and q is represented as follows: if data points p and q are very close, the reachability distance is the k-distance of q; if data points p and q are very far apart, the reachability distance is their actual distance.

[0045] The beneficial effects of this invention are as follows: This invention utilizes two different anomaly detection algorithm models to detect sample electricity consumption data. The algorithms themselves have certain limitations. After optimizing the algorithms by using knowledge graphs for group segmentation, the accuracy of the algorithms is improved to a certain extent. This is the advantage of knowledge graph technology in anomaly detection algorithm models. Knowledge graphs can well reflect the data processing structure and effectively improve the accuracy of anomaly detection. Attached Figure Description

[0046] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0047] Figure 1 This is a schematic diagram of a power data knowledge graph storage scheme model framework for constructing a data quality improvement method based on knowledge graphs according to an embodiment of the present invention. Detailed Implementation

[0048] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0049] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0050] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0051] This invention is described in detail with reference to the schematic diagrams. When detailing the embodiments of this invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not adhering to the usual scale. Furthermore, the schematic diagrams are merely examples and should not be construed as limiting the scope of protection of this invention. In actual fabrication, the three-dimensional spatial dimensions of length, width, and depth should be included.

[0052] Furthermore, in the description of this invention, it should be noted that the terms "upper," "lower," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used solely for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. In addition, the terms "first," "second," or "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0053] Unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" in this invention should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; similarly, they can refer to mechanical connections, electrical connections, or direct connections, or indirect connections through an intermediate medium, or internal connections between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0054] Example 1

[0055] Using knowledge graphs can not only optimize the judgment of anomalies and improve the accuracy of outlier detection, but also reflect the logic of data storage and processing. It can also trace the source of the obtained outliers to find out the cause of the anomalies, saving unnecessary human and material costs. Knowledge graphs are of great significance for the optimization and future development of anomaly detection systems.

[0056] This embodiment uses two different anomaly detection algorithm models to detect sample electricity consumption data. The algorithm itself has certain limitations. After optimizing the algorithm by using knowledge graph for group segmentation, the accuracy of the algorithm is improved to a certain extent.

[0057] Reference Figure 1 This is the first embodiment of the present invention, which provides a data quality improvement method based on knowledge graphs, specifically including:

[0058] S1: Preprocess the table creation statements in the database, then perform entity extraction and relation extraction. It should be noted that the preprocessing includes:

[0059] The table creation process of each table is extracted using regular expressions, and then analyzed sequentially.

[0060] At the same time, use regular expressions to remove unnecessary comments and statements to avoid interference;

[0061] The regular expression rules are briefly described as the statements between "create table" and ";", which extract the generation process of a single intermediate table for subsequent analysis.

[0062] Specifically, it also includes:

[0063] After obtaining the table creation statements for each table separately, the SQL statements are analyzed.

[0064] Based on the logical relationships found in the analysis results, the subordinate relationships between field entities and table entities, as well as the generation relationship between the original table and this table, are extracted.

[0065] S2: Automated knowledge graph creation by extracting relationships between data warehouse processing flows from the database. It should be noted that knowledge graph creation includes:

[0066] Using graph structures to apply knowledge triples<key,value> Stored in the form of;

[0067] Entities are stored in the form of nodes;

[0068] The relationships between entities are stored in the form of edges between nodes, thus efficiently displaying the knowledge information of the knowledge graph.

[0069] Preferably, in this embodiment, it should be noted that after the schema layer of the knowledge graph is constructed and the entity triples are extracted, the knowledge triples need to be injected into the data layer to complete the knowledge storage, so as to perform business scenarios such as knowledge querying and subsequent abnormal data tracing. Storing the knowledge triples of the knowledge graph in the form of a graph database is a relatively convenient and efficient method. Subsequently, the Cypher query language, which is specifically developed for graph databases, can be used to easily perform CRUD operations on the stored knowledge.

[0070] S3: Knowledge graph-based relation retrieval maps relationships between data processing and manipulation processes. It should also be noted that knowledge graph-based relation retrieval includes:

[0071] Based on the effectiveness of graph databases, Neo4j is used to store knowledge.

[0072] Neo4j is a NoSQL database. The types and attributes of Neo4j nodes and edges are stored separately.

[0073] Neo4j also includes full transaction support, powerful graph search capabilities, and a certain degree of horizontal scalability;

[0074] Full transaction support means that Neo4j requires each data modification to be completed within a transaction, which helps ensure data consistency.

[0075] Neo4j’s powerful graph search capabilities mean that it has clients that support a range of popular languages, including Cypher, a language specifically designed for graph databases, making it easy for users to manipulate and search the Neo4j database.

[0076] Neo4j's ability to scale horizontally means that it can improve performance by performing read-write splitting using Read Replica.

[0077] Furthermore, the association mapping includes:

[0078] The node retrieval of the knowledge graph is used to simulate the data processing and manipulation process, and the logical relationships of data processing are mapped to the logical relationships between nodes.

[0079] S4: In applying anomaly detection algorithms, the relationships within the knowledge graph are utilized to supplement data for anomaly detection of similar data features, improving the anomaly detection performance of large datasets. This step also requires clarification that the anomaly detection algorithm employed includes a method based on Local Anomaly Factors (LOF), the specific algorithm definition of which is as follows:

[0080] For a given data point p, the k-distance k-distance(p) of p determines the neighbors of p in the dataset. The condition that k-distance(p) equals the Euclidean distance d(p, o) between data point p and data point o must be satisfied:

[0081] (1) The dataset contains at least k data points, excluding p. The condition d(p, o) <= d(p, o) is satisfied.

[0082] (2) The dataset contains at most k-1 data points, excluding p. Satisfying d(p, o) <d(p,o)。

[0083] Specifically, calculating the k-distance k-distance(p) of p includes:

[0084] Calculate the distance from p to all data points in the dataset;

[0085] Choose k distinct minimum distances;

[0086] The maximum value among the k minimum distances is the k-distance of p;

[0087] Let N be the set of all neighboring data points whose distance to p is less than or equal to the k-distance (k-distance(p)) to p. k-d (p);

[0088] The reachability distance from object p to object q is defined as: reach-dk(p, q) = max{kd(q), d(p, q)};

[0089] The reachability distance between data points p and q is represented as follows: if data points p and q are very close, the reachability distance is the k-distance of q; if data points p and q are very far apart, the reachability distance is their actual distance.

[0090] The local reachability density of an object p is defined as follows:

[0091]

[0092] The local outlier factor for data point p is defined as follows:

[0093]

[0094] The local anomaly factor is an indicator that measures whether a data point is anomalous compared to its neighbors. It is highly dependent on the choice of the value of k. When k is fixed, the local anomaly factor reflects the degree of anomalousness of a data point.

[0095] Furthermore, the basic idea of ​​Isolation Forest is to use trees to select data from a dataset, isolating feature data that meets certain conditions to the leaf nodes of a tree. This process is repeated until the termination condition is met. After one tree is built, a new isolation tree is built in a loop until an isolation forest is formed. Then, the path length of a certain data leaf node is calculated. The shorter the path length, the earlier it was isolated, i.e., the more likely it is to be an outlier. The algorithm is defined as follows:

[0096] Path length h(x): The number of edges traversed from the root node to the current leaf node.

[0097] Abnormal score s(x, n):

[0098] in

[0099] c(n) = 2H(n-1) - (2(n-1) / n)

[0100]

[0101] Where H(i) is the harmonic number, approximately ln(i) + 0.5772156649

[0102] x represents the xth isolated tree, and n represents the total number of isolated trees.

[0103] Preferably, this invention utilizes two different anomaly detection algorithm models to detect sample electricity consumption data. The algorithms themselves have certain limitations. After optimizing the algorithms by using knowledge graphs for group segmentation, the accuracy of the algorithms is improved to a certain extent. This is the advantage of knowledge graph technology in anomaly detection algorithm models. Knowledge graphs can well reflect the data processing structure and effectively improve the accuracy of anomaly detection.

[0104] Example 2

[0105] To better verify the superior technical effect of the technical solution provided by this invention, this embodiment takes power big data as an example. As shown in Table 1, it is a partial sample data, where pap_r is the peak-to-average value of electricity consumption data. During data preprocessing, it is necessary to calculate the peak electricity consumption of the day, i.e., the value of pap_r1 + pap_r2 + pap_r3. Then, it is necessary to merge the user information and other features in the table, and perform processing such as removing zeros, deduplication, and deleting useless columns. Then, according to the requirements of the anomaly detection algorithm, the data is processed to normalize the values ​​and reduce the dimensionality to 2 dimensions using PCA. Finally, the preprocessed data that can be used for anomaly detection is obtained.

[0106] Table 1: Original data table for outlier detection.

[0107] INDEX PAP_R PAP_R1 PAP_R2 PAP_R3 PAP_R4 128611 0 148.7611 1694.988 749.8567 945.3453 128632 2430 680 1000 360 410 128634 0 100.7943 715.7486 288.4743 1040.204 128639 1220.2 310 430 120 370 128654 396.7 140.9582 160.2018 32.4127 57.812

[0108] After the dataset undergoes the first round of anomaly detection, the advantages of knowledge graphs in easily identifying related entities are leveraged to group users by their region and industry, and then a second round of anomaly detection and correction is performed.

[0109] Unlike the first round, enterprise users do not identify outliers by grouping all enterprises together. Instead, they group enterprises that are similar to their own industry and geographically close, characterized by similar electricity consumption patterns. This approach can improve the accuracy of outlier detection algorithms. For example, some enterprises may have normal electricity consumption based on their industry and geographical environment, but they may still be identified as outliers because they need to be compared with all groups. Using knowledge graphs for group segmentation can avoid these hard-to-detect errors.

[0110] Table 2: Outlier Detection Results.

[0111]

[0112]

[0113] Referring to Table 2, which shows the results of outlier detection, there are significant differences in the judgment of the results between the LOF algorithm and the isolated forest algorithm in this embodiment. This further illustrates the problem of unsatisfactory detection results caused by manually setting parameters in existing methods. After introducing a knowledge graph model for group segmentation, it can be seen that the results have been improved on the basis of the original results, which verifies the real and effective technical effect of the method of the present invention.

[0114] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A data quality improvement method based on knowledge graphs, characterized in that: include, The table creation statements in the database are preprocessed before entity extraction and relation extraction are performed. Automatically build a knowledge graph by extracting the relationships between data warehouse processing flows in the database; Based on the relation retrieval of the knowledge graph, the data processing and processing procedures are mapped to related relationships. In the process of applying anomaly detection algorithms, the relationships in the knowledge graph are used to provide data supplementation for anomaly detection of similar data features, thereby improving the anomaly detection effect of big data. The association mapping includes using node retrieval from a knowledge graph to simulate the data processing and manipulation process, mapping the logical relationships of data processing to the logical relationships between nodes.

2. The data quality improvement method based on knowledge graphs according to claim 1, characterized in that: The preprocessing includes, The table creation process of each table is extracted using regular expressions, and then analyzed sequentially. At the same time, the regular expression is used to remove unnecessary comments and statements to avoid interference; The regular expression rule can be briefly described as the statement between "create table" and ";", which extracts the generation process of a single intermediate table for subsequent analysis.

3. The data quality improvement method based on knowledge graphs according to claim 2, characterized in that: It also includes, After obtaining the table creation statements for each table separately, the SQL statements are analyzed. Based on the logical relationships found in the analysis results, the subordinate relationships between field entities and table entities, as well as the generation relationship between the original table and this table, are extracted.

4. The data quality improvement method based on knowledge graphs according to claim 1 or 3, characterized in that: Establishing the knowledge graph includes, Using graph structures to apply knowledge triples<key,value> Stored in the form of; Entities are stored in the form of nodes; The relationships between entities are stored in the form of edges between nodes, thus efficiently displaying the knowledge information of the knowledge graph.

5. The data quality improvement method based on knowledge graphs according to claim 4, characterized in that: Relationship retrieval based on the knowledge graph includes, Based on the effectiveness of graph databases, Neo4j is used to store knowledge. Neo4j is a NoSQL database, and the categories and attributes of Neo4j nodes and edges are stored separately. Neo4j also includes full transaction support, powerful graph search capabilities, and a certain degree of horizontal scalability. The full transaction support means that Neo4j requires each data modification action to be completed within a transaction, which helps to ensure data consistency. The powerful graph search capability means that Neo4j has clients that support a range of popular languages, including Cypher, a language specifically designed for graph databases, making it easy for users to operate and search the Neo4j database. The aforementioned horizontal scaling capability indicates that Neo4j can improve performance through read-write splitting via Read Replica.

6. The data quality improvement method based on knowledge graphs according to claim 5, characterized in that: The anomaly detection algorithm employed includes a method based on the Local Anomaly Factor (LOF), the specific algorithm definition of which is as follows: For a given data point p, the k-distance k-distance(p) of p can determine the neighbor members of p in the dataset. The condition that k-distance(p) is equal to the Euclidean distance d(p, o) between data point p and data point o needs to satisfy two requirements.

7. The data quality improvement method based on knowledge graphs according to claim 6, characterized in that: The two requirements include, The dataset contains at least k data points, excluding p. The condition d(p, o) <= d(p, o) is satisfied. The dataset contains at most k-1 data points, excluding p. Satisfying d(p, o) <d(p,o)。 8. The data quality improvement method based on knowledge graphs according to claim 6 or 7, characterized in that: Calculating the k-distance k-distance(p) of p includes, Calculate the distance from p to all data points in the dataset; Choose k distinct minimum distances; The maximum value among the k minimum distances is the k-distance of p.

9. The data quality improvement method based on knowledge graphs according to claim 8, characterized in that: It also includes, Let N be the set of all neighboring data points whose distance to p is less than or equal to the k-distance (k-distance(p)) to p. k-d (p); The reachability distance from object p to object q is defined as: reach-dk(p, q) = max{kd(q), d(p, q)}; The reachability distance between data points p and q is represented as follows: if data points p and q are very close, the reachability distance is the k-distance of q; if data points p and q are very far apart, the reachability distance is their actual distance.

Images