Data quality evaluation method and system based on graph neural network
By constructing data attributes and relationship correlation subgraphs and using graph neural networks for feature extraction, the problems of more manual participation, low efficiency and insufficient accuracy in the existing technology are solved, and efficient and accurate data quality evaluation is achieved.
Patent Information
- Application Number
- CN202510621722.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-15
- Publication Date
- 2025-08-15
AI Technical Summary
The prior art has problems such as excessive manual participation, low detection efficiency, insufficient accuracy and inability to locate specific abnormal data in data quality evaluation.
Using a graph neural network-based method, by constructing data attribute correlation subgraphs and data relationship correlation subgraphs, the graph neural network is used for feature extraction, and the processing process is automated to distinguish between normal and abnormal data.
It reduces manual participation, improves detection speed and accuracy, improves the detection granularity of abnormal data, and can locate specific abnormal data.
Smart Images
Figure CN120492807A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data quality assessment, and in particular to a data quality assessment method and system based on graph neural network. Background Art
[0002] With the explosive growth and increasing complexity of data, traditional data processing methods have become relatively inefficient and inaccurate in identifying and resolving data quality issues.
[0003] Currently, the most common approach to data analysis and data quality assessment is manual modeling. In addition to modeling data types and relationships, this approach also requires defining data attribute types, lengths, mandatory fields, input formats, and data accuracy (specific details vary depending on the data analysis platform), resulting in a significant workload. Furthermore, data relationship quality assessment is typically based on manually formulated, patterned matching rules. Rules beyond user experience are difficult to fully establish, requiring only accumulated experience gained through data usage.
[0004] Another approach is to evaluate the fields within the rows of relational data to obtain an overall evaluation value. This method can assess the overall quality of the dataset based on some constraints, but it does not assess whether certain rows in the dataset do not meet the general rules of the dataset.
[0005] How to solve the above technical problems is the subject faced by the present invention. Summary of the Invention
[0006] In order to address the shortcomings of the existing technology, the present invention provides a data quality assessment method and system based on graph neural network, which reduces manual participation, has high detection speed, high accuracy and excellent granularity.
[0007] The technical solution adopted by the present invention to solve its technical problem is: In one aspect, the present invention provides a data quality assessment method based on a graph neural network, comprising the following steps: Configure data, input dataset and preprocess dataset data.
[0008] A data attribute association subgraph and a data relationship association subgraph are constructed for each data type of the preprocessed data set to obtain a data association subgraph set.
[0009] A graph neural network is applied to process the data association subgraph set to obtain a result set, and the results of the result set are analyzed to distinguish normal data from abnormal data.
[0010] Outputs a collection of abnormal data.
[0011] Preferably, the configuration data includes the definition of input data entities, attribute definitions, relationship definitions, the relevance of subgraph generation, and the call pipeline; It should be noted that the definition of the input data entity clearly defines the various entities involved in the dataset. For example, in an e-commerce system dataset, entities may include "user," "product," "order," and so on. For each entity, its unique identifier must be determined. For example, the "user" entity uses "user ID" as the primary key, and the "product" entity uses "product ID" as the primary key.
[0012] Attribute definitions specify the attributes of each entity. For example, for a "user" entity, attributes could include "name," "age," "gender," "registration date," "contact information," and so on.
[0013] Relationship definitions are used to define the relationships between entities. For example, in an e-commerce dataset, there is an "order" relationship between "user" and "order," and a "contains" relationship between "order" and "item."
[0014] Defines the degree of association for subgraph generation, determining the closeness and extent of associations between entities when constructing a subgraph. This can be set to different levels of association, such as first-degree association (considering only directly connected entities) or second-degree association (considering directly connected entities as well as entities connected to directly connected entities).
[0015] By calling the pipeline, the processing process is automated to reduce manual participation.
[0016] The preprocessing includes value standardization, value replacement, value mapping, value calculation, and value supplementation.
[0017] It should be noted that value standardization converts attribute values into a unified format or range to eliminate dimensional differences. Value replacement refers to replacing specific values in a dataset. For example, replace "male" with 1 and "female" with 0 in the "gender" attribute. Value mapping refers to mapping attribute values to other values or categories. For example, map the specific category name in the "product category" attribute to a more abstract category group, such as mapping "mobile phone" and "tablet" to the "electronic device" category. Value calculation refers to calculating new attribute values based on existing attribute values. For example, calculate the "unit price of product" based on the "order amount" and "product quantity". Value supplementation refers to using appropriate methods to supplement missing values in a dataset. For numerical attributes, you can use the mean, median, or mode to fill in missing values; for categorical attributes, you can infer based on other related attributes or use default values to fill in.
[0018] Preferably, constructing a data attribute association subgraph and a data relationship association subgraph for each data type of the preprocessed data set, and obtaining a data association subgraph set includes: According to the configuration data, the primary key and attributes of the data type of the preprocessed dataset are constructed as nodes respectively, and the association edges are constructed between the primary key nodes and the attribute nodes to obtain the data attribute association subgraph; For example, for the "User" entity, consider each user's "User ID" as a node, and attributes such as "Age" and "Gender" and their corresponding values as nodes. Build associative edges between the primary key node and the attribute nodes. Edge types can be named based on the attribute names, such as "Age" and "Gender." For example, the "User ID = 1" node is connected to the "Age = 25" node via an "Age" edge, and the "User ID = 1" node is connected to the "Gender = Male" node via a "Gender" edge.
[0019] Build association edges between primary key nodes related to data types to obtain a data relationship association subgraph.
[0020] For example, between the "User" and "Order" entities, the "User ID=1" node and the "Order ID=1001" node are connected by the "Place Order" edge; between the "Order" and "Product" entities, the "Order ID=1001" node and the "Product ID=2001" node are connected by the "Include" edge.
[0021] Construct a data attribute association subgraph and a data relationship association subgraph for each data type in turn until all data types are constructed, and obtain a set of data association subgraphs.
[0022] Preferably, the applying of the graph neural network to process the data association subgraph set to obtain a result set, and analyzing the results of the result set to distinguish normal data from abnormal data includes: Applying a graph neural network to perform feature extraction on the data attribute association subgraph and the data relationship association subgraph respectively to obtain a result set, wherein the result set includes an attribute result set and a relationship result set; It should be noted that for the data attribute association subgraph, the graph neural network will learn the feature representations of nodes and edges to capture the association information between attributes. For example, by learning the features of the edges between the "user ID" node and attribute nodes such as "age" and "gender", the potential relationship between user attributes can be understood. For the data relationship association subgraph, the graph neural network will learn the relationship features between entities and mine the association patterns between entities. For example, by learning the features of the "order" relationship edge between "user" and "order", the user's ordering behavior pattern can be discovered. The result set obtained includes an attribute result set and a relationship result set, which correspond to the feature extraction results of the data attribute association subgraph and the data relationship association subgraph respectively. The model output result is a value between normal 1 and abnormal 0.
[0023] Sort all the results in the attribute result set and the relationship result set by numerical value; Set the percentage parameter n%, and after sorting, the data corresponding to n% of the results are abnormal data.
[0024] It should be noted that the value of the percentage parameter n% is determined based on specific business requirements and data characteristics. For example, in scenarios with high data quality requirements, n% can be set to 10%; in scenarios with relatively low data quality requirements, n% can be set to 20%.
[0025] Preferably, the graph neural network adopts any one of R-GCN, KRAT, and DPGNN.
[0026] Preferably, the abnormal data set includes attribute abnormal data and relationship abnormal data.
[0027] In another aspect, the present invention provides a data quality assessment system based on a graph neural network, comprising: Data input module, used to configure data, input data sets and preprocess data sets; The data processing and analysis module is used to construct a data association subgraph set and perform feature extraction on the data association subgraph set to distinguish normal data from abnormal data; The data output module is used to output abnormal data sets in a visual form.
[0028] The beneficial effects of the present invention are: reduced manual intervention, high detection speed, high accuracy, and excellent granularity. By defining a call pipeline, the processing process is automated, reducing manual intervention. A collection of association graphs is constructed and feature vectors are extracted through a graph neural network, which then analyzes abnormal data, effectively improving detection speed and accuracy. Data attribute association subgraphs and data relationship association subgraphs are constructed separately, and attributes and relationships are analyzed to obtain abnormal conditions. Compared to existing technologies that can only evaluate and analyze the entire data set and cannot locate specific abnormal rows, this effectively improves the granularity of data analysis. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 A diagram showing the steps of the method of the present invention.
[0030] Figure 2 It is a system module diagram of the present invention. DETAILED DESCRIPTION
[0031] In order to clearly illustrate the technical features of this solution, the solution is described below through specific implementation methods.
[0032] Example 1: See also Figure 1As shown, this embodiment is a data quality assessment method based on a graph neural network, comprising the following steps: S1. Configure data, input data set and preprocess the data set.
[0033] Configuration data includes the definition of input data entities, attribute definitions, relationship definitions, the association degree of subgraph generation, and the call pipeline; It should be noted that the definition of the input data entity clearly defines the various entities involved in the dataset. For example, in an e-commerce system dataset, entities may include "user," "product," "order," and so on. For each entity, its unique identifier must be determined. For example, the "user" entity uses "user ID" as the primary key, and the "product" entity uses "product ID" as the primary key.
[0034] Attribute definitions specify the attributes of each entity. For example, for a "user" entity, attributes could include "name," "age," "gender," "registration date," "contact information," and so on.
[0035] Relationship definitions are used to define the relationships between entities. For example, in an e-commerce dataset, there is an "order" relationship between "user" and "order," and a "contains" relationship between "order" and "item."
[0036] Defines the degree of association for subgraph generation, determining the closeness and extent of associations between entities when constructing a subgraph. This can be set to different levels of association, such as first-degree association (considering only directly connected entities) or second-degree association (considering directly connected entities as well as entities connected to directly connected entities).
[0037] By calling the pipeline, the processing process is automated to reduce manual participation.
[0038] Preprocessing includes value standardization, value replacement, value mapping, value calculation, and value supplementation.
[0039] It should be noted that value standardization converts attribute values into a unified format or range to eliminate dimensional differences. Value replacement refers to replacing specific values in a dataset. For example, replace "male" with 1 and "female" with 0 in the "gender" attribute. Value mapping refers to mapping attribute values to other values or categories. For example, map the specific category name in the "product category" attribute to a more abstract category group, such as mapping "mobile phone" and "tablet" to the "electronic device" category. Value calculation refers to calculating new attribute values based on existing attribute values. For example, calculate the "unit price of product" based on the "order amount" and "product quantity". Value supplementation refers to using appropriate methods to supplement missing values in a dataset. For numerical attributes, you can use the mean, median, or mode to fill in missing values; for categorical attributes, you can infer based on other related attributes or use default values to fill in.
[0040] S2. Construct a data attribute association subgraph and a data relationship association subgraph for each data type of the preprocessed data set to obtain a data association subgraph set.
[0041] S20. According to the configuration data, the primary key and attributes of the data type of the preprocessed data set are respectively constructed as nodes, and association edges are constructed between the primary key nodes and the attribute nodes to obtain a data attribute association subgraph.
[0042] For example, for the "User" entity, consider each user's "User ID" as a node, and attributes such as "Age" and "Gender" and their corresponding values as nodes. Build associative edges between the primary key node and the attribute nodes. Edge types can be named based on the attribute names, such as "Age" and "Gender." For example, the "User ID = 1" node is connected to the "Age = 25" node via an "Age" edge, and the "User ID = 1" node is connected to the "Gender = Male" node via a "Gender" edge.
[0043] S21. Build association edges between primary key nodes related to data types to obtain a data relationship association subgraph.
[0044] For example, between the "User" and "Order" entities, the "User ID=1" node and the "Order ID=1001" node are connected by the "Place Order" edge; between the "Order" and "Product" entities, the "Order ID=1001" node and the "Product ID=2001" node are connected by the "Include" edge.
[0045] S22. Construct a data attribute association subgraph and a data relationship association subgraph for each data type in turn until all data types are constructed, thereby obtaining a data association subgraph set.
[0046] S3. Apply graph neural network to process the data association subgraph set to obtain a result set, and analyze the results of the result set to distinguish normal data from abnormal data.
[0047] S30. Apply a graph neural network to perform feature extraction on the data attribute association subgraph and the data relationship association subgraph respectively to obtain a result set, which includes an attribute result set and a relationship result set; wherein the graph neural network adopts any one of R-GCN, KRAT, and DPGNN.
[0048] It should be noted that for the data attribute association subgraph, the graph neural network learns the feature representations of nodes and edges to capture the association information between attributes. For example, by learning the features of the edges between the "user ID" node and attribute nodes such as "age" and "gender", the potential relationship between user attributes can be understood. For the data relationship association subgraph, the graph neural network learns the relationship features between entities and mines the association patterns between entities. For example, by learning the features of the "order" relationship edge between "user" and "order", the user's ordering behavior pattern can be discovered. The resulting result set includes an attribute result set and a relationship result set, which correspond to the feature extraction results of the data attribute association subgraph and the data relationship association subgraph, respectively.
[0049] S31. Sort all results in the attribute result set and the relationship result set by numerical value.
[0050] S32. Set a percentage parameter n%, and after sorting, the data corresponding to n% of the results are abnormal data.
[0051] It should be noted that the value of the percentage parameter n% is determined based on specific business requirements and data characteristics. For example, in scenarios with high data quality requirements, n% can be set to 10%; in scenarios with relatively low data quality requirements, n% can be set to 20%.
[0052] S4. Output the abnormal data set.
[0053] The abnormal data set includes attribute abnormal data and relationship abnormal data. Example 2: See also Figure 2 As shown, this embodiment is a data quality assessment system based on a graph neural network, including: Data input module, used to configure data, input data sets and preprocess data sets; The data processing and analysis module is used to construct a data association subgraph set and perform feature extraction on the data association subgraph set to distinguish normal data from abnormal data; The data output module is used to output abnormal data sets in a visual form.
[0054] Technical features not described in the present invention can be achieved through or by adopting existing technologies and will not be described in detail here. Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by ordinary technicians in this technical field within the essential scope of the present invention should also fall within the scope of protection of the present invention.
Claims
1. A data quality assessment method based on graph neural network, characterized in that: The following steps are involved: Configure data, input data sets and preprocess data sets; Construct a data attribute association subgraph and a data relationship association subgraph for each data type of the preprocessed data set to obtain a data association subgraph set; Apply graph neural networks to process the data association subgraph set to obtain a result set, and analyze the results of the result set to distinguish normal data from abnormal data; Outputs a collection of abnormal data.
2. The data quality assessment method based on graph neural network according to claim 1 is characterized in that: The configuration data includes the definition of input data entities, attribute definitions, relationship definitions, the relevance of subgraph generation, and the call pipeline; The preprocessing includes value standardization, value replacement, value mapping, value calculation, and value supplementation.
3. The data quality assessment method based on graph neural network according to claim 2 is characterized in that: The data attribute association subgraph and the data relationship association subgraph are constructed for each data type of the preprocessed data set, and the obtained data association subgraph set includes: According to the configuration data, the primary key and attributes of the data type of the preprocessed dataset are constructed as nodes respectively, and the association edges are constructed between the primary key nodes and the attribute nodes to obtain the data attribute association subgraph; Build association edges between primary key nodes related to data types to obtain a data relationship association subgraph; Construct a data attribute association subgraph and a data relationship association subgraph for each data type in turn until all data types are constructed, and obtain a set of data association subgraphs.
4. The data quality assessment method based on graph neural network according to claim 3 is characterized in that: The application of the graph neural network to process the data association subgraph set to obtain a result set, and analyzing the results of the result set to distinguish normal data from abnormal data includes: Applying a graph neural network to perform feature extraction on the data attribute association subgraph and the data relationship association subgraph respectively to obtain a result set, wherein the result set includes an attribute result set and a relationship result set; Sort all the results in the attribute result set and the relationship result set by numerical value; Set the percentage parameter n%, and after sorting, the data corresponding to n% of the results are abnormal data.
5. The data quality assessment method based on graph neural network according to claim 4 is characterized in that: Any one of the graph neural networks R-GCN, KRAT, and DPGNN.
6. The data quality assessment method based on graph neural network according to claim 5 is characterized in that: The abnormal data set includes attribute abnormal data and relationship abnormal data.
7. A data quality assessment system based on graph neural network, characterized in that: include: Data input module, used to configure data, input data sets and preprocess data sets; The data processing and analysis module is used to construct a data association subgraph set and perform feature extraction on the data association subgraph set to distinguish normal data from abnormal data; The data output module is used to output abnormal data sets in a visual form.