Data quality testing and repair methods, devices, equipment, media and program products
By constructing a business indicator knowledge graph and automatically generating verification rules using a pre-trained large model, the problems of lagging verification rules and missed detection of implicit logical conflicts in data quality detection are solved, achieving data repair with high accuracy and logical consistency.
Patent Information
- Application Number
- CN202611000069.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-06
- Publication Date
- 2026-11-17
AI Technical Summary
In existing technologies, because the verification rules need to be configured and maintained manually one by one, the accuracy of data quality detection decreases, making it difficult to detect hidden logical conflicts across fields, and data distortion is easily introduced during the repair process.
By combining the construction of a business indicator knowledge graph with a pre-trained large model, data quality verification rules are automatically generated. Hidden logical conflicts are identified by utilizing the business semantic relationships between fields, and abnormal data repair strategies are determined based on the knowledge graph.
It improves the accuracy of data quality detection, reduces the rate of missed detection of implicit logical conflicts, ensures that the repair results conform to business logic, and avoids data distortion.
Smart Images

Figure CN123115102A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a data quality detection and repair method, apparatus, equipment, medium and program product. Background Technology
[0002] With the development of data platform technology, data platforms are widely used to aggregate and manage data from multiple business systems. The quality of the aggregated data directly affects the accuracy of upper-level business analysis and decision-making. Therefore, it is usually necessary to perform data quality testing on the data imported into the data platform and repair any abnormal data detected.
[0003] In related technologies, technical personnel typically configure validation rules one by one according to business needs. For example, they set non-empty, value range, or format constraints for fields, and then validate the data item by item according to the configured validation rules, mark abnormalities, and repair abnormal data through manual or default value filling.
[0004] Thus, in implementing the above solution, because the verification rules need to be manually configured and maintained one by one, it is difficult to synchronize them in a timely manner when business definitions or data structures change. This leads to problems such as lagging verification rules and decreased detection accuracy. Furthermore, the configured verification rules are mostly aimed at explicit anomalies of a single field, making it difficult to detect implicit logical conflicts across fields that require a comprehensive understanding of the business meanings of multiple fields. This can result in such anomalies being missed. In addition, when repairing abnormal data, the lack of utilization of the business semantic relationships between fields means that simple filling cannot guarantee that the repair results conform to the business logic, which can lead to data distortion. Summary of the Invention
[0005] This application provides a data quality detection and repair method, apparatus, equipment, medium, and program product, which are used to reduce the false negative rate of cross-field implicit logical conflicts in data quality detection and avoid introducing data distortion during data repair.
[0006] In a first aspect, embodiments of this application provide a data quality detection and repair method, which includes: constructing a business indicator knowledge graph based on business documents and a data dictionary of a data platform, wherein the data dictionary is used to describe the field names, data types, value ranges, non-null constraints, and indicator definitions of data fields in the data platform; inputting the business documents and the data dictionary into a pre-trained large model to generate data quality verification rules; detecting the target data to be detected in the data platform based on the data quality verification rules and the business indicator knowledge graph to identify abnormal data; reasoning about the abnormal data based on the business indicator knowledge graph, determining the repair strategy corresponding to the abnormal data based on the reasoning results, and repairing the abnormal data based on the determined repair strategy.
[0007] The technical solution provided in this application brings at least the following beneficial effects: By inputting business documents and data dictionaries into a pre-trained large model to generate data quality verification rules, there is no need to manually configure each verification rule one by one, and the verification rules can be updated as business definitions or data structures change, improving the consistency between the verification rules and actual business, thereby improving the accuracy of detection; By constructing a business indicator knowledge graph and combining it during detection, implicit logical conflicts across fields can be identified by utilizing the business semantic relationships between fields, thereby reducing the false negative rate of such anomalies; By reasoning about abnormal data and determining repair strategies based on the business indicator knowledge graph, the repair results can be made consistent with business logic, thereby avoiding the introduction of data distortion during the repair process.
[0008] One possible implementation involves constructing a business metric knowledge graph, which includes: extracting business metrics, data fields, business dimensions, and business constraint rules from business documents and data dictionaries; and building the knowledge graph using business metrics, data fields, and business dimensions as nodes, and the relationships between metrics and fields, the attribution relationships between fields and dimensions, the business constraint relationships between metrics determined by business constraint rules, and the logical relationships between fields as edges. This achieves a deep binding between business semantics and data elements, providing support for the detection and repair reasoning of implicit logical conflicts across fields.
[0009] Another possible implementation involves inputting business documents and a data dictionary into a pre-trained large model to generate data quality verification rules. This includes: preprocessing the business documents and data dictionary to obtain preprocessed results; inputting the preprocessed results into the pre-trained large model, which then performs semantic understanding and logical reasoning on the preprocessed results to generate data quality verification rules. These rules include statistical verification rules for detecting missing values, incorrect field types, and extreme value anomalies, as well as semantic verification rules for detecting implicit logical conflicts across fields. This achieves the automatic generation of statistical and semantic anomaly detection rules.
[0010] Another possible implementation method, as described above, includes: when business documents or data dictionaries are updated, inputting the updated content into a pre-trained large model to regenerate data quality verification rules, and replacing, adding, or deleting existing data quality verification rules. This allows data quality verification rules to automatically iterate with dynamic changes in business operations, eliminating the need for manual configuration of each rule individually.
[0011] Another possible implementation involves using data quality verification rules and a business indicator knowledge graph to detect target data in the data platform and identify anomalous data. This includes: performing statistical feature detection and business semantic compliance checks on the target data based on data quality verification rules to identify missing values, incorrect field types, extreme value anomalies, and data that violates business semantics; performing graph reasoning based on the business indicator knowledge graph to identify implicit logical conflicts across fields or dimensions in the target data; performing time-series trend learning on the time-series indicator data in the target data to identify time-series anomalies; and confirming the detected data as anomalous data and marking the anomaly type, anomaly location, and related fields to form an anomalous data list. In this way, multi-dimensional collaborative detection simultaneously identifies both common anomalies and implicit logical conflicts, reducing the false negative rate.
[0012] Another possible implementation involves reasoning about abnormal data based on a business metric knowledge graph, determining the corresponding repair strategy based on the reasoning results, and then repairing the abnormal data according to the determined repair strategy. This includes: when the abnormal data is a missing value, filling in the missing value based on the semantic relationships of related fields in the business metric knowledge graph; when the abnormal data is an outlier, correcting the outlier based on the logical relationships of related fields in the business metric knowledge graph; when the abnormal data is a logical conflict, performing logical error correction based on the business constraint relationships in the business metric knowledge graph; and when the abnormal data is a time-series anomaly, determining the cause of the time-series anomaly based on the relationships between time-series metric data and related fields in the business metric knowledge graph, and repairing the time-series anomaly based on the cause. In this way, corresponding repair strategies are determined for different types of anomalies, ensuring that the repair results conform to business semantics.
[0013] Another possible implementation involves, after repairing the abnormal data, performing a secondary verification on the repaired data based on data quality verification rules. If the repaired data does not conform to the data quality verification rules, it is then repaired again. This ensures that the repaired data complies with the data quality verification rules.
[0014] Another possible implementation method, as described above, includes: linking data lineage to locate the source of abnormal data and analyzing its downstream impact; based on the detection results, repair results, source, and downstream impact of the abnormal data, generating a data quality report, which includes a data quality level determined based on the anomaly rate and repair success rate. This achieves source tracing and downstream impact analysis of abnormal data, improving the traceability of data governance.
[0015] Secondly, embodiments of this application provide a data quality detection and repair device, which includes: a graph construction unit, used to construct a business indicator knowledge graph based on the business documents and data dictionary of the data platform, wherein the data dictionary is used to describe the field names, data types, value ranges, non-null constraints, and indicator definitions of the data fields in the data platform; a rule learning unit, used to input the business documents and data dictionary into a pre-trained large model to generate data quality verification rules; a quality detection unit, used to detect the target data to be detected in the data platform based on the data quality verification rules and the business indicator knowledge graph, and identify abnormal data; and a repair unit, used to reason about the abnormal data based on the business indicator knowledge graph, determine the repair strategy corresponding to the abnormal data according to the reasoning result, and repair the abnormal data based on the determined repair strategy.
[0016] One possible implementation is that the aforementioned graph construction unit is specifically used to: extract business metrics, data fields, business dimensions, and business constraint rules from business documents and data dictionaries; and construct a business metric knowledge graph with business metrics, data fields, and business dimensions as nodes, and with the association between metrics and fields, the attribution relationship between fields and dimensions, the business constraint relationship between metrics determined by business constraint rules, and the logical association relationship between fields as edges.
[0017] Another possible implementation is that the rule learning unit mentioned above is specifically used to: preprocess business documents and data dictionaries to obtain preprocessed results; input the preprocessed results into a pre-trained large model, and have the pre-trained large model perform semantic understanding and logical reasoning on the preprocessed results to generate data quality verification rules. The data quality verification rules include statistical verification rules for detecting missing values, field type errors and extreme value anomalies, as well as semantic verification rules for detecting implicit logical conflicts across fields.
[0018] Another possible implementation is that the rule learning unit mentioned above is also used to: input the updated content into the pre-trained large model to regenerate data quality verification rules when business documents or data dictionaries are updated, and to replace, add or delete existing data quality verification rules.
[0019] Another possible implementation is that the aforementioned quality detection unit is specifically used to: perform statistical feature detection and business semantic compliance detection on the target data based on data quality verification rules, and identify missing values, incorrect field types, extreme value anomalies, and data that violates business semantics in the target data; perform graph reasoning based on the business indicator knowledge graph to identify implicit logical conflicts across fields or dimensions in the target data; perform time-series trend learning on the time-series indicator data in the target data to identify time-series anomalies; determine the detected data as abnormal data, and mark the anomaly type, anomaly location, and related fields of the abnormal data to form an abnormal data list.
[0020] Another possible implementation is that the above-mentioned repair unit is specifically used to: fill in missing values based on the semantic relationships of related fields in the business indicator knowledge graph when the abnormal data is a missing value; correct abnormal values based on the logical relationships of related fields in the business indicator knowledge graph when the abnormal data is an outlier; perform logical error correction based on the business constraint relationships in the business indicator knowledge graph when the abnormal data is a logical conflict; and determine the cause of the time-series anomaly based on the relationship between the time-series indicator data and related fields in the business indicator knowledge graph, and repair the time-series anomaly according to the cause.
[0021] Another possible implementation is that the above-mentioned repair unit is also used to: perform secondary verification on the repaired data based on the data quality verification rules, and when the repaired data does not meet the data quality verification rules, repair the repaired data again.
[0022] Another possible implementation is that the above-mentioned repair unit is also used to: link the data lineage, locate the source of abnormal data, and analyze the downstream impact range of abnormal data; based on the detection results, repair results, source and downstream impact range of abnormal data, generate a data quality report, which includes a data quality level, and the data quality level is determined based on the anomaly rate and repair success rate.
[0023] Thirdly, this application provides an electronic device comprising: a processor and a memory; the memory stores a program or instructions executable on the processor, wherein the program or instructions, when executed by the processor, implement the method of the first aspect described above.
[0024] Fourthly, this application provides a readable storage medium on which a program or instructions are stored, which, when executed by a computer, implement the method of the first aspect described above.
[0025] Fifthly, this application provides a computer program product stored in a storage medium, which, when executed by a computer, implements the method described in the first aspect.
[0026] In a sixth aspect, embodiments of this application provide a chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.
[0027] The beneficial effects of the second to sixth aspects mentioned above are described in the corresponding description of the first aspect and will not be repeated here. Attached Figure Description
[0028] Figure 1A schematic diagram of the network architecture for a data quality detection and repair method provided in this application embodiment;
[0029] Figure 2 A flowchart illustrating a data quality detection and repair method provided in an embodiment of this application;
[0030] Figure 3 A flowchart illustrating another data quality detection and repair method provided in this application embodiment;
[0031] Figure 4 A flowchart illustrating yet another data quality detection and repair method provided in this application embodiment;
[0032] Figure 5 A flowchart illustrating yet another data quality detection and repair method provided in this application embodiment;
[0033] Figure 6 A flowchart illustrating yet another data quality detection and repair method provided in this application embodiment;
[0034] Figure 7 A flowchart illustrating yet another data quality detection and repair method provided in this application embodiment;
[0035] Figure 8 A flowchart illustrating the implementation process of a data quality detection and repair method provided in this application embodiment;
[0036] Figure 9 A flowchart illustrating step S1 of a data quality detection and repair method provided in this application embodiment;
[0037] Figure 10 This is a flowchart illustrating step S2 of a data quality detection and repair method provided in an embodiment of this application.
[0038] Figure 11 A flowchart illustrating step S3 of a data quality detection and repair method provided in this application embodiment;
[0039] Figure 12 A flowchart illustrating step S4 of a data quality detection and repair method provided in this application embodiment;
[0040] Figure 13 A flowchart illustrating step S5 of a data quality detection and repair method provided in this application embodiment;
[0041] Figure 14 This is a schematic diagram of the structure of a data quality detection and repair device provided in an embodiment of this application;
[0042] Figure 15 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0043] The data quality testing and repair methods, apparatus, equipment, media, and program products provided in this application will be described in detail below with reference to the accompanying drawings.
[0044] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.
[0045] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.
[0046] The terms "at least one," "at least one of," etc., used in the specification and claims of this application refer to any one, any two, or a combination of two or more of the included items. For example, at least one of a, b, and c can mean: "a," "b," "c," "a and b," "a and c," "b and c," and "a, b, and c," where a, b, and c can be single or multiple. Similarly, "at least two" refers to two or more items, and its meaning is similar to that of "at least one."
[0047] In the description of this application, unless otherwise stated, "a plurality of" means two or more.
[0048] The data quality detection and repair method, apparatus, equipment, medium, and program product provided in this application embodiment can be applied to data quality governance scenarios in data middleware platforms.
[0049] In existing technologies, technical personnel typically configure validation rules one by one according to business needs, and then validate the data item by item according to the configured validation rules and repair abnormal data by filling, deleting, etc. This has the defects of delayed validation rule updates, missed detection of implicit logical conflicts across fields, and easy distortion of repair.
[0050] To address the aforementioned technical issues, embodiments of this application provide a data quality detection and repair method, apparatus, device, medium, and program product. By combining a pre-trained large model with a business indicator knowledge graph, it identifies implicit logical conflicts across fields without requiring manual configuration of verification rules one by one, and ensures that the repair results conform to business semantics.
[0051] The data quality detection and repair methods, apparatus, equipment, media, and program products provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.
[0052] Figure 1 This illustration shows the network architecture of a data quality detection and repair method provided in an embodiment of this application. For example... Figure 1 As shown, the network architecture includes a data quality detection and repair device 101 and a terminal device 102. The data quality detection and repair device 101 and the terminal device 102 are interconnected.
[0053] In some embodiments, the data quality detection and repair device 101 may be a server, a computer, or a processor or processing unit within a server or computer. The server may be a single server or a server cluster consisting of multiple servers. It should be noted that the embodiments of this application do not limit the specific device form of the data quality detection and repair device 101. Figure 1 The data quality detection and repair device 101 is used as an example of a single server.
[0054] In some embodiments, the terminal device may be a mobile phone, tablet computer, laptop computer, handheld computer, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, personal computer (PC), ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc., and the embodiments of this application do not specifically limit it. Figure 1 The example shown is a mobile phone, with terminal device 102 as an example.
[0055] In some embodiments, the data quality detection and repair device 101 is used to construct a business indicator knowledge graph, generate data quality verification rules, detect and repair the target data to be detected, and generate a data quality report; the terminal device 102 is used to provide the target data to be detected to the data quality detection and repair device 101, and to receive and display the data quality report generated by the data quality detection and repair device 101.
[0056] It should be noted that the network architecture described in the embodiments of this application is for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and does not constitute a limitation on the technical solutions provided in the embodiments of this application. As network architectures evolve, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.
[0057] See Figure 2 This is a flowchart illustrating a data quality detection and repair method provided in an embodiment of this application. Figure 2 As shown, the data quality detection and repair method provided in this application embodiment can be implemented by the above-mentioned data quality detection and repair device, specifically including the following steps 201 to 204.
[0058] Step 201: The data quality detection and repair device constructs a business indicator knowledge graph.
[0059] In some embodiments, the data quality detection and repair device collects business data, business documents, and a data dictionary from the data platform. It extracts business metrics, data fields, business dimensions, and business constraint rules from the business documents and data dictionary. Using business metrics, data fields, and business dimensions as nodes, and with the relationships between metrics and fields, the attribution relationships between fields and dimensions, the business constraint relationships between metrics determined by business constraint rules, and the logical relationships between fields as edges, a business metric knowledge graph is constructed to achieve deep binding between business semantics and data elements. The business metric knowledge graph can be implemented as a graph stored in a graph database; however, this application embodiment does not limit the specific type of graph database.
[0060] In some embodiments, when constructing a business indicator knowledge graph, the data quality detection and repair device also extracts indicator definitions from business documents and data dictionaries, and writes the extracted indicator definitions as attributes of the corresponding business indicator nodes into the business indicator knowledge graph.
[0061] In some embodiments, in the business indicator knowledge graph, the attributes of business indicator nodes include indicator ID, indicator name, indicator scope, and statistical period; the attributes of data field nodes include field ID, field name, data type, value range, and non-null constraint; the attributes of business dimension nodes include dimension ID, dimension name, dimension level, and associated dimension; the attributes of the association relationship between indicators and fields include association weight; the attributes of the attribution relationship between fields and dimensions include attribution priority; the attributes of the business constraint relationship between indicators include constraint rules and constraint thresholds; and the attributes of the logical association relationship between fields include association rules and association type.
[0062] Step 202: The data quality detection and repair device inputs the business documents and data dictionary into the pre-trained large model to generate data quality verification rules.
[0063] In some embodiments, the data quality detection and repair device inputs business documents and a data dictionary into a pre-trained large model. The pre-trained large model performs semantic understanding and logical reasoning to generate data quality verification rules. These rules include statistical verification rules for detecting missing values, incorrect field types, and extreme value anomalies, as well as semantic verification rules for detecting implicit logical conflicts across fields. The pre-trained large model can employ a language model with semantic understanding and logical reasoning capabilities. This application embodiment does not limit the specific type of pre-trained large model.
[0064] Step 203: The data quality detection and repair device detects and repairs the target data based on data quality verification rules and business indicator knowledge graphs to identify abnormal data.
[0065] In some embodiments, the data quality detection and repair device acquires the target data to be detected in the data platform, integrates statistical features, business semantic constraints and cross-field relationships to perform multi-dimensional detection on the target data, and determines the detected data as abnormal data.
[0066] Step 204: The data quality detection and repair device infers abnormal data based on the business indicator knowledge graph, determines the repair strategy corresponding to the abnormal data based on the inference results, and repairs the abnormal data based on the determined repair strategy.
[0067] In some embodiments, the data quality detection and repair device performs logical reasoning on abnormal data in conjunction with a business indicator knowledge graph, determines a repair strategy corresponding to the abnormal data based on the reasoning results, and repairs the abnormal data based on the determined repair strategy.
[0068] In some embodiments, the data quality detection and repair device combines a business indicator knowledge graph to perform logical reasoning, analyze the causes of anomalies, and adaptively select repair strategies. For missing values, it selects a reasonable filling method based on the semantic relationships of related fields in the business indicator knowledge graph, such as filling based on business scenario association inference or filling based on the average of similar data. For outliers, it selects repair methods such as anomaly correction or interpolation replacement based on statistical features and business logic. For logical conflicts, it automatically performs logical error correction based on business constraint rules in the business indicator knowledge graph to ensure that the repaired data meets business semantic requirements.
[0069] In some embodiments, combined with Figure 2 ,like Figure 3 As shown, step 201 above can be implemented through steps 201a and 201b.
[0070] Step 201a: The data quality detection and repair device extracts business indicators, data fields, business dimensions, and business constraint rules from business documents and data dictionaries.
[0071] Step 201b: The data quality detection and repair device uses business indicators, data fields, and business dimensions as nodes, and uses the association between indicators and fields, the attribution relationship between fields and dimensions, the business constraint relationship between indicators determined by business constraint rules, and the logical association relationship between fields as edges to construct a business indicator knowledge graph.
[0072] This enables a deep binding between business semantics and data elements, providing support for the detection and repair reasoning of implicit logical conflicts across fields.
[0073] In some embodiments, combined with Figure 2 ,like Figure 4 As shown, step 202 above can be implemented through steps 202a and 202b.
[0074] Step 202a: The data quality detection and repair device preprocesses the business documents and data dictionary to obtain the preprocessing results.
[0075] Step 202b: The data quality detection and repair device inputs the preprocessing results into the pre-trained large model. The pre-trained large model performs semantic understanding and logical reasoning on the preprocessing results to generate data quality verification rules. The data quality verification rules include statistical verification rules for detecting missing values, field type errors, and extreme value anomalies, as well as semantic verification rules for detecting implicit logical conflicts across fields.
[0076] In this way, the detection rules for statistical anomalies and semantic anomalies can be automatically generated.
[0077] In some embodiments, combined with Figure 2 ,like Figure 5 As shown, step 203 above can be implemented through steps 203a and 203b.
[0078] Step 203a: The data quality detection and repair device performs statistical feature detection and business semantic compliance detection on the target data based on data quality verification rules, identifies missing values, incorrect field types, extreme value anomalies, and data that violates business semantics in the target data, and performs graph reasoning based on the business indicator knowledge graph to identify implicit logical conflicts across fields or dimensions in the target data.
[0079] Step 203b: The data quality detection and repair device performs time series trend learning on the time series indicator data in the target data, identifies time series anomalies, determines the detected data as abnormal data, and marks the anomaly type, anomaly location and related fields of the abnormal data to form an abnormal data list.
[0080] Thus, through multi-dimensional collaborative detection of statistical feature detection, business semantic compliance detection, graph reasoning, and time-series trend learning, it is possible to identify cross-field implicit logical conflicts while recognizing regular anomalies, thereby reducing the false negative rate of such anomalies.
[0081] In some embodiments, the above-mentioned multi-dimensional detection achieves fully automatic, multi-dimensional, and implicitly perceptible anomaly detection without manual configuration through a four-layer collaborative detection mechanism, as detailed below:
[0082] Intelligent detection at the statistical level: Based on information such as field type, value range, and non-null constraints automatically extracted from the data dictionary by the large model, it automatically performs missing value detection, field type error detection, and extreme value anomaly detection (such as age exceeding the reasonable value range, amount being negative, etc.), without the need for manual setting of detection thresholds;
[0083] Intelligent Semantic Rule Detection: Relying on the semantic understanding capabilities of large models for business documents and business specifications, it automatically mines constraint rules at the business semantic level (such as matching birth date with age, order time not later than payment time, regional code corresponding to administrative level, etc.), and performs semantic compliance detection on target data based on these rules to identify abnormal data that violate business semantics.
[0084] Cross-field logical conflict detection: Based on the business indicator knowledge graph constructed in step 201, the graph reasoning capability is used to analyze the relationship and constraint between data fields, business indicators, and business dimensions, and to identify implicit logical conflicts across fields and dimensions (such as logical mismatch between sales revenue and sales volume, unit price, and administrative level contradictions between provinces and cities). Such implicit conflicts cannot be detected by conventional single rule detection.
[0085] Intelligent Time-Series Trend Detection: For time-series indicator data (such as daily active users, transaction volume, production capacity, etc.) in the data platform, the large model automatically learns the time-series trend and periodic characteristics of the data, and identifies time-series anomalies such as sudden cliff-like fluctuations, abnormal surges, and trend drifts, without the need for manual preset fluctuation thresholds.
[0086] In some embodiments, combined with Figure 2 ,like Figure 6 As shown, step 204 above can be implemented through steps 204a and 204b.
[0087] Step 204a: The data quality detection and repair device infers abnormal data based on the business indicator knowledge graph and determines the repair strategy corresponding to the abnormal data based on the inference results.
[0088] Step 204b: The data quality detection and repair device repairs the abnormal data based on the determined repair strategy.
[0089] In practice, when the abnormal data is a missing value, the missing value is filled in based on the semantic relationship of the related fields in the business indicator knowledge graph; when the abnormal data is an outlier, the outlier is corrected based on the logical relationship of the related fields in the business indicator knowledge graph; when the abnormal data is a logical conflict, logical error correction is performed based on the business constraint relationship in the business indicator knowledge graph; when the abnormal data is a time-series anomaly, the cause of the time-series anomaly is determined based on the relationship between the time-series indicator data and the related fields in the business indicator knowledge graph, and the time-series anomaly is repaired according to the cause.
[0090] In this way, corresponding repair strategies can be determined for different types of anomalies, so that the repair results conform to business semantics.
[0091] In the data quality detection and repair method provided in this application, when business documents or data dictionaries are updated, the data quality detection and repair device inputs the updated content into the pre-trained large model to regenerate data quality verification rules, and replaces, adds, or deletes existing data quality verification rules, so that the data quality verification rules automatically iterate with the dynamic changes of business.
[0092] In some embodiments, combined with Figure 2 ,like Figure 7 As shown, after step 204 above, the data quality detection and repair method provided in this application embodiment may further include the following steps 205 and 206.
[0093] Step 205: The data quality detection and repair device performs a second verification on the repaired data based on the data quality verification rules. If the repaired data does not meet the data quality verification rules, the repaired data is repaired again.
[0094] Step 206: The data quality detection and repair device links with the data lineage to locate the source of abnormal data and analyze the downstream impact range of abnormal data. Based on the detection results, repair results, source and downstream impact range of abnormal data, a data quality report is generated. The data quality report includes the data quality level, which is determined based on the anomaly rate and repair success rate.
[0095] In some embodiments, when the data quality detection and repair device locates the source of abnormal data, it locates the source business system, ETL process, intermediate table, or indicator calculation layer; when analyzing the downstream impact of abnormal data, it analyzes the reports, business indicators, and business applications associated with the abnormal data; the generated data quality report clarifies the data quality level, anomaly details, repair effect, source tracing results, and improvement suggestions, providing decision support for data governance.
[0096] In some embodiments, the data quality level is assessed based on the anomaly rate, the repair success rate, and the degree of impact. For example, Excellent: anomaly rate ≤ 1%, repair success rate ≥ 99%; Acceptable: 1% < anomaly rate ≤ 5%, repair success rate ≥ 95%; Unacceptable: anomaly rate > 5% or repair success rate < 95%. Wherein, anomaly rate = total amount of abnormal data / total amount of data to be detected, and repair success rate = amount of data repaired / total amount of abnormal data.
[0097] It should be noted that the execution order of steps 205 and 206 described above is not limited in this embodiment. For example, step 205 can be executed first, followed by step 206; or step 206 can be executed first, followed by step 205; or steps 205 and 206 can be executed simultaneously. Figure 4 The example given is that step 205 is executed first, followed by step 206, and does not constitute a limitation on the technical solutions provided in the embodiments of this application.
[0098] In this way, by linking the data lineage to trace the source of abnormal data and analyze its downstream impact, and generating data quality reports, the traceability of data quality problem handling can be improved.
[0099] The data quality detection and repair method of this application will be described below through specific embodiments.
[0100] like Figure 8 As shown, the implementation process of the data quality detection and repair method provided in this application embodiment includes the following steps:
[0101] S1. Construct a knowledge graph of business indicators for the government data platform. Collect data on permanent residents, social security, and government services, as well as corresponding business documents and data dictionaries from the government data platform. Extract business indicators such as the number of permanent residents, the number of people paying social security, and the volume of government services processed. Data fields such as ID number, date of birth, age, social security payment amount, and processing time are also extracted, along with business dimensions such as region, time, and department. Business constraint rules such as age equals the current year minus the date of birth, social security payment amount is not lower than the minimum payment standard, and processing time is not later than the current time are also identified. Construct a knowledge graph of business indicators based on a graph database.
[0102] In some embodiments, S1 above also defines the criteria for the indicators (e.g., the permanent resident population is defined as the population that has lived continuously in the jurisdiction for 6 months or more).
[0103] In some embodiments, such as Figure 9 As shown, the specific implementation method of S1 above is as follows, which consists of 5 feasible steps, fully adaptable to the government data platform scenario, and can be executed directly:
[0104] 1. Data and Document Acquisition Practice: Using the ETL tool of the government data platform, batch collect resident registration data tables, social security payment record tables, and government service processing record tables (formatted uniformly as CSV or JSON); at the same time, download the officially released business documents (PDF format) and data dictionary (Excel format) through the government intranet, filter valid documents (removing expired and obsolete versions), and store them uniformly in a designated folder on the local server, naming them according to the rule of "data type-collection date" (e.g., "resident data-20260430"), for easy retrieval later.
[0105] 2. Practical Operation of Core Element Extraction: A combination of manual assistance and automated tool extraction is used. ① Data Field Extraction: A Python script (which can call the Pandas library) is used to read the collected data table and automatically extract all field names and data types (e.g., ID number is a string, age is an integer). Field names are manually checked and corrected (to avoid name conflicts, e.g., unifying "Social Security Contribution Amount" and "Social Security Payment Amount" as the same field); ② Business Indicator Extraction: Business documents are manually reviewed to select quantifiable and statistical indicators. Using a data dictionary, the associated fields for each indicator are identified (e.g., "Permanent Resident Population" is associated with "ID Number" and "Length of Residence"), forming an indicator-field association table; ③ Business Dimension Extraction: Data is categorized into three core dimensions: "Region (Province / City / District), Time (Year / Month / Day), and Department (Civil Affairs / Social Security / Household Registration)," clarifying the hierarchical relationship of each dimension (e.g., Region dimension: Province → City → District); ④ Constraint rules and definition extraction: Manually sort out the constraint clauses in business documents, and transform the rules described in natural language into quantifiable and reasonable logical expressions (such as "age = current year - date of birth" and "social security payment amount ≥ local minimum standard (600 yuan / month in 2026)"). Clarify the definition of each indicator and form a document.
[0106] 3. Knowledge Graph Node and Relationship Definition Practice: Based on the Neo4j graph database, define the specifications for nodes and relationships in advance. ① Node Definition: Set a unique ID for each of the three types of core nodes (business metrics, data fields, and business dimensions) (format: "node type-serial number", such as "business metric-001"). Supplement necessary attributes (refer to the storage implementation method in the following steps) to ensure that each node attribute is complete and unambiguous. ② Relationship Definition: Clarify the association rules for the four types of relationships. For example, the "metric-field" association is only used to bind metrics to their corresponding data fields, and the "field-field" logical association is only used to bind fields with logical constraints (such as birth date and age) to avoid relationship redundancy.
[0107] 4. Knowledge Graph Construction Practice: Using the Neo4j Desktop client, create a new government data platform knowledge graph project. ① Node Import: Classify the extracted business indicators, data fields, and business dimensions by node type and organize them into an Excel import template (including fields such as ID, name, and attributes). Import them in batches using Neo4j's "Import Data" function. After importing, manually verify the number of nodes and the accuracy of attributes. ② Relationship Import: Based on the sorted relationships, write Cypher import statements (such as "MATCH (a:business indicator{name:'permanent resident population'}), (b:data field{name:'ID number'})CREATE (a)-[r:association{association weight:0.8}]->(b)"). Execute the statements in batches to create relationships. After execution, check the accuracy of the relationship associations through Neo4j's visualization interface and correct association errors (such as incorrectly associated fields and indicators).
[0108] 5. Practical Verification: After construction, verify the usability of the knowledge graph using Cypher queries. For example, query all data fields associated with "Number of Permanent Residents" (statement: MATCH (a:Business Indicator{Name:'Number of Permanent Residents'})-[r:Association]->(b:Data Field) RETURN a.Name, b.Name) to verify the correctness of the associated fields; query the logical association between "Date of Birth" and "Age" (statement: MATCH (a:Data Field{Name:'Date of Birth'})-[r:Logical Association]->(b:Data Field{Name:'Age'}) RETURN r.Association Rule) to verify the correct import of constraint rules, ensuring that the knowledge graph can support subsequent logical reasoning and association analysis.
[0109] Using business metrics, data fields, and business dimensions as nodes, and with "metric-field" associations, "field-dimension" attribution, "metric-metric" constraints, and "field-field" logical associations as edges, a knowledge graph of business metrics for the government data platform is constructed and stored using the Neo4j graph database. The specific storage implementation method (connecting to the above practical steps) is as follows: First, based on the node-relationship model of the Neo4j graph database, three types of core node labels are defined: "business metrics," "data fields," and "business dimensions." Attribute information is added to each node. The attributes of the "business metrics" node include metric ID, metric name, metric definition, and statistical period. The attributes of the "data fields" node include field ID, field name, data type, value range, and non-null constraints. The attributes of the "business dimensions" node include... The process includes defining dimension ID, dimension name, dimension level, and associated dimensions. Next, it defines four types of relationship tags: "Association" (corresponding to the association between an indicator and a field, with the attribute being association weight), "Attribution" (corresponding to the attribution between a field and a dimension, with the attribute being attribution priority), "Constraint" (corresponding to the business constraint relationship between indicators, with the attributes being constraint rules and constraint thresholds), and "Logical Association" (corresponding to the logical association relationship between fields, with the attributes being association rules and association type). Finally, the extracted business indicators, data fields, business dimensions, and various association relationships are imported in batches into the Neo4j graph database using Cypher statements (the import method is clearly defined in the practical steps), completing the storage and deployment of the knowledge graph, achieving deep binding between business semantics and data elements, and supporting subsequent logical reasoning and association analysis.
[0110] S2. Input government business documents and data dictionaries into the pre-trained large model. The pre-trained large model performs semantic understanding and logical reasoning to generate statistical and semantic verification rules. When the business documents or data dictionary are updated, the verification rules are regenerated to achieve automatic iteration.
[0111] In some embodiments, in S2 above, a pre-trained large model (such as ERNIE or ChatGLM) is selected. The government business documents and data dictionary collected in S1 are input into the large model. The semantic understanding module of the large model parses the business logic and data constraints in the documents, and the logical reasoning module extracts data verification rules. For example, constraints such as "age should match the date of birth" and "social security payment amount shall not be lower than the local minimum standard" are parsed from the business documents, and corresponding semantic verification rules are automatically generated. The value range and data type of the fields are parsed from the data dictionary, and statistical verification rules (such as missing value detection rules and extreme value detection rules) are automatically generated.
[0112] Establish a rule iteration mechanism so that when government business documents and data dictionaries are updated, the updated content is automatically input into the large model, and new verification rules are relearned and generated, realizing rule iteration without human intervention and adapting to the dynamic changes of government business.
[0113] In some embodiments, such as Figure 10 As shown, the specific implementation method of S2 above is as follows, which consists of 4 feasible steps, connecting the documents and data collection results of S1, fully adapting to government affairs scenarios, requiring no professional algorithmic background, and can be executed directly:
[0114] 1. Practical Guide to Large Model Selection and Environment Deployment: Prioritize pre-trained large models that are compatible with the government intranet environment and support Chinese semantic understanding (such as ERNIE 3.0 and ChatGLM 3), avoiding models that require external network access; use a dedicated government server for deployment, configured with at least 16 CPU cores, at least 64GB of memory, and at least 24GB of video memory, and install Python 3.8+ and PyTorch 2.0+ environments, installing model-related dependency packages (such as PaddlePaddle and Transformers) via pip commands; if the government intranet restricts external network downloads, import the model weight files offline in advance and deploy them as a local inference service to ensure stable model access and data security.
[0115] 2. Document Preprocessing and Input Practice: The business documents (PDF format) and data dictionary (Excel format) collected and filtered by S1 are preprocessed as follows: ① Document Format Conversion: A Python script (using PyPDF2 and pandas libraries) is used to convert the PDF documents to TXT format, extracting the plain text content and removing invalid information such as headers, footers, and watermarks; for the Excel format data dictionary, core columns such as "field name, data type, value range, and NOT NULL constraint" are extracted and saved as CSV format; ② Content Splitting and Labeling: The preprocessed text is split into "business specification paragraphs" and "data dictionary entries," and the content type is labeled (such as "business constraints" and "field attributes") to avoid misinterpretation by the large model; ③ Batch Input into the Model: Through the model's local inference interface, the preprocessed text is batch input into the large model, and inference parameters are set (temperature coefficient 0.3-0.5 to ensure rigorous output rules; maximum generated length 512 characters to avoid redundancy), triggering the model's semantic understanding and logical reasoning.
[0116] 3. Practical Operation of Automatic Generation and Filtering of Validation Rules: ① Rule Generation: Based on the input document content, the large model automatically outputs two types of validation rules: statistical validation rules (such as "ID number field is a string type with a fixed length of 18 digits", "Age field is an integer with a value range of 1-120 years old", "Social security payment amount is not empty") and semantic validation rules (such as "The age calculated from the year corresponding to the birth date field should be consistent with the age field", "The processing time field must not be later than the current system time", "Social security payment amount ≥ 600 yuan / month (minimum standard in 2026)"); ② Rule Filtering and Standardization: Duplicate and invalid rules (such as logical contradictions and rules that cannot be implemented) are automatically removed. Manual checks are only performed on a sample basis (the sampling rate is not less than 10%) to confirm the accuracy of the rules. The filtered rules are organized into an Excel spreadsheet according to "rule ID, rule type, rule description, associated fields, and applicable scenarios" to unify the rule format and facilitate subsequent calls and iterations.
[0117] 4. Practical Operation of Automatic Rule Iteration and Maintenance: ① Iteration Trigger: Set up a scheduled detection task (e.g., every Sunday at 23:00) to automatically detect the folders storing business documents and data dictionaries in S1 using a Python script. If file updates are found (e.g., revisions to business specifications, additions of fields to the data dictionary), the model relearning process is automatically triggered to generate new validation rules; ② Rule Update: Compare the newly generated rules with the original rules, automatically replace expired and revised rules, add rules corresponding to newly added fields, and delete rules for obsolete fields, without requiring manual deletion or addition; ③ Rule Storage and Retrieval: Store the final validation rules in a relational database (e.g., MySQL), and create a rule index (indexed by rule type and related fields) for quick retrieval during S3 data detection. At the same time, retain the rule iteration history (including iteration time, iteration reason, and modification content) to achieve traceability.
[0118] S3. Through a four-layer collaborative approach involving statistical layer, semantic rule layer, cross-field logic layer, and time-series trend layer, the target data to be detected is inspected to identify missing values, incorrect field types, extreme value anomalies, semantic violations, as well as implicit logical conflicts and time-series anomalies such as mismatch between age and date of birth, contradictions between the administrative levels of provinces and cities, and the number of permanent residents exceeding the total population of the region. The detected data is identified as anomalous data and a list of anomalous data is generated.
[0119] In some embodiments, in S3 above, the target data to be tested from the government data platform (such as statistics on the permanent resident population of a certain region, social security payment data) is obtained, and multi-dimensional testing is carried out based on the verification rules generated in S2 through a four-layer collaborative testing mechanism:
[0120] Intelligent detection at the statistical layer: Based on field information extracted from the data dictionary by the large model, it automatically detects missing values (such as missing ID number, missing social security payment amount), incorrect field type (such as entering a date field as a string), and extreme value anomalies (such as age being 150 years old, social security payment amount being negative), without the need for manual setting of detection thresholds;
[0121] Intelligent Semantic Rule Detection: Relying on the semantic understanding capabilities of large models, based on automatically extracted semantic rules such as "processing time ≤ current time" and "social security payment amount ≥ local minimum standard", it detects semantic violations and anomalies in the data, such as "processing time is a future date" and "social security payment amount is lower than the minimum standard".
[0122] Cross-field logical conflict detection: Based on the knowledge graph of business indicators of the government data platform, the association constraints between fields are analyzed through graph reasoning to detect hidden logical conflicts, such as "age is 20 years old but birth date is 1980" (age and birth date contradictory), "province is Jiangsu Province but city is Guangzhou" (administrative level contradiction), "permanent resident population is greater than the total population of the region" (cross-indicator logical conflict).
[0123] Intelligent detection of time-series trends: For time-series indicators (such as monthly government affairs processing volume) in the government data platform, the system automatically learns the time-series trends and periodic characteristics of processing volume through a large model, and identifies time-series anomalies such as "monthly processing volume suddenly drops by 80% compared to previous periods", without the need for manual preset fluctuation thresholds.
[0124] After the detection is completed, mark the anomaly type (missing value, outlier, logical conflict, time sequence anomaly), anomaly location (specific field, indicator), and related fields (such as age anomaly related to the date of birth field) for each abnormal data, forming an abnormal data list.
[0125] In some embodiments, such as Figure 11 As shown, the detailed implementation of S3 above is as follows, connecting with the verification rules generated by S2, and is divided into four core stages: "detection preparation, four-layer collaborative detection, anomaly marking, and list generation". Each stage has specific operation steps, tools, and parameters, which can be directly implemented to ensure accurate and efficient detection, and meet the practical needs of the government data platform:
[0126] 1. Detection Preparation Practice: Complete the detection environment deployment and data preprocessing in advance to lay the foundation for four-layer collaborative detection. Specific operations are as follows: ① Environment Deployment: Select a dedicated government server, install Python 3.8+ environment, and install the necessary dependency packages (pandas, neo4j, transformers, numpy) to ensure normal access to the Neo4j knowledge graph in S1 and the verification rules stored in MySQL in S2; ② Target Data Acquisition and Preprocessing: Through the API interface of the government data platform, batch acquire the target data to be detected (formatted as CSV or JSON) and store it in a designated local folder; use a Python script (calling the pandas library) to preprocess the target data, removing blank and duplicate rows, unifying field names (keeping them consistent with field names in S1 and S2 to avoid detection failure due to inconsistent field names), and standardizing the format of date, numeric, and other fields (e.g., unifying dates to "YYYY-MM-DD" and removing redundant symbols from numeric fields); ③ Verification rule invocation: Connect to the MySQL database via Python script, call the verification rules generated by S2 according to the categories of "statistical class" and "semantic class", associate the corresponding associated fields and constraint thresholds, and generate a detection rule configuration file (JSON format) for easy unified invocation of subsequent four-layer detection.
[0127] 2. Four-layer collaborative detection practice (core component): Detection is performed sequentially in the order of "statistical layer → semantic rule layer → cross-field logic layer → time-series trend layer". After each layer of detection is completed, the corresponding anomalies are marked. The entire process is automated, with manual intervention only for anomaly sampling and handling special cases. The specific practice is as follows:
[0128] (1) Statistical layer intelligent detection practice: Core detection of routine basic anomalies without manual intervention. Specific operations: ① Rule invocation: The script reads the "statistical verification rules" in the detection rule configuration file, including field type, value range, non-null constraints, etc.; ② Automated detection: Batch execution of detection through Python script. Specific detection logic: a. Missing value detection: Traverse all data fields and determine whether there are null values, empty strings, "unknown" and other invalid values. If they match, mark them as missing value anomalies; b. Field type error detection: Compare the actual type of the data field with the field type defined in the rule (e.g., ID number should be a string, age should be an integer). If they do not match (e.g., age field is a string), mark it as a field type anomaly; c. Extreme value anomaly detection: Based on the field value range defined in the rule, determine whether the data exceeds the reasonable range (e.g., age > 120 years old, social security payment amount < 0 yuan). If it exceeds the range, mark it as an extreme value anomaly; ③ Detection parameter settings: Set batch detection batch (process 1000 data points per batch to avoid server lag), detection timeout threshold (single batch detection timeout ≤ 30 seconds, if timeout occurs, re-initiate detection); ④ Temporary label: After detection, mark each abnormal data point with a "statistical layer abnormality" label, and record the abnormal field and the reason for the abnormality (e.g., "Age field: extreme value abnormality, value 150 years old, exceeding the reasonable range of 1-120 years old").
[0129] (2) Semantic rule intelligent detection practice: The core detection business semantic level anomalies are based on the semantic class verification rules generated by S2. The specific operation is as follows: ① Rule association: The script associates the data completed by the statistical layer with the semantic class verification rules, and clarifies the associated fields corresponding to each rule (such as associating "processing time ≤ current system time" with the "processing time" field); ② Automated detection: The Python script links with the local inference interface of the large model to perform semantic compliance detection on the data. The specific detection logic is as follows: a. Single field semantic detection: Determine whether the field value conforms to the business semantics (such as the "gender" field only allows "male" and "female", and if "other" appears, it is marked as abnormal); b. Cross-field semantic detection: Based on the semantic rules, determine whether the associated field value conforms to the business logic (such as the "processing time" field must not be later than the current system time, and the "social security payment amount" must not be 0 when the "social security payment status" is "normal"); ③ Special case handling: If a semantically ambiguous anomaly is detected (such as "the processing time is a future date, but the remarks are 'appointment processing'"), it is marked as "semantic anomaly - requires review", and is temporarily not included in the anomaly list, and is pushed to manual verification; ④ Temporary marking: After the detection is completed, the semantically abnormal data is marked with the "semantic rule layer anomaly" label, and the anomaly reason and associated semantic rules are supplemented (such as "processing time field: semantic anomaly, the processing time is 2026-05-10, which is later than the current system time 2026-04-30, violating the rule 'processing time ≤ current time'").
[0130] (3) Practical operation of cross-field logical conflict detection: The core is to detect implicit logical conflicts, relying on the knowledge graph of business indicators of the government data platform built by S1. The specific operation is as follows: ① Graph connection: The knowledge graph built by S1 is connected by calling Neo4j through Python script to obtain the logical relationship and constraint rules between fields and indicators; ② Reasoning detection: The script performs logical reasoning detection for each data, combined with the relationship in the knowledge graph. The specific detection logic is as follows: a. Implicit conflict detection between fields: Based on the logical relationship between fields in the knowledge graph, it is judged whether the values of the related fields in the data are contradictory (e.g., "age" is 20 years old and "date of birth" is 1980, which violates the logical relationship rule of "age = current year - date of birth"; "province" is Jiangsu Province and "city" is Guangzhou City, which violates the regional dimension hierarchical relationship rule); b. Logical conflict detection between indicators: Based on the constraint relationship between indicators in the knowledge graph, it is judged whether the values of the related indicators are contradictory (e.g., "number of permanent residents" is greater than "total population of the region", which violates the constraint relationship between the two); ③ Inference statement execution: For each suspected conflicting data, a Cypher inference statement is automatically generated, and the Neo4j graph is called for verification to confirm whether the conflict actually exists (e.g., to detect "age and date of birth contradiction", the statement is executed: MATCH (a: data field {name: 'age'})-[r: logical association]->(b: data field {name: 'date of birth'}) RETURN r. association rule, to verify whether the data violates the rule); ④ Temporary labeling: After the detection is completed, the implicit logical conflict data is labeled with the "cross-field logical conflict anomaly" mark, and the conflicting fields, conflict reasons and related knowledge graph logical relationships are recorded.
[0131] (4) Practical operation of intelligent detection of time series trends: The core is to detect abnormal fluctuations in time series indicators. Relying on the time series learning capability of the large model, the specific operation is as follows: ① Time series data screening: Screen the time series indicator data (such as monthly government affairs processing volume, monthly social security payment number, monthly increase in permanent residents) in the target data through Python script, sort them by time dimension (year / month / day), and organize them into a time series dataset; ② Trend learning: Input the time series dataset into the large model deployed in S2, set the learning parameters (window size = 12, that is, learn the trend based on the data of the past 12 periods; confidence threshold = 0.9, to ensure accurate trend identification), and let the large model automatically learn the normal trend and periodic characteristics of the time series data (such as the seasonal fluctuation pattern of monthly government affairs processing volume); ③ Anomaly detection: Based on the learned trend, the large model performs anomaly identification on the target time series data. The specific detection types are: a. Sudden fluctuation anomaly (such as monthly processing volume decreasing by 80% or surging by 100% compared with the average of previous periods); b. Trend drift anomaly (such as processing volume continuously decreasing for 3 consecutive months, deviating from the normal trend); c. ④ Temporary labeling: After the detection is completed, the time-series abnormal data is labeled with the "time-series trend abnormality" label, and the abnormal indicators, abnormal fluctuations and suspected causes are recorded (such as "monthly government affairs processing volume: time-series abnormality, down 85% compared with the previous month, suspected data collection omission or policy adjustment").
[0132] 3. Practical Guidelines for Anomaly Labeling: Standardize anomaly labeling to ensure clear and traceable anomaly information, facilitating subsequent S4 repair and source tracing. Specific operations include: ① Standardizing the labeling format: Each anomaly data entry's labeling information includes seven core elements: "Data ID, Anomaly Type, Anomaly Field, Anomaly Reason, Association Rule / Graph Relationship, Detection Level, and Detection Time"; ② Label storage: Use a Python script to batch write the labeling information of all anomaly data into an Excel spreadsheet, sorting it by anomaly type (missing values, outliers, logical conflicts, and time-series anomalies); ③ Anomaly sampling inspection: Manually sample 10% of the anomaly data to confirm the accuracy of the anomaly labeling (e.g., whether there are any misjudgments, and whether the anomaly reason description is accurate). If a misjudgment is found, delete the corresponding anomaly label and update the anomaly list.
[0133] 4. Practical Operation of Anomaly Data List Generation: Integrate all detected anomaly data to form a standardized anomaly list, providing a clear basis for S4 remediation. Specific operations include: ① List Organization: Organize the anomaly data confirmed by sampling into the order of "anomaly type → associated fields → detection level", and supplement "associated data ID, anomaly severity" (divided into general, moderate, and severe, such as severe for missing key fields and general for missing non-key fields); ② List Verification: The script automatically verifies the completeness of the list, ensuring that the 7 core marker elements of each anomaly data are complete and error-free; ③ List Storage and Push: Store the anomaly data list (Excel format) in a designated folder on the local server and push it to the government data platform management backend to remind relevant personnel to pay attention. The list naming follows the rule of "detection date - anomaly list" (e.g., "20260430 - government data anomaly list") for easy subsequent retrieval and traceability.
[0134] S4. Based on the business indicator knowledge graph, reason about abnormal data, determine and repair strategies for missing values, outliers, logical conflicts and time-series anomalies, and then perform secondary verification on the repaired data based on data quality verification rules.
[0135] In some embodiments, in S4 above, for the abnormal data marked in S3, logical reasoning is performed using a business metric knowledge graph to adaptively select a repair strategy:
[0136] Missing value repair: For records with missing ID numbers, based on the relationship between "ID number and name, registered address" in the knowledge graph, other valid records with the same name and registered address are queried to complete the ID number; for records with missing social security payment amounts, based on the relationship between "social security payment amount and payment base, payment ratio" in the knowledge graph, combined with the payment standards of the same region and position, the missing value is deduced and filled.
[0137] Outlier correction: For outlier records where the age is 150, the age is recalculated based on the birth date, according to the constraint relationship between "age and date of birth" in the knowledge graph, and the outlier is corrected; for records where the social security payment amount is negative, it is automatically corrected to 0 and marked as requiring manual review.
[0138] Logical conflict correction: For records where "age and date of birth contradict each other", the age is recalculated based on the date of birth, according to the constraint "age = current year - date of birth" in the knowledge graph, thus correcting the logical conflict; for records where "social security payment amount is lower than the minimum standard", the payment amount is automatically adjusted to the minimum standard and an alarm message is generated to remind relevant departments to verify.
[0139] Time-series anomaly repair: For time-series anomalies such as a sudden drop in government service volume, the cause of the anomaly is investigated by combining the relationship between "service volume and business time period and policy adjustment" in the knowledge graph. If it is due to missing data collection, the data is automatically supplemented. If it is due to policy impact, the anomaly is marked and the reason is noted.
[0140] In some embodiments, such as Figure 12 As shown, the detailed implementation of S4 above is as follows, connecting with the abnormal data list of S3, and is divided into four core stages: "abnormal classification and sorting, graph reasoning, adaptive repair, and secondary verification". Each stage has specific operational steps that can be directly implemented, balancing repair accuracy and business compliance:
[0141] 1. Practical Operation of Anomaly Data Classification and Sorting: First, read the anomaly data list (Excel format) generated by S3. Then, use a Python script (calling the pandas library) to classify the data according to anomaly type (missing values, outliers, logical conflicts, and time-series anomalies). At the same time, associate the node information in the knowledge graph and supplement the two columns of "anomaly field associated nodes" and "association constraint rules" to form a classified anomaly list. For example, classify "age anomaly" as an outlier and associate it with the "age" and "date of birth" data field nodes and the logical association rule "age = current year - date of birth". Classify "missing social security payment amount" as a missing value and associate it with the "social security payment amount", "contribution base", and "contribution ratio" field nodes and their association relationships to support subsequent targeted repair. After classification, conduct manual sampling (sampling rate not less than 15%) to confirm the accuracy of classification and avoid repair errors caused by misjudgment of anomaly types.
[0142] 2. Knowledge Graph Reasoning Practice (Core Component): Open the Neo4j Desktop client and connect to the knowledge graph of business indicators in the government data platform built by S1. For different types of abnormal data, use Cypher reasoning statements to mine the causes and correlation evidence of the anomalies, providing support for the repair strategy. The specific operations are as follows: ① Missing Value Reasoning: For a record with "missing social security payment amount", execute the Cypher reasoning statement (MATCH (a: data field {name: 'social security payment amount'})-[r: logical association]->(b: data field {name: 'contribution base'}), (a)-[s: logical association]->(c: data field {name: 'contribution ratio'}) RETURN b.value range, c.value range, r.association rule) to obtain the value range and association rule of contribution base and contribution ratio (social security payment amount = contribution base × contribution ratio), and determine the basis for filling missing values; ② Outlier Reasoning: For an abnormal record with "age is 150 years old", execute the reasoning statement (MATCH (a: data field {name: 'age'})-[r: logical association]->(b: data field {name: 'date of birth'}) RETURN r. association rule, a. value range), obtain the association rule between age and date of birth and the reasonable value range of age, and clarify the cause of the anomaly (incorrect date of birth entry or incorrect age calculation); ③ Logical conflict reasoning: For conflict records where "province is Jiangsu Province but city is Guangzhou City", execute the reasoning statement (MATCH (a: data field {name: 'province'})-[r: attribution]->(b: business dimension {name: 'region'}), (c: data field {name: 'city'})-[s: attribution]->(b) RETURN r. association rule, s. association rule, b. dimension level), obtain the hierarchical relationship of the region dimension and the attribution rule between province and city, and confirm the cause of the conflict (mismatch between city and province attribution); ④ Time-series anomaly reasoning: For the anomaly of "monthly government affairs processing volume suddenly drops by 80%", execute the reasoning statement (MATCH (a: data field {name: 'province'})-[r: attribution]->(b: business dimension {name: 'region'}), (c: data field {name: 'city'})-[s: attribution]->(b) RETURN r. association rule, s. association rule, b. dimension level), obtain the hierarchical relationship of the region dimension and the attribution rule between province and city, and confirm the cause of the conflict (mismatch between city and province attribution); (a: Business Indicator {Name: 'Monthly Government Affairs Processing Volume'})-[r: Related]->(b: Data Field {Name: 'Business Period'}), (a)-[s: Related]->(c: Data Field {Name: 'Policy Adjustment'}) RETURN b. Value Range, c. Value Range), related business period, policy adjustment and other fields, to investigate the cause of the anomaly (data collection omission, policy change, etc.).
[0143] 3. Adaptive Repair Practice (Categorized Implementation): Based on the above graph inference results, adaptive repair strategies are selected for different types of abnormal data. The specific operations are as follows: all repair operations are automated using Python scripts; human intervention is only responsible for verifying abnormal cases.
[0144] (1) Practical operation of missing value repair: ① Missing key fields (such as ID number, date of birth): Based on the relationship between fields in the knowledge graph, use Python script to query other valid data of the same subject and perform correlation filling; for example, for a record with a missing ID number, the script automatically matches other valid records with the same "name, household registration address, contact number", extracts the ID number to fill, and marks it with "correlation filling" after filling; ② Missing non-key fields (such as social security payment amount, processing time): Based on the logical association rules in the knowledge graph, combined with the statistical characteristics of data of the same dimension and type, perform inference filling; for example, if the social security payment amount is missing, the script calculates the filling value according to the inferred rule "social security payment amount = payment base × payment ratio", combined with the average payment base and payment ratio of the same region and the same position; if the associated data cannot be obtained, the median of the non-missing values of the same field is used for filling, and the "inference filling" or "median filling" mark is used after filling.
[0145] (2) Practical operation of outlier repair: ① Numerical outliers (such as age 150 years old, negative social security payment amount): Based on the value range constraints and logical association rules in the knowledge graph, the outlier is automatically corrected; for example, if the age is outlier, the script recalculates the age based on the birth date (age = current year - birth date) and corrects it to a reasonable value; if the social security payment amount is negative, it is automatically corrected to 0 and marked "outlier correction - needs review", reminding manual verification to check whether it is an input error; ② Format outliers (such as incorrect date format, insufficient number of digits in ID card number): The format is automatically corrected by calling the format standardization tool through the Python script; for example, the date format "2026 / 04 / 30" is corrected to "2026-04-30", and if the number of digits in ID card number is less than 18, it is marked as "format outlier - cannot be automatically corrected" and pushed to manual processing.
[0146] (3) Logical Conflict Repair Practice: Based on the business constraint rules in the knowledge graph, automatic logical error correction is performed, prioritizing the data to conform to business semantics; ① Logical conflict between fields (such as contradiction between age and date of birth): Based on the field with "high credibility" in the knowledge graph (such as the date of birth having higher credibility than age), the conflicting field is automatically corrected; for example, if the age is 20 years old and the date of birth is 1980, the age is recalculated to 46 years old based on the date of birth, the age field is corrected, and "Logical Error Correction - Base Field: Date of Birth" is marked; ② Dimensional Attribution Conflict (such as mismatch between province and city): The hierarchical relationship of the regional dimension in the knowledge graph is automatically matched, and the conflicting field is corrected; for example, if the province is Jiangsu Province and the city is Guangzhou City, the city is automatically corrected to a city in Jiangsu Province (such as Nanjing City), and "Logical Error Correction - Regional Attribution Correction" is marked, while generating alarm information to remind relevant departments to verify the original data.
[0147] (4) Practical operation of time series anomaly repair: Based on the anomaly reasoning of the graph, targeted repair is carried out without blindly correcting the data to ensure the rationality of the time series trend; ① Time series anomaly caused by missing data collection: Through the Python script linking the collection interface of the government data platform, the missing data collection period is automatically supplemented and filled into the corresponding position, and marked "Time series repair - supplementary collection"; ② Time series anomaly caused by policy adjustment and business period change: The data is not corrected, but the reason for the anomaly is noted after the anomaly record (such as "Monthly processing volume decrease: due to policy adjustment, some business is suspended"), and is highlighted in the subsequent data quality report to provide reference for business decision-making; ③ Time series anomaly caused by data entry error: The erroneous data is automatically corrected, and combined with the time series trend of the same period, the corrected data is ensured to conform to the overall trend, and marked "Time series repair - error correction".
[0148] 4. Secondary Verification Practice: After the repair is completed, a secondary verification process is automatically triggered to ensure that the repaired data meets the requirements; ① Automatic Verification: The verification rules generated by S2 are called through a Python script to perform a full check on the repaired data to check whether there are still anomalies and whether it meets the business constraint rules; ② Manual Review: 5% of the repaired data that passes the automatic verification is sampled for inspection; all abnormal records that fail the automatic verification and are marked as "required for review" are manually reviewed to confirm the repair plan and make manual adjustments if necessary; ③ Result Archiving: After the secondary verification passes, the repaired data is updated to the government data platform, and the list of abnormal data is updated at the same time, marked "repaired", "repair method", and "repair time" to retain the repair record and achieve traceability of the repair process; if the secondary verification fails, it returns to the repair stage, readjusts the repair strategy, and continues until the verification passes.
[0149] After the repair is completed, the repaired data is subjected to a second verification to ensure that the data meets the verification rules and business semantic requirements.
[0150] S5, linking the data lineage of the government data platform, locates the source of abnormal data and analyzes the downstream impact range. Based on the detection results, repair results, source and downstream impact range, a data quality report is generated. The data quality report includes the data quality level determined based on the anomaly rate and repair success rate.
[0151] In some embodiments, in S5 above, the data lineage of the government data platform is linked to locate the source of abnormal data: for example, if an abnormal number of ages are detected in the permanent resident population data of a certain area, the source business system (such as the household registration system) is traced back to find that it is caused by an error in the date of birth during household registration; the downstream impact of the abnormal data is analyzed, such as the abnormal permanent resident population data of the area will affect the regional social security allocation, government service resource allocation and other related reports and business applications.
[0152] Based on the analysis of detection results (anomaly type, anomaly quantity, and missed detection rate), repair results (repair quantity and repair success rate), anomaly tracing information (source system and error cause), and impact scope, a data quality report is automatically generated, which clarifies the data quality level (e.g., excellent, qualified, unqualified) and proposes targeted improvement suggestions (e.g., optimizing the data entry verification of the household registration system and regularly updating business constraint rules), providing decision support for government data governance.
[0153] In some embodiments, such as Figure 13 As shown, the detailed implementation of S5 above is as follows, connecting the abnormal data list of S3 and the repair results of S4. It is divided into five core links: "data lineage linkage, anomaly tracing practice, downstream impact analysis, automatic generation of quality reports, and report archiving and push". The entire process is mainly automated with manual assistance, which can be directly implemented to ensure accurate tracing and standardized reporting, meeting the traceability and auditability requirements of government data governance.
[0154] 1. Data Lineage Linkage Practice: Complete the connection and configuration of the data lineage link of the government data platform in advance to lay the foundation for anomaly tracing. Specific operations include: ① Lineage Link Connection: Use Python scripts to call the lineage interface of the government data platform (such as the API provided by Apache Atlas and DataHub) to obtain the complete data lineage information, including the complete link of the source business system, ETL processing link, intermediate table, target data table, related indicators, and downstream reports / business applications executed in sequence. Store the lineage information in the Neo4j graph database (linked with the S1 knowledge graph to supplement the "lineage relationship" nodes and edges); ② Lineage Information Calibration: Manually verify the accuracy of the lineage link, focusing on verifying the relationship between "source system and data table", "data table and indicator", and "indicator and downstream application", and correct erroneous associations in the lineage link (such as incorrectly associated business systems or missing ETL processing links); ③ Lineage Query Configuration: Write standardized Cypher query statements to quickly query the complete lineage link of a certain abnormal data, such as "MATCH". (a: Data record {data ID:'XXX'})-[r: blood relation]->(b) RETURN b. Node type, b. Node name, r. Association link”, to ensure that subsequent tracing can be quickly accessed.
[0155] 2. Practical Anomaly Tracing (Core Step): For the anomaly data list generated by S3, conduct batch anomaly tracing to locate the source and root cause of the problem. Specific operations include: ① Batch Tracing Trigger: Use a Python script to read the anomaly data list, extract the data ID and anomaly fields of each anomaly, automatically call the configured Cypher query statement to query the complete lineage of each anomaly, associate it with the anomaly causes recorded in S4, and conduct source tracing analysis; ② Typed Tracing Implementation: a. Missing Value / Format Anomaly Tracing: Tracing back to the data entry stage of the source business system to check whether it is due to omissions or non-standard entry formats (e.g., missing ID numbers are due to omissions in the household registration system, and incorrect date formats are due to inconsistent formats in the social security system); simultaneously check the ETL processing stage to confirm whether it is due to lost fields or incorrect formats during data extraction and transformation; b. Outlier / Logical Conflict Source Tracing: Trace back to the input verification stage of the source business system to check whether it is caused by input errors or missing verification rules (e.g., abnormal age is due to incorrect birth date input during household registration, mismatch between province and city is due to lack of administrative level verification during input); at the same time, combine with the S1 knowledge graph to check whether the business constraint rules are not updated in time, resulting in the source system's verification rules being lagging behind; c. Time-Series Anomaly Source Tracing: Trace back to the data collection stage to check whether it is due to collection interface failure or abnormal collection frequency causing data omission; at the same time, trace back to the business level to check whether it is due to policy adjustments or business process changes causing time-series data anomalies; ③ Source Tracing Result Recording: Record complete source tracing information for each abnormal data, including "source node (e.g., household registration system, ETL processing stage), error stage (e.g., input stage, collection stage), root cause (e.g., input error, interface failure), responsible entity (e.g., household registration management department, technical operation and maintenance department)", and organize it into a source tracing list (Excel format), which is linked and bound to the abnormal data list and repair list; ④ Source tracing verification: 20% of the source tracing results are manually checked to confirm that the source location is accurate and the root cause analysis is reasonable. If an error is found in the source tracing, the bloodline link is re-queried and the source tracing information is corrected.
[0156] 3. Downstream Impact Scope Analysis Practice: Based on the data lineage, analyze the impact of abnormal data on downstream businesses, clarify the scope and severity of the impact. Specific operations include: ① Impact Scope Query: Using a Python script to call the data lineage interface, for each abnormal data point, query its downstream associated intermediate tables, business indicators, reports, and business applications (e.g., social security allocation reports and government service resource allocation applications associated with abnormal resident population data), generating an "Abnormal Data - Downstream Impact" association table; ② Impact Severity Classification: Based on the type, quantity, and importance of downstream applications of abnormal data, classify the impact severity into three levels: Level 1 (Severe): Abnormal data affects core business applications and key decision reports (e.g., abnormal social security payment data affects social security disbursement applications); Level 2 (Slightly Severe): Abnormal data affects general business reports and non-core applications (e.g., abnormal government processing time affects ordinary statistical reports); Level 3 (General): Abnormal data has no direct downstream impact or minimal impact (e.g., missing non-critical fields); ③ Impact Analysis Record: Record the impact scope, impact severity, and affected business entities (e.g., social security departments, government service departments) for each abnormal data point, adding them to the traceability list to provide a basis for subsequent improvement suggestions; ④ Impact Warning: For abnormal data with Level 1 and Level 2 impact, an early warning message (including anomaly details, scope of impact, and source tracing results) is automatically generated and pushed to the affected business departments and technical operations and maintenance departments to remind them to pay attention and handle the situation in a timely manner.
[0157] 4. Practical Operation of Automatic Data Quality Report Generation: Based on detection results, remediation results, source tracing results, and impact scope analysis, a standardized data quality report is automatically generated without manual writing. Specific operations include: ① Report Template Configuration: Pre-configure a government data quality report template (Word format), clearly defining the core modules included in the report: Detection Overview, Anomaly Details, Remediation Effect, Source Tracing Analysis, Impact Scope, Quality Level Assessment, Improvement Suggestions, and Attachments (Anomaly List, Source Tracing List, Remediation List); ② Data Population: Automatically read the anomaly data list from S3, the remediation results from S4, the source tracing list from S5, and the impact analysis records using a Python script (calling the python-docx library), and batch populate them into the report template. Specific population content includes: a. Detection Overview: Total amount of data to be detected, detection time, detection batch, total amount of anomaly data, and anomaly rate; b. Anomaly Details: Statistical analysis of the number and percentage of anomalies by anomaly type, with a focus on explaining the specific circumstances of implicit logical conflicts and time-series anomalies; c. Remediation Effect: Total amount of data repaired, repair success rate, reasons for unrepaired data (e.g., format anomalies that cannot be automatically repaired), and secondary verification pass rate; d. Source tracing analysis: Distribution of abnormal data sources (e.g., proportion of source business systems, proportion of ETL processes), statistics of main error causes; e. Scope of impact: Quantity of abnormal data affected at each level, core applications / reports affected; f. Quality level assessment: Automatically assess quality level based on anomaly rate, repair success rate, and degree of impact (Excellent: Anomaly rate ≤ 1%, repair success rate ≥ 99%; Pass: 1% < Anomaly rate ≤ 5%, repair success rate ≥ 95%; Unpass: Anomaly rate > 5% or repair success rate < 95%); g. Improvement suggestions: Based on the source tracing results, provide targeted suggestions for different error causes (e.g., data entry errors: optimize data entry verification in the source system; ETL failures: strengthen monitoring of the ETL process; rule lag: optimize the automatic rule iteration mechanism); ③ Report optimization: Automatically standardize report format (unify font, line spacing, and heading levels), generate report table of contents, and ensure report clarity and standardization; at the same time, automatically calculate relevant statistical indicators (e.g., anomaly rate = total abnormal data / total data to be detected, repair success rate = amount of data repaired / total abnormal data) to ensure data accuracy.
[0158] 5. Report Archiving and Push Practice: After generating the report, archive and push it through multiple channels to ensure relevant departments receive it in a timely manner. Specific operations include: ① Report Archiving: Store the automatically generated quality report (Word format), associated anomaly list, source tracing list, and repair list in the document management system of the government data platform. Naming should follow the rule of "Detection Date - Data Quality Report" (e.g., "20260430 - Government Data Quality Report"). Record the report archiving time and the person who generated it to ensure report traceability; ② Multi-channel Push: Automatically push the report to designated recipients using Python scripts, including: a. Technical Operations Department: Push the complete report and source tracing list to facilitate troubleshooting and optimizing governance processes; b. Business Department: Push the core summary of the report (anomaly details, scope of impact, improvement suggestions) to understand the impact of data quality on business; c. Data Governance Manager: Push the complete report and statistical reports to understand the overall data quality status and make governance decisions; ③ Report Retention: Retain historical versions of reports, recording the content and time of each modification, to facilitate subsequent comparative analysis of data quality trends and provide a reference for long-term data governance.
[0159] Thus, through S1 to S5, in the context of government data platform, intelligent detection and adaptive repair of data quality can be completed without manual configuration of each verification rule, and anomaly tracing and data quality report generation can be achieved.
[0160] It should be noted that the descriptions of each step S1 to S5 in this embodiment can be found in the descriptions in the above embodiments, and will not be repeated here.
[0161] It should be noted that the above-described method embodiments, or the various possible implementations of the method embodiments, can be executed individually, or, provided there is no conflict, they can be combined with each other. The specific implementation can be determined according to actual usage requirements, and this application embodiment does not impose any restrictions on this.
[0162] As can be seen, the above mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, the embodiments of this application provide corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the modules and algorithm steps of the various examples described in the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0163] This application embodiment can divide the data quality detection and repair device into functional modules based on the above method example. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. Optionally, the module division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.
[0164] In some embodiments, this application also provides a data quality detection and repair apparatus. This data quality detection and repair apparatus may include one or more functional modules for implementing the data quality detection and repair methods described in the above method embodiments.
[0165] For example, Figure 14 This is a schematic diagram of a data quality detection and repair device provided in an embodiment of this application. Figure 14 As shown, the data quality detection and repair device 900 includes: a map construction unit 901, a rule learning unit 902, a quality detection unit 903, and a repair unit 904.
[0166] The aforementioned graph construction unit 901 is used to construct a business indicator knowledge graph; the aforementioned rule learning unit 902 is used to input business documents and data dictionaries into a pre-trained large model to generate data quality verification rules; the aforementioned quality detection unit 903 is used to detect the target data to be detected based on the data quality verification rules and the business indicator knowledge graph, and to identify abnormal data; the aforementioned repair unit 904 is used to reason about the abnormal data based on the business indicator knowledge graph, determine the repair strategy corresponding to the abnormal data based on the reasoning results, and repair the abnormal data based on the determined repair strategy.
[0167] In the data quality detection and repair device provided in this application, the rule learning unit 902 generates data quality verification rules by inputting business documents and data dictionaries into a pre-trained large model. This eliminates the need for manual configuration of each verification rule, and the verification rules can be updated as business definitions or data structures change, improving the consistency between the verification rules and actual business operations, thereby enhancing the accuracy of detection. The graph construction unit 901 constructs a business indicator knowledge graph. When the quality detection unit 903 performs detection, it combines this business indicator knowledge graph to identify implicit logical conflicts across fields using the business semantic relationships between fields, thereby reducing the false negative rate of such anomalies. The repair unit 904 infers from the abnormal data based on the business indicator knowledge graph and determines the repair strategy, ensuring that the repair results conform to business logic, thus avoiding the introduction of data distortion during the repair process.
[0168] In some embodiments, the graph construction unit 901 is specifically used to: extract business metrics, data fields, business dimensions, and business constraint rules from business documents and data dictionaries; and construct a business metric knowledge graph with business metrics, data fields, and business dimensions as nodes, and with the association between metrics and fields, the attribution relationship between fields and dimensions, the business constraint relationship between metrics determined by business constraint rules, and the logical association relationship between fields as edges.
[0169] In some embodiments, the rule learning unit 902 is specifically used to: preprocess business documents and data dictionaries to obtain preprocessing results; input the preprocessing results into a pre-trained large model, and have the pre-trained large model perform semantic understanding and logical reasoning on the preprocessing results to generate data quality verification rules. The data quality verification rules include statistical verification rules for detecting missing values, field type errors, and extreme value anomalies, as well as semantic verification rules for detecting implicit logical conflicts across fields.
[0170] In other embodiments, the quality detection unit 903 is specifically used to: perform statistical feature detection and business semantic compliance detection on the target data based on data quality verification rules, and identify missing values, incorrect field types, extreme value anomalies, and data that violate business semantics in the target data; perform graph reasoning based on the business indicator knowledge graph to identify implicit logical conflicts across fields or dimensions in the target data; perform time-series trend learning on the time-series indicator data in the target data to identify time-series anomalies; determine the detected data as abnormal data, and mark the anomaly type, anomaly location, and associated fields of the abnormal data to form an abnormal data list.
[0171] In some other embodiments, the repair unit 904 is specifically used to: fill in the missing value based on the semantic relationship of the associated fields in the business indicator knowledge graph when the abnormal data is a missing value; correct the abnormal value based on the logical relationship of the associated fields in the business indicator knowledge graph when the abnormal data is an abnormal value; perform logical error correction based on the business constraint relationship in the business indicator knowledge graph when the abnormal data is a logical conflict; and determine the cause of the time-series abnormality based on the relationship between the time-series indicator data and the associated fields in the business indicator knowledge graph when the abnormal data is a time-series abnormality, and repair the time-series abnormality according to the cause.
[0172] In some embodiments, the rule learning unit 902 is further configured to: input the updated content into the pre-trained large model to regenerate data quality verification rules when the business document or data dictionary is updated, and replace, add or delete existing data quality verification rules.
[0173] In some embodiments, the repair unit 904 is further configured to: perform a secondary verification on the repaired data based on the data quality verification rules, and when the repaired data does not conform to the data quality verification rules, repair the repaired data again.
[0174] In some embodiments, the repair unit 904 is further configured to: link the data lineage, locate the source of the abnormal data, and analyze the downstream impact range of the abnormal data; and generate a data quality report based on the detection results, repair results, source and downstream impact range of the abnormal data. The data quality report includes a data quality level, which is determined based on the anomaly rate and the repair success rate.
[0175] It should be noted that the data quality detection and repair device can implement all the processes implemented in the above method embodiments and achieve the same beneficial effects. To avoid repetition, it will not be described again here.
[0176] In the case where the functions of the integrated modules described above are implemented in hardware, this application provides a possible structural schematic diagram of the electronic device involved in the above embodiments. For example... Figure 15 As shown, the electronic device 90 includes: a processor 92, a communication interface 93, and a bus 94. Optionally, the electronic device 90 may also include a memory 91.
[0177] Processor 92 may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 92 may be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 92 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.
[0178] Communication interface 93 is used to connect with other devices via a communication network. This communication network can be Ethernet, wireless access network, wireless local area network (WLAN), etc.
[0179] The memory 91 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.
[0180] As one possible implementation, the memory 91 can exist independently of the processor 92. The memory 91 can be connected to the processor 92 via a bus 94 and is used to store instructions or program code. When the processor 92 calls and executes the instructions or program code stored in the memory 91, it can implement the data quality detection and repair method provided in the embodiments of this application.
[0181] In another possible implementation, memory 91 can also be integrated with processor 92.
[0182] Bus 94 can be an Extended Industry Standard Architecture (EISA) bus, etc. Bus 94 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 15 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.
[0183] Through the above description of the implementation methods, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the service calling device can be divided into different functional modules to complete all or part of the functions described above.
[0184] This application also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described data quality detection and repair method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.
[0185] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.
[0186] This application also provides a readable storage medium storing a program or instructions. When executed by a computer, the program or instructions implement the data quality detection and repair method provided in the above embodiments. It is understood that all or part of the processes in the above method embodiments can be executed by computer instructions instructing related hardware. The readable storage medium can be any of the foregoing embodiments or memory. The readable storage medium can also be an external storage device of the service invocation device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, flash card, etc., equipped on the service invocation device. Further, the readable storage medium can include both internal storage units of the service invocation device and external storage devices. The readable storage medium is used to store the computer program and other programs and data required by the service invocation device. The readable storage medium can also be used to temporarily store data that has been output or will be output.
[0187] This application also provides a computer program product, which is stored in a storage medium and, when executed by a computer, implements the data quality detection and repair method provided in the above embodiments.
[0188] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.
[0189] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0190] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.
Claims
1. A method for data quality detection and repair, the method comprising: include: Based on the business documents and data dictionary of the data platform, a business indicator knowledge graph is constructed. The data dictionary is used to describe the field name, data type, value range, non-null constraint, and indicator definition of the data fields in the data platform. Input the business documents and the data dictionary into the pre-trained large model to generate data quality verification rules; Based on the data quality verification rules and the business indicator knowledge graph, the target data to be detected in the data platform is detected to identify abnormal data. The abnormal data is inferred based on the business indicator knowledge graph, the corresponding repair strategy is determined based on the inference result, and the abnormal data is repaired based on the determined repair strategy.
2. The method of claim 1, wherein, The construction of the business metric knowledge graph includes: Extract business metrics, data fields, business dimensions, and business constraint rules from the business documents and the data dictionary; Using the business metrics, data fields, and business dimensions as nodes, and the association between metrics and fields, the attribution between fields and dimensions, the business constraint relationships between metrics determined by the business constraint rules, and the logical association relationships between fields as edges, a knowledge graph of the business metrics is constructed.
3. The method of claim 1, wherein, The step of inputting business documents and data dictionaries into a pre-trained large model to generate data quality verification rules includes: The business documents and the data dictionary are preprocessed to obtain preprocessing results; The preprocessing results are input into the pre-trained large model, which performs semantic understanding and logical reasoning on the preprocessing results to generate the data quality verification rules. The data quality verification rules include statistical verification rules for detecting missing values, field type errors, and extreme value anomalies, as well as semantic verification rules for detecting implicit logical conflicts across fields.
4. The method of claim 1, wherein, The method further includes: When the business document or the data dictionary is updated, the updated content is input into the pre-trained large model to regenerate data quality verification rules, and existing data quality verification rules are replaced, added, or deleted.
5. The method of claim 1, wherein, Based on the data quality verification rules and the business indicator knowledge graph, the target data to be detected in the data platform is detected to identify abnormal data, including: Based on the data quality verification rules, statistical feature detection and business semantic compliance detection are performed on the target data to identify missing values, incorrect field types, extreme value anomalies, and data that violates business semantics in the target data. Based on the business indicator knowledge graph, graph reasoning is performed to identify implicit logical conflicts across fields or dimensions in the target data; Perform time series trend learning on the time series index data in the target data to identify time series anomalies; The detected data is identified as abnormal data, and the abnormal data is marked with its abnormal type, abnormal location, and associated fields to form an abnormal data list.
6. The method of claim 1, wherein, The process of reasoning about the abnormal data based on the business indicator knowledge graph, determining the corresponding repair strategy for the abnormal data based on the reasoning result, and repairing the abnormal data based on the determined repair strategy includes: When the abnormal data is a missing value, the missing value is filled in based on the semantic relationship of the associated fields in the business indicator knowledge graph; When the abnormal data is an outlier, the outlier is corrected based on the logical relationship between the related fields in the business indicator knowledge graph. When the abnormal data is a logical conflict, logical error correction is performed based on the business constraint relationship in the business indicator knowledge graph. When the abnormal data is a time-series abnormality, the cause of the time-series abnormality is determined based on the association relationship between the time-series indicator data and the associated fields in the business indicator knowledge graph, and the time-series abnormality is repaired according to the cause.
7. The method of claim 1, wherein, After repairing the abnormal data, the method further includes: The repaired data is then subjected to a second verification based on the data quality verification rules. If the repaired data does not conform to the data quality verification rules, the repaired data is repaired again.
8. The method of claim 1, wherein, The method further includes: By linking the data lineage, the source of the abnormal data can be located, and the downstream impact range of the abnormal data can be analyzed. Based on the detection results and repair results of the abnormal data, the source and the downstream impact range, a data quality report is generated. The data quality report includes a data quality level, which is determined based on the anomaly rate and the repair success rate.
9. A data quality detection and repair apparatus, characterized by, include: The graph construction unit is used to construct a business indicator knowledge graph based on the business documents and data dictionary of the data platform. The data dictionary is used to describe the field name, data type, value range, non-null constraint and indicator definition of the data fields in the data platform. The rule learning unit is used to input the business documents and the data dictionary into a pre-trained large model to generate data quality verification rules. The quality detection unit is used to detect the target data to be detected in the data platform based on the data quality verification rules and the business indicator knowledge graph, and to identify abnormal data. The repair unit is used to reason about the abnormal data based on the business indicator knowledge graph, determine the repair strategy corresponding to the abnormal data according to the reasoning result, and repair the abnormal data based on the determined repair strategy.
10. An electronic device, comprising: It includes a processor and a memory, the memory storing programs or instructions that can run on the processor, the programs or instructions being executed by the processor to implement the data quality detection and repair method as described in any one of claims 1-8.
11. A readable storage medium, characterized by, The readable storage medium stores a program or instructions, which, when executed by a computer, implement the data quality detection and repair method as described in any one of claims 1-8.
12. A computer program product, characterised in that, The computer program product is stored in a storage medium, and when the computer program product is executed by a computer, it implements the data quality detection and repair method as described in any one of claims 1-8.
