Data quality checking method and system based on checking scheme matching

Abstract

The present application relates to the technical field of data processing, and specifically provides a data quality inspection method and system based on inspection scheme matching, comprising: constructing an inspection scheme library in advance, each scheme being encapsulated with an inspection rule, an applicable field, a table structure feature and a data content feature. A target inspection table is obtained, its metadata, data content and field information are extracted; multi-dimensional matching is performed based on the field, metadata and data content to obtain a candidate scheme. The target inspection scheme is determined according to the inspection rule in the candidate scheme and data quality inspection is performed; historical rectification records associated with the candidate scheme are aggregated, and a rectification scheme is generated in combination with the inspection result. The present application realizes intelligent matching and rectification closed loop of the inspection scheme, and improves the data quality management efficiency.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, specifically relating to a data quality inspection method and system based on inspection scheme matching. Background Technology

[0002] With the popularization of information technology, data has become the core foundation for enterprise business decisions. However, data quality issues directly affect operational efficiency and decision accuracy. Traditional data quality inspection methods rely heavily on manual operation, which is inefficient and prone to errors. Some existing intelligent matching methods match inspection rules by checking the metadata information of the table (such as table name and field name), but they ignore the data content itself, making it difficult to solve the "name-for-nothing" problem—for example, the "name" field in a financial table and a material table may have completely different meanings, and the required inspection rules may be mismatched.

[0003] In addition, existing methods mainly focus on the creation of inspection rules, and pay insufficient attention to the subsequent rectification of inspection results, which is precisely the most time-consuming and labor-intensive process in data governance.

[0004] Therefore, there is an urgent need for a data quality inspection method that can combine the dual characteristics of metadata and data content, achieve accurate matching of inspection schemes, and form a closed loop of inspection and rectification. Summary of the Invention

[0005] In view of the above-mentioned shortcomings of the prior art, the present invention provides a data quality inspection method and system based on inspection scheme matching to solve the above-mentioned technical problems.

[0006] In a first aspect, the present invention provides a data quality inspection method based on inspection scheme matching, comprising: A pre-built inspection scheme library is constructed, wherein each inspection scheme encapsulates at least: one or more inspection rules, applicable fields, table structure characteristics applicable to the inspection scheme, and data content characteristics; Obtain the target checklist for data quality inspection, extract the metadata and data content information of the target checklist, and determine its domain information based on the data content information; Based on the domain information, metadata information, and data content information of the target checklist, a multi-dimensional match is performed with the checklists in the checklist library to determine one or more candidate checklists. Based on the inspection rules encapsulated in the candidate inspection scheme, an applicable target inspection scheme is determined for the target inspection table, and a data quality inspection is performed on the target inspection table based on the target inspection scheme. Aggregate the historical rectification records pre-associated with each candidate inspection plan, and generate a rectification plan based on the historical rectification records and data quality inspection results.

[0007] Secondly, the present invention provides a data quality inspection system based on inspection scheme matching, comprising: The pre-storage module is used to pre-build an inspection scheme library, wherein each inspection scheme encapsulates at least: one or more inspection rules, applicable fields, table structure features applicable to the inspection scheme, and data content features; The target parsing module is used to obtain the target checklist to be checked for data quality, extract the metadata information and data content information of the target checklist, and determine the domain information to which it belongs based on the data content information. The scheme matching module is used to perform multi-dimensional matching with the inspection schemes in the inspection scheme library based on the domain information, metadata information and data content information of the target inspection table, so as to determine one or more candidate inspection schemes. The scheme execution module is used to determine the applicable target inspection scheme for the target inspection table according to the inspection rules encapsulated in the candidate inspection scheme, and to perform data quality inspection on the target inspection table based on the target inspection scheme; The rectification assistance module is used to aggregate the historical rectification records pre-associated with each candidate inspection plan, and generate rectification plans based on the historical rectification records and data quality inspection results.

[0008] Thirdly, a device is provided, comprising: Memory, used to store data quality inspection programs based on inspection scheme matching; A processor is configured to implement the steps of the data quality inspection method based on check scheme matching as provided in the first aspect when executing the data quality inspection program based on check scheme matching.

[0009] Fourthly, a computer-readable medium is provided, on which a data quality inspection program based on inspection scheme matching is stored, wherein when the data quality inspection program based on inspection scheme matching is executed by a processor, the data quality inspection method based on inspection scheme matching provided in the first aspect is implemented.

[0010] The beneficial effects of this invention are as follows: The data quality inspection method and system based on inspection scheme matching provided by this invention effectively solves the problem of rule mismatch caused by homonyms in traditional methods by constructing an inspection scheme library encapsulating table structure features and data content features, and adopting a hybrid matching mechanism of metadata initial screening + data content fine matching, thus significantly improving the accuracy of inspection scheme recommendations. Furthermore, by associating historical rectification records and generating intention-aligned rectification guidelines, a closed loop of inspection-rectification-reuse is formed, enabling the continuous accumulation of data governance experience and significantly improving the intelligence level and overall efficiency of data quality management. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic flowchart of a method according to an embodiment of the present invention.

[0013] Figure 2 This is a schematic flowchart illustrating the matching process of a method with an inspection scheme according to an embodiment of the present invention.

[0014] Figure 3 This is a schematic flowchart illustrating the generation and fusion inspection scheme of a method according to an embodiment of the present invention.

[0015] Figure 4 This is a schematic block diagram of a system according to an embodiment of the present invention. Detailed Implementation

[0016] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0017] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0018] The data quality inspection method based on inspection scheme matching provided in this embodiment of the invention is executed by a computer device, and correspondingly, the data quality inspection system based on inspection scheme matching runs in the computer device.

[0019] Figure 1 This is a schematic flowchart illustrating a method according to an embodiment of the present invention. Wherein, Figure 1 The implementing entity can be a data quality inspection system based on inspection scheme matching. Depending on different requirements, the order of the steps in this flowchart can be changed, and some can be omitted.

[0020] like Figure 1 As shown, the method includes: S1. A pre-built inspection scheme library is constructed, wherein each inspection scheme encapsulates at least: one or more inspection rules, applicable fields, table structure features applicable to the inspection scheme, and data content features; S2. Obtain the target checklist for data quality inspection, extract the metadata information and data content information of the target checklist, and determine its domain information based on the data content information; S3. Based on the domain information, metadata information and data content information of the target checklist, perform multi-dimensional matching with the checklists in the checklist library to determine one or more candidate checklists; S4. Based on the inspection rules encapsulated in the candidate inspection scheme, determine the applicable target inspection scheme for the target inspection table, and perform data quality inspection on the target inspection table based on the target inspection scheme; S5. Aggregate the historical rectification records pre-associated with each candidate inspection plan, and generate a rectification plan based on the historical rectification records and data quality inspection results.

[0021] In one embodiment of the present invention, based on step S1, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0022] A pre-built library of inspection plans can be stored in a relational database, graph database, or file system. Each inspection plan is encapsulated as an independent data object, containing at least the following four types of information: (a) Set of inspection rules Each check rule should include at least the rule type (e.g., "NOTENAL Validation", "NOTENAL Validation", "NOTENAL Validation", "NOTENAL Validation", "NOTENAL Validation", "NOTENAL Validation"), rule parameters (e.g., regular expression `^[A-Z0-9]+$`, value range `[0,100]`), and the identifier of the target field associated with the rule. For example, a rule for the "material code" field can be represented as: `{ruleType:"REGEX", field: "material_code", params: {pattern: "^MT[0-9]{6}$"}}`.

[0023] (II) Applicable Fields The "Applicable Area" field identifies the business scope to which the inspection plan applies, such as "finance," "materials management," or "medical field." This field is stored as a tag for easy filtering based on the area information.

[0024] (III) Table Structure Features Table structure features include metadata information such as table name, list of field names, field types (e.g., VARCHAR, DECIMAL, DATE), and field descriptions. For example, a check scheme for a "financial voucher table" might have the following table structure features: table name "voucher", field "voucher_no" (VARCHAR, "voucher number"), field "amount" (DECIMAL, "amount"), and field "posting_date" (DATE, "posting date").

[0025] (iv) Data content characteristics The data content characteristics include the correct data samples applicable to the solution. The system can extract several records from historical normal data as samples, for example, storing sample values ​​such as "FV-20240001" and "FV-20240002" for the "voucher number" field mentioned above. These sample data are used for subsequent second-stage matching (precise data content matching), by comparing the data samples in the target check table with the correct samples stored in the solution to determine whether they belong to the same business semantics.

[0026] Construction method: The initial solution library can be generated by importing existing historical inspection solutions, manual configuration, or from standard data quality specification templates. When each solution is added to the library, the system automatically extracts its relevant table structure features (by parsing the database schema) and data content features (by sampling correct data), and encapsulates and stores them along with the rules. The solution library supports incremental updates; newly created solutions can be dynamically added after evaluation.

[0027] In one embodiment of the present invention, based on step S2, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0028] After obtaining the target check table for data quality inspection, its metadata information (such as table name, field names, field types, and field comments) and data content information are extracted first. To determine the business domain to which the target check table belongs, a domain identification method based on data content is adopted, specifically including the following sub-steps: Step 1: Data Sampling Perform lightweight sampling on the target check table, such as extracting the first 1000 rows of data for each field, or randomly sampling 1% of the total number of rows in the table (or sampling the entire table if the number of rows is less than 1000). The sampling process should avoid full table scans to reduce performance overhead.

[0029] Step 2: Field-level domain probability prediction The system extracts the field name, field annotation, and sample data for each field, concatenating them into a text input. This text is then fed into a pre-trained text classification model, such as a lightweight classifier based on BERT or FastText. The model predefines several domain categories, such as "finance," "materials," "medical," and "human resources," and outputs the probability distribution of each field belonging to each domain. For example, a field named "amount" with sample values ​​of "123.45" and "678.90" might output: {finance: 0.85, materials: 0.10, other: 0.05}.

[0030] Step 3: Field type classification and weight setting Fields are categorized into core business fields and non-core fields based on their metadata (e.g., field names containing keywords such as "price", "quantity", "date", "id", or field types such as DECIMAL, DATE, INT). Core business fields include, but are not limited to: amount fields (e.g., "amount", "price"), quantity fields (e.g., "quantity", "count"), date fields (e.g., "create_date", "posting_date"), and business entity identifier fields (e.g., "invoice_id", "material_code"). Non-core fields include auxiliary fields such as remarks, descriptions, and status. The weight of core fields is set to 2 to 5 times that of non-core fields; for example, a core field weight of 2.0 and a non-core field weight of 1.0.

[0031] Step 4: Weighted summation of table-level domains For each predefined domain, calculate the weighted total score for all fields in the target checklist:

[0032] in For the weight of the field, This represents the probability that a field belongs to this domain. For example, if a table contains two core fields (amount and date) and one non-core field (remarks), then the score for the finance domain is: 2.0 × 0.85 + 2.0 × 0.90 + 1.0 × 0.20 = 3.7.

[0033] Step 5: Candidate Domain Determination and Manual Backup The domain with the highest weighted total score is selected as a candidate domain, and its score is checked against a preset threshold (e.g., 2.5). If the condition is met, the candidate domain is directly identified as the domain information in the target checklist; otherwise, the domain information is marked as "general domain," and a manual confirmation request is sent to the user interface, prompting the user to manually annotate the domain. After receiving the manually annotated domain from the user, the system uses it as the final domain information and may selectively store the annotation result in the knowledge base for incremental training of the subsequent model.

[0034] In one embodiment of the present invention, based on step S3, the following will provide a possible embodiment and its specific implementation will be described in a non-limiting manner. Please refer to [the relevant documentation]. Figure 2 .

[0035] S201. Domain Pre-filtering Based on the domain information of the target checklist (e.g., "financial domain"), the checklist library is traversed to filter checklists whose applicable domains are the same as or compatible with the domain information. Compatibility relationships can be defined using a predefined domain compatibility mapping table; for example, "financial domain" and "audit domain" are considered compatible, and "materials domain" and "inventory domain" are considered compatible. After filtering, a first candidate set of checklists is obtained, denoted as... ,in This represents the number of schemes that passed the domain filtering.

[0036] S202. Metadata Similarity Calculation For each inspection scheme in the first candidate scheme set Calculate the metadata information of the target checklist and Metadata similarity scores between table structure features.

[0037] Let the table structure feature vector of the target checklist be... ,plan The table structure feature vector is The table structure features include: table name (after word segmentation and vectorization), set of field names, distribution of field types, and semantic vectors of field annotations. Weighted cosine similarity is used for calculation.

[0038] in, The characteristic dimensions representing metadata (such as table name similarity, Jaccard coefficient of field name set, field type consistency score, etc.). Preset weights for each feature. As a simplified implementation, weighted Euclidean distance can also be used and converted to similarity:

[0039]

[0040] Output each solution Metadata similarity score .

[0041] S203. Data Content Similarity Calculation Calculate the content similarity score between the data content information of the target checklist and the data content features of each scheme in the first candidate scheme set.

[0042] The data content characteristics mainly include the set of correct data samples for each field. Let's define the target checklist. a certain field The sampled data sequence is ,plan Corresponding fields The correct sample sequence for (field name matching) is .

[0043] For character fields, text similarity algorithms (such as cosine similarity based on Sentence-BERT or edit distance normalization) are used to calculate the semantic similarity between the two sample sets:

[0044] in This indicates that the entire sample set is embedded (e.g., the average of all sample vectors is taken).

[0045] For numerical fields, compare numerical features such as order of magnitude and precision. First, extract the statistical measure of the numerical feature: mean. Standard deviation Similarity can be calculated in the following ways: minimum value, maximum value, quantiles, etc.

[0046] in This is to prevent division by zero for extremely small positive numbers.

[0047] For the entire table, the similarity of all fields is weighted and averaged to obtain the solution. Content similarity score:

[0048] in Assign field weights (e.g., primary key fields have lower weights, while core business fields have higher weights). This indicates whether to select either text or numeric similarity calculation method based on the field type.

[0049] S204. Weighted Fusion Score For each inspection scheme in the first candidate scheme set The metadata similarity score and content similarity score are weighted and fused to obtain the comprehensive matching score:

[0050] in This is a preset metadata similarity weight, which can be adjusted according to the business scenario. In general application scenarios, it can be set... This means that content similarity is weighted slightly higher than metadata similarity to reflect the dominant role of data content features in matching accuracy. Non-linear fusion can also be used, for example:

[0051] Alternatively, introduce a domain matching indicator:

[0052] in Provides a domain matching indicator function (1 for exact match, 0.5 for compatible match). Domain-specific weights are assigned. To simplify implementation, this embodiment uses a linear weighting method.

[0053] S205. Result Filtering Sort all solutions in the first candidate solution set by their comprehensive matching scores in descending order, and select those with comprehensive matching scores higher than a preset threshold. (For example The selected solutions are used as the final matching result for multi-dimensional matching. If the number of solutions with scores higher than the threshold exceeds the system's set limit... (For example, 10), then only the top scorers are retained. One option.

[0054] In one specific implementation, after multi-dimensional matching is performed, if the overall matching score of all candidate solutions is lower than a preset threshold (i.e. no suitable candidate inspection solution is matched), the system automatically enters the inspection solution self-generation process, which includes the following steps.

[0055] Step S301: Constructing a heterogeneous map Based on the existing inspection scheme library, a heterogeneous map can be constructed in advance or dynamically as needed. The graph contains three types of nodes: Table nodes store metadata such as table names, table comments, and business categories (e.g., "financial" and "materials"). Field node: Stores field name, field type, enumeration value list, statistical features (such as mean, variance, null value rate, etc.); Rule node: Stores rule types (such as "NOTING", "UNIQUE", "NOTING CURRENT", "Cross-table Consistency") and rule parameters.

[0056] The edges between nodes include: Table-to-field edge: indicates an inclusion relationship; Edge from a field to a rule: indicates that the rule applies to that field; Edges from rule to rule: These represent common associations or execution dependencies between rules (e.g., "not empty check" must be performed before "format check" can be performed).

[0057] The graph is stored in a graph database (such as Neo4j) or an in-memory graph structure, supporting efficient nearest neighbor retrieval.

[0058] Step S302: Extract the comprehensive feature vector of the target checklist For the current target checklist The system extracts and fuses multi-dimensional features to generate a comprehensive feature vector. : Structural embedding: Graph Neural Networks (GNNs) are used to encode the table structure (field order, nesting relationships) of the target checklist into structural vectors. Specifically, the target checklist is viewed as a star graph (with table nodes at the center and field nodes around it), and a fixed-dimensional structural embedding vector is obtained by aggregating these nodes using a Graph Convolutional Network (GCN). .

[0059] Semantic embedding: The table name, field names, and field annotations of the target checklist are encoded using a pre-trained natural language processing model (such as BERT or RoBERTa). The semantic embedding vector is obtained by averaging the semantic vectors of each field or by using attention pooling. .

[0060] Data distribution embedding: Lightweight sampling is performed on the target checklist (e.g., the first 500 rows of each field) to extract data distribution features: for numerical fields, the mean, variance, skewness, and kurtosis are calculated; for text fields, the average length, proportion of special characters, and entropy are calculated. These features are then normalized and concatenated to obtain the data distribution embedding vector. .

[0061] Feature fusion: The three vectors mentioned above are fused through a fully connected layer.

[0062] in and These are trainable parameters (or preset fixed weights). The final result is a comprehensive feature vector. .

[0063] Step S303: Solution reasoning based on attention mechanism In heterogeneous maps In China, utilizing Search for the most similar Top There are K table nodes. Similarity is calculated using cosine similarity or Euclidean distance. Let the retrieval result be a set of similar table nodes. .

[0064] For each similar table node, extract the set of rule nodes connected to it through the graph path (from table to field to rule and rule to rule), and record the frequency of occurrence of each rule and the co-occurrence relationship between rules.

[0065] Then, an attention mechanism is used to calculate the weight contribution of each similar table node to the target check table. as query vector The feature vector of each similar table node (That is, the comprehensive features stored by the node in the table when constructing the graph) as the key vector Calculate attention weights:

[0066] in The similarity is either the dot product or the cosine similarity.

[0067] The occurrence probabilities of each rule node are aggregated based on attention weights to obtain each candidate rule. Aggregate score:

[0068] in Indicates the first Rules in the rule set associated with similar table nodes Does it occur (or how often does it occur)? Select rules with aggregate scores higher than the threshold to form a candidate rule set.

[0069] Furthermore, by utilizing the co-occurrence relationships between rules (obtained from the "rule → rule edge" in the graph) and attention weights, the execution dependency order between rules is predicted. For example, if the "not empty" validation rule and the "format validation" rule have strong co-occurrence and their historical execution order is mostly the former before the latter, then an ordered rule sequence is generated. And the corresponding execution priority.

[0070] Step S304: Dynamic regular parameter fitting For each rule in the rule sequence generated in step S303, the system automatically generates appropriate rule parameters based on the actual data samples of the target checklist, rather than directly reusing the parameters of historical schemes.

[0071] Regular expression rules: Extract sample values ​​of the target field (e.g., user ID samples are "USR_001" and "USR_002"), and generate regular expressions using the longest common substring or pattern mining algorithm, such as "^USR_$".

[0072] Value range rule: The minimum and maximum values ​​of the statistical sample are set to a value range of... Or, expand by a certain percentage based on business tolerance.

[0073] The null value rate rule: Calculate the proportion of null values ​​in the sample and set the tolerance threshold to the proportion of null values ​​in the sample plus a buffer value (such as 0.05).

[0074] Cross-table consistency rules: If a foreign key relationship is found between the target check table and another table (through metadata or field name similarity), a rule that "must exist in the other table" is automatically generated and the parameters of the associated fields are fitted.

[0075] Step S305: Closed-loop verification and warehousing The generated inspection plan is simulated and executed in the background (Dry Run), that is, the plan is used to perform data quality checks on the target inspection table, but no actual output is given to the user.

[0076] If the error rate is too high (e.g., exceeding 90%), the judgment rule parameters are too strict. The system will automatically roll back and relax the parameters (e.g., expand the range of values, reduce the strictness of regular expression matching) and regenerate the solution.

[0077] If the error rate is extremely low (e.g., 0%), the judgment rule may be invalid or too lenient, so the rule should be automatically strengthened (e.g., by adding additional format restrictions or adding joint checks).

[0078] After one or more rounds of adjustments, when the error rate falls within a preset reasonable range (such as 5% to 30%), the inspection plan is confirmed as a usable plan.

[0079] Finally, the generated inspection plan, along with the target inspection table's domain, table structure characteristics (table name, field names, etc.), data content characteristics (correct data samples), and statistical information of the newly generated rules, is added as new nodes and edges to the heterogeneous graph. In this process, the solution library is allowed to evolve automatically. Simultaneously, the inspection solution is returned to the user for formal data quality checks on the target checklist.

[0080] Through the above implementation methods, even without matching any existing solutions, the system can automatically generate new, parameter-adaptive inspection solutions based on historical knowledge graphs and deep learning models, thereby significantly reducing the burden of manually configuring rules and achieving coverage of unknown data tables.

[0081] In one embodiment of the present invention, based on step S4, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0082] S401. Candidate Solution Push and User Interaction In one specific implementation, after completing multi-dimensional matching, one or more candidate inspection schemes are obtained. These candidate schemes are then pushed to the user interface in list form, displaying summary information such as the name, applicable field, comprehensive matching score, and number of rules included in each scheme.

[0083] Users can issue one of the following commands through the interactive interface: First instruction (single selection): The user clicks to select a candidate solution from the list, directly setting that solution as the final target inspection solution, and then performs data quality checks.

[0084] Second instruction (rule-level fusion): The user selects two or more candidate schemes (e.g., scheme...). and Then click the "Smart Fusion" button to enter the following rule-level fusion processing flow, which generates a fused inspection plan as the target inspection plan.

[0085] S402. Check Intent Recognition When a user triggers the fusion process, the inspection intent of the target checklist is first determined. The intent is selected from four predefined categories: Compliance, Consistency, Accuracy, and Integrity.

[0086] Methods for obtaining intent include: Explicit Input: The user interface provides text input boxes where users can describe the inspection target in natural language, such as "Please focus on checking the format compliance of financial documents." This natural language input is denoted as... The data is fed into a pre-trained intent classification model (such as a BERT fine-tuned model). The model outputs the probability distribution of the intent category:

[0087] Select the category with the highest probability as the inspection intent. .

[0088] Implicit Inference: If the user does not provide a natural language description, the intent is inferred from the metadata of the target checklist (table name, field comments, and business module) through keyword mapping or a lightweight classifier. For example, if the table name contains "material master data" and multiple field comments contain "code" and "format", the intent defaults to compliance.

[0089] S403. Scheme Decomposition and Rule Node Representation Each candidate inspection scheme selected by the user Decomposed into a set of rule nodes Each rule node It should contain at least the following metadata: Related target fields ; Rule Type ; Rule parameters (such as regular expression strings, value ranges) (null value tolerance threshold); Execution Dependencies (For example express Must (Previously executed).

[0090] S404. Intent Filtering Based on Data Fingerprint and Vector Similarity In order to remove rule nodes that do not match the inspection intent, this embodiment adopts a soft filtering method that does not require manual rule labeling and AST parsing, and is based entirely on the similarity between the historical error data of the rules and the semantics of the intent.

[0091] S404.1 Constructing the data fingerprint vector of the rule For each rule in the set of rule nodes It traces back its historical execution records to obtain erroneous data samples detected by the rule. If the number of erroneous samples... (For example If the rule is not specified, it will be skipped (it will be retained by default or marked as pending manual confirmation).

[0092] When there are sufficient erroneous samples, features are extracted and normalized to generate data fingerprint vectors. Specifically, the processing is based on the data type of the fields associated with the rules: Text field: string for each error sample ,calculate:

[0093]

[0094]

[0095] The mean of the feature vectors of all erroneous samples is obtained. Then, perform max-min normalization on each dimension.

[0096] Numeric field: For each incorrect value ,calculate:

[0097] Take the mean of the absolute values ​​of the Z-scores of all incorrect samples. negative proportion Degree of deviation from the mean (e.g.) ), combined into And normalize.

[0098] Final Rules The data fingerprint vector is denoted as .

[0099] S404.2 Intent Vectorization By using preset mapping rules, the inspection intent is mapped to predefined standard intent description text. For example: Compliance Intent → "Check the data format, value range, and required fields" Consistency Intent → "Check cross-table joins and business logic consistency" Accuracy Intent → "Check for numerical fluctuations, outliers, and statistical characteristics" Integrity Intent → "Check null value rate and missing data" The standard description text is referred to as Convert it into an intent vector using a pre-trained semantic encoding model (such as Sentence-BERT, BGE):

[0100] S404.3 Vector Space Alignment because (Statistical space) and Since the semantic spaces are located in different vector spaces, their similarity cannot be directly calculated. The system projects both into the same joint vector space through a trainable linear transformation (or a pre-aligned projection matrix). :

[0101] in , , This is the bias term. The projection matrix can be pre-trained on historical data through contrastive learning (e.g., making the matching rule-intent pairs highly similar).

[0102] S404.4 Similarity Calculation and Soft Filtering Calculate the cosine similarity between the aligned rule vector and the intent vector:

[0103] Set the first threshold (e.g., 0.8) and the second threshold (For example, 0.5). The filtering logic is as follows: like Then retain the rules. ; like Then the elimination rule ; like Then, the Large Language Model (LLM) is invoked for auxiliary decision-making. Specifically, prompt words are constructed: Rule code: <SQL / script or natural language description of rule r>; Standard intent: <Standard intent description text T_intent_std>; Please determine whether the rule meets the intent, and output only "Keep" or "Remove".

[0104] Get the output of LLM; if it says "Keep", keep it; otherwise discard it.

[0105] Through the above steps, a filtered set of rule nodes is obtained. .

[0106] S405. Conflict Detection and Weighted Voting Resolution The filtered rule nodes may come from different candidate schemes, and similar rules targeting the same field may conflict. For example, scheme The requirement is that the value range of the `status` field is {0, 1}, and the solution... The requirement is {Y, N}.

[0107] Iterate through all fields, for each field Collect a subset of the rules that apply to it. For the same rule type (e.g., value range) If multiple rule instances have conflicting parameters, a weighted voting process will be performed: Each rule node Voting weight Defined as:

[0108] in It is a rule The overall matching score between the candidate solution and the target checklist (from S204). It is the frequency of occurrence of this rule type in similar historical schemes. This is the balance coefficient (e.g., 0.6).

[0109] For the conflict rules in this field, calculate the value of each candidate parameter. Weighted score:

[0110] The parameter value with the highest score is selected as the rule parameter after fusion. If the highest scores are the same, the priority is determined by a preset priority (e.g., the user-specified scheme takes precedence) or by manual intervention.

[0111] After conflict resolution, at most one rule is retained for each rule type of each field.

[0112] S406. Dependency Reconstruction and Execution Flow Generation The rule nodes after conflict resolution retain their original execution dependencies (the rule chains from the original solution). Using all rule nodes as vertices and dependencies as directed edges, a directed graph is constructed. Circular dependencies are detected and removed (e.g., through topological sorting), ultimately resulting in a directed acyclic graph (DAG).

[0113] in .

[0114] This DAG represents the execution flow of the fusion inspection scheme. It can be further output as a linear sequence according to topological order for the execution engine to call.

[0115] S407. Parameter Adaptive Correction For each rule in the post-fusion inspection scheme Instead of directly reusing the original scheme's parameters, the rule parameters are modified based on correct data samples independent of the target checklist. Sources of correct data samples include: Check the correct data samples corresponding to the solution that are pre-stored in the solution library; User-specified standard reference data (such as historically validated correct datasets).

[0116] Suppose the rules in the correct sample The associated target field is Extract the correct sample set for this field. ( (Number of samples). Parameter fitting is performed according to the rule type: S407.1 Regular Expression Validation Analyze the common patterns of the samples. For example, if the sample values ​​are "USR_001", "USR_002", and "USR_123", a regular expression can be generated using the longest common substring or pattern mining algorithms. One implementation involves extracting the longest common prefix and suffix of all samples and identifying the format of variable parts (such as the number of digits). The generated regular expression is:

[0117] in The length is determined by the maximum length of the numerical portion in the sample. This is used to finalize the update rule parameters. .

[0118] S407.2 Value Range Verification Calculate the minimum and maximum values ​​of the correct samples:

[0119] Correct the value range parameter to Or expand outwards according to a preset tolerance level. (For example ),get For integer fields, further rounding is possible.

[0120] S407.3 Verification of numerical fluctuations or outliers Calculate the mean of the correct samples and standard deviation :

[0121] Set the outlier detection threshold to:

[0122] in This is a configurable parameter, with a default value of 3 (corresponding to the 3-sigma criterion). The system uses this threshold as a rule parameter.

[0123] S407.4 Null Value Rate Check The proportion of missing values ​​in the correct sample:

[0124] The null tolerance threshold is adjusted as follows:

[0125] in This is a preset buffer value (e.g., 0.05). This allows the proportion of null values ​​in the new data to be slightly higher than the proportion in the correct samples.

[0126] S407.5 Other Rule Types For cross-table consistency checks, foreign key relationships of related fields can be extracted from the correct samples; for enumeration value checks, the set of unique values ​​appearing in the correct samples is taken as the allowed enumeration values.

[0127] The corrected parameters Update the corresponding rule node to generate a fusion inspection scheme with adaptive parameters.

[0128] S408. Outputs and Applications Finally, the output of the fused inspection scheme includes: a set of rule nodes (filtered, resolved, and parameter corrected); and an execution flow in the form of a directed acyclic graph.

[0129] Once the user confirms, a formal data quality check can be performed on the target checklist based on the fusion solution.

[0130] Through the above specific implementation methods, the present invention realizes the generation of inspection schemes based on user intent and intelligent fusion of multiple schemes. It does not require manual definition of rule labels or parsing of complex logic, and makes full use of historical error data fingerprints and semantic vector matching, and has a high degree of automation and scalability.

[0131] In one embodiment of the present invention, based on step S5, a possible embodiment will be given below, and its specific implementation will be described in a non-limiting manner.

[0132] S501. Constructing a rectification memory map In one specific implementation, a rectification memory map is pre-constructed. This serves as a knowledge base for historical rectification experiences. Based on the original "table-field-rule heterogeneous graph," this graph expands upon it with two new node types: Error pattern nodes: These are used to abstractly describe a specific type of data error, such as "null value error," "format mismatch error," "value range out of bounds error," and "cross-table inconsistency error." Each error pattern node contains an error type identifier and a brief description.

[0133] Rectification Action Node: This describes an executable rectification operation, such as "fill with default values", "upstream system data entry", "regular expression cleaning", "manual review", etc. Each rectification action node can be associated with a specific SQL script, Python code snippet, or manual operation instructions.

[0134] The nodes are defined with the following edge relationships: Rule Node Error pattern node: Indicates that a certain inspection rule will detect this type of error during execution.

[0135] Error mode node Rectification Action Node: Represents the most frequently used rectification action for this type of error in history, and can be weighted (frequency).

[0136] Rectification Action Nodes Historical rectification records: Each historical rectification record is a specific instance, containing metadata such as execution time, executor, rectification script, and effect evaluation.

[0137] The graph is stored using a graph database (such as Neo4j) or an in-memory graph structure, supporting efficient nearest neighbor retrieval and path traversal.

[0138] S502. Error Context Coding When the target checklist After executing the matching (or fusion) generated inspection plan, the specific inspection results are obtained: a set of erroneous data records. For each type of error (e.g., a formatting error in a field), multi-dimensional context encoding is performed to form a comprehensive error context vector. .

[0139] S502.1 Error Feature Vectorization For a certain type of error, let its corresponding set of error data samples be . ( (This represents the number of erroneous records; if it exceeds the preset sampling limit, random sampling will be used.) Extract the following statistical features: Error type: represented using one-hot encoding or embedding, such as a vector corresponding to null errors. .

[0140] Metadata of the error field: field name, field type (obtained as a vector after semantic embedding) ).

[0141] Distribution characteristics of erroneous data: For text-based errors, calculate the average length, the proportion of special characters, and the mean of character entropy; For numerical errors, calculate Z. The mean of the absolute scores, the proportion of negative scores, and the deviation from the normal range. These features are then normalized and concatenated. .

[0142] The final error feature vector is:

[0143] S502.2 Intent Injection The inspection intents (compliance / consistency / accuracy / completeness) obtained from the scheme synthesis phase are used as strong constraints. The intent categories are converted into an intent embedding vector. (This can be obtained through pre-training or lookup tables).

[0144] S502.3 Error Fingerprint Generation Lightweight sampling is performed on the erroneous data itself to generate an error fingerprint. For example, for format errors, the prefixes, suffixes, and alphanumeric patterns of all erroneous values ​​are extracted to obtain a pattern set. This pattern set is then hashed or serialized to serve as the fingerprint. .

[0145] The overall context vector is:

[0146] S503. RAG-based retrieval and re-ranking use In rectifying memory maps A two-stage retrieval process is conducted.

[0147] S503.1 Preliminary Recall First, based on the current error type (e.g., "null value error"), all potentially related historical rectification action nodes are found using the edges "rule → error pattern" and "error pattern → rectification action" in the graph. Then, all corresponding historical rectification records are recalled using the "rectification action → source edge" to obtain an initial candidate set. Its size is (For example, 100 items).

[0148] S503.2 Fine sorting right Each historical rectification record Extract its scene feature vector This includes: the domain of the original inspection scheme of the record, the inspection intent at the time, and the statistical characteristics of the erroneous data (if stored in the history). Then, the current context vector is calculated. and The cosine similarity is used, and an additional scene matching score is introduced:

[0149] in This is an intent matching indicator (add 0.2 if the intent corresponding to the historical rectification record is the same as the current intent, and add 0.1 if they are compatible). These are preset coefficients. Scores are sorted in descending order, with the top scores taken. (For example, 5 results) as the refined search results .

[0150] S504. LLM-driven rectification scheme synthesis The result of fine sorting Each historical rectification record fragment, along with the detailed context of the current error, is structured into a prompt and sent to the large language model.

[0151] Example of a Prompt structure: You are a senior data governance expert. Please generate a remediation plan for the current data errors based on the following information.

[0152] [Current Task] Target Checklist: Financial Vouchers Error field: invoice_code Error type: Format error (regular expression validation failed) Intended purpose of inspection: Compliance. [Current Error Data Sample] ['123-456', 'ABC-789', 'INVALID', 'FV_001', 'FV_002'] [Refer to historical experience (from similar solutions)] 1. Record A (Material domain, Intent: Compliance): Corrective action: Use the SQL REGEXP_REPLACE function to remove non-alphanumeric characters.

[0153] Script example: UPDATE table SET code = REGEXP_REPLACE(code, '[^A-Z0-9]',''); Results: Success rate reached 92%.

[0154] 2. Record B (Financial domain, Intent: Compliance): Corrective action: The formatting error was found to be due to OCR recognition failure. It is recommended to roll back to the upstream system and retransmit.

[0155] Manual operation: Contact the invoice scanning system administrator to retrieve the original image again.

[0156] [instruction] Please analyze the above information comprehensively and generate a rectification plan that includes an "automatic repair script" and "human review suggestions." If multiple possibilities exist, please specify the applicable conditions for each. Output format: First, provide a confidence score (0-100), then list the "automatic repair plan" and "human intervention guidelines."

[0157] Based on the above prompts, the large language model outputs a structured rectification plan.

[0158] S505. Dual-track output of executable code and suggestions Parsing the LLM output yields the following: Confidence score If the score is below a preset threshold (e.g., 60), the system will mark it as "Manual intervention recommended, repair will not be performed automatically"; otherwise, proceed to the next step.

[0159] Automated repair scripts: The system extracts SQL, Python, or data cleaning rule code from the LLM response. Syntax validation or parameter binding is performed as needed (e.g., replacing example table names with the actual target table names). The generated scripts can be executed directly on the data quality platform.

[0160] Manual intervention guidelines: presented in Markdown or rich text format, including operating steps, precautions, and potential risks.

[0161] In addition, the generated remediation plan (including the error context, referenced history, final generated scripts and guidelines) will be stored back into the remediation memory graph as a new remediation record. Specifically, a new rectification action node is created (if the script mode is new) or an existing node is reused, and weights are increased through edge error patterns and common solutions to achieve continuous evolution of the knowledge base.

[0162] Through the above specific implementation methods, the present invention achieves end-to-end automation from inspection results to "customized rectification solutions". Through the generation capability of large language models, it can combine historical experience from different sources and dynamically write rectification solutions with executable code, which significantly reduces the workload of manually analyzing errors and writing repair code.

[0163] In this embodiment, the data quality inspection system based on inspection scheme matching can be divided into multiple functional modules according to the functions it performs, such as... Figure 4 As shown. The module referred to in this invention is a series of computer program segments that can be executed by at least one processor and perform a fixed function, and is stored in memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0164] Although the present invention has been described in detail with reference to the accompanying drawings and preferred embodiments, the present invention is not limited thereto. Various equivalent modifications or substitutions can be made to the embodiments of the present invention by those skilled in the art without departing from the spirit and essence of the invention, and such modifications or substitutions should all be within the scope of the present invention. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should also be covered within the protection scope of the present invention.

Claims

1. A data quality checking method based on checking scheme matching, characterized in that, include: A pre-built inspection scheme library is constructed, wherein each inspection scheme encapsulates at least: one or more inspection rules, applicable fields, table structure characteristics applicable to the inspection scheme, and data content characteristics; Obtain the target checklist for data quality inspection, extract the metadata and data content information of the target checklist, and determine its domain information based on the data content information; Based on the domain information, metadata information, and data content information of the target checklist, a multi-dimensional match is performed with the checklists in the checklist library to determine one or more candidate checklists. Based on the inspection rules encapsulated in the candidate inspection scheme, an applicable target inspection scheme is determined for the target inspection table, and a data quality inspection is performed on the target inspection table based on the target inspection scheme. Aggregate the historical rectification records pre-associated with each candidate inspection plan, and generate a rectification plan based on the historical rectification records and data quality inspection results.

2. The method of claim 1, wherein, Determining the domain information to which the data belongs based on the data content information includes: Data sampling is performed on the target checklist to obtain sample data for each field; Extract the field name, field annotation, and semantic features of the sample data for each field. Predict the probability value of each field belonging to each predefined domain using a pre-trained text classification model to obtain the domain probability distribution of each field. The field type is determined based on the field's metadata information, and the fields are divided into core business fields and non-core fields; among them, the core business fields include: amount field, quantity field, date field, and business main entity identifier field; The domain probability distributions of all fields in the target check table are summed by weight, wherein the weight values ​​of the core business fields are higher than the weight values ​​of the non-core fields, to obtain the weighted total score for each predefined domain. The region with the highest weighted total score is selected as the candidate region, and it is determined whether the weighted total score of the candidate region is greater than or equal to a preset threshold. If so, then the candidate domain is determined as the domain information of the target checklist; If not, the domain information of the target checklist is marked as a general domain, and a manual confirmation request is output. The manually labeled domain input from the outside is received as the final domain information.

3. The method of claim 1, wherein, Based on the domain information, metadata information, and data content information of the target checklist, multi-dimensional matching is performed with the checklists in the checklist library, including: Based on the domain information of the target checklist, checklists with the same or compatible applicable domains are selected from the checklist library to obtain a first candidate checklist set. Calculate the metadata similarity score between the metadata information of the target check table and the table structure features of each check scheme in the first candidate scheme set; Calculate the content similarity score between the data content information of the target checklist and the data content features of each checklist in the first candidate scheme set; For each inspection scheme in the first candidate scheme set, the metadata similarity score and the content similarity score are weighted and fused to obtain a comprehensive matching score; The inspection scheme with a comprehensive matching score higher than a preset threshold is selected as the matching result of the multi-dimensional matching.

4. The method according to claim 1, characterized in that, If no candidate inspection scheme is matched, the method further includes: Based on the existing inspection scheme library, a heterogeneous graph containing table nodes, field nodes, rule nodes and their relationships is constructed. Structural features, semantic features, and data distribution features are extracted from the target checklist and fused to generate a comprehensive feature vector. In the heterogeneous graph, similar table nodes are retrieved based on the comprehensive feature vector, and the rule nodes connected to the similar table nodes are aggregated using an attention mechanism to predict and generate a rule sequence and execution priority suitable for the target checklist. Based on the actual data samples of the target checklist, generate appropriate rule parameters for the rules in the rule sequence; The generated inspection plan is simulated and executed. The parameters are adjusted according to the error rate. The adjusted inspection plan, the domain, table structure characteristics, and data content characteristics of the target inspection table are added to the heterogeneous graph.

5. The method according to claim 4, characterized in that, An attention mechanism is used to aggregate rule nodes connected to similar table nodes to predict and generate a rule sequence and execution priority suitable for the target check table, including: In the heterogeneous graph, based on the comprehensive feature vector of the target checklist, the vector similarity between it and each table node is calculated, and the Top-K table nodes are selected as similar table nodes according to the similarity from high to low. For each similar table node, all reachable rule nodes are extracted through the paths in the heterogeneous graph, and the frequency of each rule node appearing in the inspection schemes associated with the Top-K similar table nodes and the co-occurrence frequency of any two rule nodes appearing in the same inspection scheme are recorded. The comprehensive feature vector of the target check table is used as the query vector, and the comprehensive feature vector of each similar table node is used as the key vector. The attention weight of each similar table node to the target check table is calculated through an attention mechanism, wherein the attention weight is positively correlated with the similarity between the query vector and the key vector. The frequency of occurrence of each rule node after being weighted by the attention weight is accumulated to obtain the aggregate score of the rule node; rule nodes with aggregate scores higher than a preset threshold are selected to form a candidate rule set; Based on the co-occurrence frequency of any two rule nodes in the candidate rule set, a rule co-occurrence graph is constructed, where nodes are rules and edge weights are co-occurrence frequencies; using the co-occurrence graph and a pre-set rule execution dependency template, or by analyzing the temporal relationship between rules in existing inspection schemes, the execution dependency order between rules is inferred. Based on the execution dependency order, the rules in the candidate rule set are arranged into a directed acyclic graph or a linear sequence as a check scheme applicable to the target checklist.

6. The method according to claim 1, characterized in that, Based on the inspection rules encapsulated in the candidate inspection schemes, determine the applicable target inspection scheme for the target inspection table, including: Push one or more candidate inspection plans to the user; In response to the user's first instruction, a candidate inspection scheme is selected from the one or more candidate inspection schemes and used as the target inspection scheme; Alternatively, in response to a second instruction from the user, rule-level fusion processing is performed on at least two candidate inspection schemes to generate a fused inspection scheme as the target inspection scheme.

7. The method according to claim 6, characterized in that, At least two candidate inspection schemes are subjected to rule-level fusion processing, including: The inspection intent of the target checklist is determined using natural language recognition technology, and the inspection intent is selected from compliance intent, consistency intent, accuracy intent, or completeness intent. Each candidate inspection scheme is decomposed into a set of rule nodes. The field information, rule type, rule parameters and execution dependencies of each rule node are retained, and mismatched rule nodes are removed according to the inspection intent. Conflicting rules for the same field in the filtered rule nodes are detected, and a weighted voting mechanism is used to resolve the conflicts. The voting weight is determined based on the matching similarity between the candidate scheme of the rule node and the target check table or the frequency of rule occurrence. The resolved rule nodes are reorganized into a directed acyclic graph according to the execution dependency relationship, which serves as the execution flow of the fusion check scheme. The parameters of the rule nodes in the fusion inspection scheme are modified. The parameters of the rule nodes include thresholds, value ranges, or regular expressions.

8. The method according to claim 7, characterized in that, According to the inspection intent, non-matching rule nodes are removed, including: Obtain historical error data samples detected by each rule node in the rule node set; extract the entropy, special character ratio, and average length of text data, or the mean absolute value, negative value ratio, and degree of deviation from the mean of Z-score of numerical data from the samples, and generate data fingerprint vectors after normalization. The inspection intent is mapped to predefined standard intent description text, which is then converted into an intent vector through a semantic encoding model. The data fingerprint vector and intent vector are projected into the same joint vector space through a linear transformation to obtain aligned rule vectors and intent vectors, and the cosine similarity between the aligned rule vectors and intent vectors is calculated. If the similarity is higher than the first threshold, the corresponding rule node is retained; if it is lower than the second threshold, the corresponding rule node is removed. If it is in between, the large language model is invoked, the rule code and standard intent description text are input, and the retention or removal is determined based on the output of the large language model.

9. The method according to claim 7, characterized in that, The parameters of the rule nodes in the fusion inspection scheme are modified, including: For each rule in the fusion inspection scheme, identify the target field that the rule is associated with for verification, and extract reference data for the target field from a pre-set correct data sample; the correct data sample is pre-stored in the inspection scheme library or is specified by the user as standard reference data; Based on the rule type, the corresponding parameter fitting strategy is adopted: If the rule type is regular expression validation, then analyze the common pattern of the reference data and automatically generate a regular expression that matches the common pattern; If the rule type is value range verification, then the minimum and maximum values ​​of the reference data are counted, and the value range parameter is corrected to the range between the minimum and maximum values, or expanded outward according to the preset tolerance. If the rule type is numerical fluctuation or outlier verification, then the mean and standard deviation of the reference data are calculated, and the threshold parameter is corrected to the mean ± N times the standard deviation, where N is a configurable parameter; If the rule type is null value rate verification, then the proportion of null values ​​in the reference data is calculated, and the null value tolerance threshold is corrected to the proportion of null values ​​plus a preset buffer value. The corrected parameters are updated to the corresponding rule nodes to generate a fusion inspection scheme with adaptive parameters.

10. A data quality inspection system based on inspection scheme matching, characterized in that, include: The pre-storage module is used to pre-build an inspection scheme library, wherein each inspection scheme encapsulates at least: one or more inspection rules, applicable fields, table structure features applicable to the inspection scheme, and data content features; The target parsing module is used to obtain the target checklist to be checked for data quality, extract the metadata information and data content information of the target checklist, and determine the domain information to which it belongs based on the data content information. The scheme matching module is used to perform multi-dimensional matching with the inspection schemes in the inspection scheme library based on the domain information, metadata information and data content information of the target inspection table, so as to determine one or more candidate inspection schemes. The scheme execution module is used to determine the applicable target inspection scheme for the target inspection table according to the inspection rules encapsulated in the candidate inspection scheme, and to perform data quality inspection on the target inspection table based on the target inspection scheme; The rectification assistance module is used to aggregate the historical rectification records pre-associated with each candidate inspection plan, and generate rectification plans based on the historical rectification records and data quality inspection results.

Images