Data lake data quality control method and system based on data rule verification

Abstract

The invention relates to a data lake data quality control method and system based on data rule verification. The method comprises the steps that business and technical documents are analyzed through a large language model, and a data verification rule is automatically generated and stored; executing the rule and collecting a result to a result table; the execution result is compared with a preset reference value, and abnormity is recognized and registered; upstream traceability and downstream influence analysis are carried out on the anomaly based on the metadata blood relationship, and a standardized influence report is generated by using the large language model again; and finally, generating a specific rectification scheme by utilizing the large language model for the third time based on the report. The system comprises corresponding function modules. According to the method, through combination of the large language model and the data blood relationship, full-process automatic and intelligent closed-loop management and control from rule generation, problem discovery, root positioning, influence analysis to scheme suggestion are realized, and the efficiency and accuracy of data quality management are remarkably improved.

Description

Technical Field

[0001] This invention relates to data governance and data quality control technologies. Specifically, it provides a data lake data quality control method and system based on data rule verification. Background Technology

[0002] In the wave of digitalization, data has become a key production factor for the banking industry. With the continuous development of commercial banking operations, the total number of its business systems continues to increase, and outdated systems are frequently upgraded, leading to an exponential growth in data volume and increasingly complex interactions between systems and data. This complexity poses a significant challenge to the stable operation and data quality of data lakes.

[0003] Data quality issues permeate the entire lifecycle of data, from its entry into the data lake to its processing, use, and management. For example, if upstream data contains erroneous data due to logical flaws during processing, this erroneous data can spread downstream, severely impacting downstream data users. Quickly locating the root cause of such data problems, accurately identifying their scope and severity, and generating effective solutions accordingly are the main challenges currently facing the field of data quality control.

[0004] In existing technologies, metadata lineage is typically used to trace the scope of impact of erroneous data. However, this method still has significant shortcomings: First, the creation of data validation rules is highly dependent on human experience, which is inefficient and prone to omissions; second, after data anomalies are discovered, locating the root cause of the problem and assessing the degree of impact still requires a large amount of manual analysis and judgment, resulting in slow response times and high requirements for the professional skills of analysts; finally, there is a lack of automated rectification plan generation mechanisms, making it difficult to form a closed-loop management of data quality issues from "discovery to analysis to processing".

[0005] Therefore, there is an urgent need in this field for a data quality control method and system that can automatically generate verification rules, intelligently analyze the impact of problems, and automatically generate rectification plans, so as to improve the efficiency and accuracy of data quality control. Summary of the Invention

[0006] The purpose of this invention is to overcome the aforementioned deficiencies of the prior art and provide a data lake data quality control method and system based on data rule verification. This invention aims to achieve automatic generation of data verification rules, accurate analysis of problem impacts, and generation of rectification plan suggestions through automation and intelligent technologies, thereby constructing an efficient, accurate, and closed-loop data quality control system.

[0007] The technical solution to achieve the objective of this invention is: a data lake data quality control method based on data rule verification, the method comprising the following steps:

[0008] S1: By parsing business knowledge documents and technical specification documents through a large language model, and combining the preset rule templates with the metadata information of the target data table, data validation rules are automatically generated and the rule information is stored in the rule configuration table;

[0009] S2: Execute the rule logic in the rule configuration table through the scheduler, obtain the rule execution result, and record the result in the rule execution result table;

[0010] S3: Compare the rule execution result with the rule baseline value preset in the rule baseline value information table to determine whether the result is abnormal, and write the abnormal information into the rule result abnormal registration table;

[0011] S4: Based on the metadata lineage, trace the abnormal records upstream to the problem data source and infer the downstream impact scope. Then, use the large language model again to summarize the lineage analysis results, generate an impact report containing the impact type and impact degree, and store it in the impact analysis result table.

[0012] S5: Based on the aforementioned impact report, a targeted data rectification plan is generated for the third time using a large language model.

[0013] Furthermore, in step S1, the metadata information includes the database name, table name, and field name; the generated data validation rule includes the rule name, rule type, rule logic, rule description, and its associated target table and fields.

[0014] Furthermore, in step S3, the rule baseline value information table pre-sets a baseline value for each rule type, and determines whether the data is abnormal by comparing the rule execution result with the baseline value.

[0015] Furthermore, in step S4, metadata lineage analysis is performed based on the field-level lineage table; the upstream tracing is to query the source information based on the target table and target field in the abnormal record; the downstream reasoning is to query the information that the source is depended on by the downstream based on the target table and target field in the abnormal record.

[0016] Furthermore, in step S4, when generating the impact report using the large language model, the input includes the problem table, problem field, affected table, and affected field information obtained from the lineage analysis, and the output is an impact report in a standard format, the content of which includes the type and degree of impact.

[0017] Furthermore, in step S5, the key elements in the impact analysis results table are used as prompt words and input into the large language model, which then outputs a description of specific rectification measures.

[0018] This invention also proposes a data lake data quality control system based on data rule verification, the system comprising:

[0019] The rule generation module is used to parse business knowledge documents and technical specification documents through a large language model, combine preset rule templates and prompt words, automatically generate data verification rules, and store the rule information in the rule configuration table;

[0020] The rule result acquisition module is used to execute the verification rule through the scheduler and collect the execution result, and store the result in the rule execution result table;

[0021] The rule result analysis module is used to compare the rule execution result with the preset rule benchmark value. If the execution result does not match the benchmark value, it is judged as an anomaly, and the anomaly record and related information are written into the rule result anomaly registration form.

[0022] The impact report generation module is used to trace the problem data source upstream and infer the scope of impact downstream based on the problem table or field pointed to by the anomaly rule, based on the metadata lineage. It also uses a large language model to summarize the analysis results, generate a standardized impact report, and store it in the impact analysis results table.

[0023] The rectification plan generation module is used to generate detailed data problem rectification descriptions and handling plans based on the impact report in the impact analysis results table using a large language model.

[0024] Furthermore, the data verification rules generated by the rule generation module include rule name, rule type, rule SQL logic, rule description, target database name, target table name, and target field name.

[0025] Furthermore, the rule result analysis module obtains the benchmark value corresponding to the rule type by querying the rule benchmark value information table, and compares the execution result in the rule execution result table with the benchmark value.

[0026] Furthermore, the impact report generation module obtains the upstream problem data source and downstream impact scope by querying the field-level lineage relationship table, and inputs the analyzed lineage path information into the large language model to generate a standardized impact report.

[0027] Compared with the prior art, the significant advantages and beneficial effects of this invention are as follows:

[0028] 1. By automatically parsing documents and generating validation rules through a large language model, the workload of manually writing rules is greatly reduced, and the rule coverage and creation efficiency are improved.

[0029] 2. By combining data lineage and rule verification results, the root cause of data problems (upstream data source) can be accurately located and its downstream impact can be assessed, which changes the ambiguity and lag of traditional manual investigation.

[0030] 3. By using large language models to generate impact reports and specific rectification plans, data engineers are provided with clear handling guidelines, which lowers the professional technical threshold for problem handling and shortens the decision-making time.

[0031] 4. This invention achieves full automation and intelligence from rule generation, problem discovery, root cause location, impact analysis, and solution generation, forming a complete closed loop for data quality control and significantly improving the maturity and effectiveness of data governance.

[0032] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0033] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0034] Figure 1 This is a schematic diagram of the data verification rule generation process in this invention.

[0035] Figure 2 This is a schematic diagram of the data verification rule execution and result collection process in this invention.

[0036] Figure 3 This is a schematic diagram of the rule result analysis process in this invention.

[0037] Figure 4 This is a schematic diagram of the impact analysis process in this invention. Detailed Implementation

[0038] It is readily understood that, based on the technical solution of this invention, various embodiments of the invention can be conceived by those skilled in the art without altering the essential spirit of the invention. Therefore, the following detailed embodiments and accompanying drawings are merely illustrative examples of the technical solution of this invention and should not be considered as the entirety of the invention or as limitations or restrictions on the technical solution of this invention. Rather, these embodiments are provided to enable those skilled in the art to gain a more thorough understanding of the invention. Preferred embodiments of the invention are described below in conjunction with the accompanying drawings, which form part of this application and, together with the embodiments of the invention, serve to illustrate the innovative concept of the invention.

[0039] The data lake data quality control system based on data rule verification proposed in this invention includes the following core modules:

[0040] Rule Generation Module: This module is used to automatically generate data validation rules. It uses a large language model to parse business knowledge documents and technical specification documents, and combines this with preset rule templates and prompts to generate formatted data validation rules for specific data tables or fields in the data lake. The rule information is then stored in the rule configuration table.

[0041] Rule Result Acquisition Module: This module executes the verification rules and collects the execution results. It uses a scheduler (such as a Python script) to batch execute the SQL statements of the verification rules generated by the rule generation module, obtains the execution results, and stores them in the rule execution result table.

[0042] Rule Result Analysis Module: This module is used to determine whether the rule execution result is abnormal. It compares the rule execution result with a preset rule baseline value. If the execution result does not match the baseline value, the rule execution result is determined to be abnormal, and the abnormality record and related information are written to the Rule Result Abnormality Registration Form.

[0043] Impact Report Generation Module: This module generates impact analysis reports based on the results of anomaly rules. Based on metadata lineage, it traces upstream the data source of the problem from the issue table or field pointed to by the anomaly rule, infers the scope of impact downstream, and summarizes the analysis results using a large language model to generate a standardized impact report, which is then stored in the impact analysis results table.

[0044] The rectification plan generation module analyzes the affected table or field data in the impact report related to the bloodline chain to derive specific rectification plans. This module inputs the structured information from the impact analysis results table into a large language model, and, guided by preset prompts, generates detailed and actionable descriptions and solutions for data-related issues.

[0045] This invention also proposes a data lake data quality control method based on data rule verification, wherein the data lake data quality control system based on data rule verification is used to execute the method. The core of this method lies in constructing an automated and intelligent data quality control closed loop, the process of which is shown in the figure below, and mainly includes the following steps:

[0046] S1: Automated rule generation

[0047] By parsing the input business knowledge documents and technical specification documents using a large language model, and combining them with preset rule template prompts and metadata information of the target data table, formatted data validation rules are generated for specific data tables or fields in the data lake, and the rule information is stored in the rule configuration table.

[0048] The metadata information includes, but is not limited to, database name, table name, and field name.

[0049] The generated data validation rules include the rule name, rule type, rule logic, rule description, and associated target tables and fields, and this information is stored in the rule configuration table.

[0050] The key table structure information of the rule configuration table (RULE_INFO) is as follows:

[0051] field name Data types Notes RULE_ID VARCHAR The unique number of the rule RULE_NAME VARCHAR Rule Chinese Name RULE_TYPE VARCHAR Rule Type RULE_SQL VARCHAR Rule Logic RULE_DESC VARCHAR Rule Description SCHEMA_NAME VARCHAR Database name to be verified TARGET_TABLE VARCHAR Target table for verification TARGET_COLUMN VARCHAR Validation field name RULE_STATUS VARCHAR Rule Status CREATE_TIME DATETIME Rule creation date and time CREATOR VARCHAR Creator ID

[0052] S2: Rule Result Collection

[0053] The rule logic in the rule configuration table is executed periodically or triggered in batches using a scheduler (such as a Python script). This involves executing SQL statements for data validation rules and obtaining the results after rule execution. The output results, execution status, execution time, and executor information of the rule execution are recorded in the rule execution result table (RULE_EXECUTE_RESULT_INFO).

[0054] The key table structure information of the rule execution result table (RULE_EXECUTE_RESULT_INFO) is as follows:

[0055]

[0056]

[0057] S3: Rule Result Analysis

[0058] The collected rule execution results are compared with predefined rule baseline values ​​to determine if there are any anomalies. A baseline value is preset for each rule type, and this baseline value is stored in the rule baseline value information table (RULE_BASE_VALUE_INFO). If the rule execution result is inconsistent with the baseline value, the rule execution result is determined to be abnormal. The records determined to be abnormal, along with their complete context information, are written to the rule result anomaly registration table (RULE_RESULT_ABNORMAL_INFO).

[0059] The key structure information of the rule baseline value information table (RULE_BASE_VALUE_INFO) is as follows:

[0060] field name Data types Notes RULE_BASE_ID VARCHAR Unique reference value number RULE_ID VARCHAR Foreign key, a unique identifier for a relationship rule. RULE_TYPE VARCHAR Rule Type BASE_VALUE VARCHAR benchmark value CREATE_TIME TIMESTAMP Creation time CREATOR VARCHAR Creator

[0061] Rule Result Anomaly Registration Form (RULE_RESULT_ABNORMAL_INFO)

[0062] field name Data types Notes ABNORMAL_ID VARCHAR The unique identifier of the abnormal result RULE_ID VARCHAR Foreign key, a unique identifier for a relationship rule. RESULT_ID VARCHAR Foreign key, associated with a unique identifier of the execution result. RULE_NAME VARCHAR Rule Chinese Name RULE_TYPE VARCHAR Rule Type RULE_SQL VARCHAR Rule Logic RULE_DESC VARCHAR Rule Description EXECUTE_RESULT VARCHAR Rule execution output BASE_VALUE VARCHAR benchmark value TARGET_SCHEMA VARCHAR Target library for verification TARGET_TABLE VARCHAR Target table for verification TARGET_COLUMN VARCHAR Validated fields CREATE_TIME TIMESTAMP Creation time CREATOR VARCHAR Creator

[0063] S4: Generation of Impact Reports

[0064] For each abnormal record in the rule result anomaly registration table, an impact analysis is performed based on metadata lineage, including:

[0065] Upstream analysis: Based on field-level lineage, trace back to the source table and source field that generated the problematic data to pinpoint the root cause of the problem.

[0066] Downstream analysis: Based on field-level lineage, deduce all downstream tables and fields affected by this abnormal data to determine the scope of the impact.

[0067] The field-level bloodline relationship is stored in the field-level bloodline relationship table (COLUMN_LINEAGE_INFO).

[0068] The key table structure information for the field-level lineage relationship table (COLUMN_LINEAGE_INFO) is as follows:

[0069] field name Data types Notes LINEAGE_ID VARCHAR Unique blood relation number SCHEMA_NAME VARCHAR The database name where the table is located TABLE_NAME VARCHAR Table name SERIAL_NO INT Field serial number COLUMN_NAME VARCHAR field name COLUMN_MAPPING VARCHAR Field processing logic SOURCE_SCHEMA_NAME VARCHAR Database name where the source table is located SOURCE_TABLE_NAME VARCHAR Source table name SOURCE_COLUMN_NAME VARCHAR Source field name CREATE_TIME TIMESTAMP Creation time CREATOR VARCHAR Creator

[0070] The kinship path information obtained from the analysis is input into a large language model. Natural language processing technology is used to generate a standard-format impact report that includes the type, degree, and description of the impact. The report content is then stored in a structured manner in the impact analysis results table (IMPACT_RESULT_ANALYSIS_INFO).

[0071] The impact analysis results table (IMPACT_RESULT_ANALYSIS_INFO) has the following key table structure information:

[0072]

[0073]

[0074] S5: Rectification Plan Generation

[0075] Based on the impact report recorded in the impact analysis results table, the affected table or field data on the lineage link are analyzed, and the large language model generates a targeted data rectification plan guided by preset prompt words.

[0076] Key elements from the impact analysis results are used as prompt words and input into the large language model.

[0077] The large language model outputs specific and actionable descriptions of rectification measures, which are recorded in the impact analysis results table or the associated rectification plan table, thereby completing the closed-loop management from problem discovery to disposal recommendations.

[0078] Through steps S1 to S5, this method achieves automatic generation of data quality rules, automatic collection and analysis of verification results, automatic tracking and assessment of the impact of problems, and intelligent suggestions for rectification plans, ultimately forming an efficient, accurate, and closed-loop data lake data quality control system.

[0079] Example: Taking "Primary Key Uniqueness Verification in Model Tables" as an example

[0080] This embodiment demonstrates how the present invention performs uniqueness verification on the primary key CUSTOMER_ID of a model table named FDM_T01_CUSTOMER_INFO, and analyzes and generates a solution after an anomaly is detected.

[0081] Step 1: Automated rule generation

[0082] Suppose that the large language model in the generation module receives the requirement from the data lake architecture specification document that "the primary key of the model table must be unique", and combines it with table information obtained from the metadata system (library: FDM, table:

[0083] FDM_T01_CUSTOMER_INFO, primary key: CUSTOMER_ID). Based on the preset "uniqueness check" rule template and hint words, the large model generates the corresponding SQL validation statement. For example,

[0084] Given the database: FDM, table: FDM_T01_CUSTOMER_INFO, primary key: CUSTOMER_ID, and the WHERE condition restricts ETL_DATE to '{TX_DATE}', generate a count of the number of uniqueness verification results for this field, outputting only the number of duplicate records and removing comments.

[0085] The large language model generates the following SQL rules and stores them in the rule configuration table RULE_INFO.

[0086]

[0087] Step 2, Rule Result Collection

[0088] The rule result collection module schedules and executes the above SQL statement via a Python script. The script writes the execution results (e.g., the number of duplicate primary keys found is 5), execution status (SUCCESS), and other information into the rule execution result table RULE_EXECUTE_RESULT_INFO.

[0089] Step 3, Rule Result Analysis

[0090] The rule result analysis module performs a database query and compares the rule execution results with the rule baseline value information table.

[0091] The baseline value in RULE_BASE_VALUE_INFO is compared. For "primary key uniqueness check", the baseline value is usually set to 0 (indicating that no duplicates are expected).

[0092] SELECT

[0093] T1.RESULT_ID,

[0094] T1.RULE_ID,

[0095] T3.RULE_NAME,

[0096] T3.RULE_SQL,

[0097] T3.RULE_DESC,

[0098] T1.RULE_TYPE,

[0099] T1.EXECUTE_RESULT,

[0100] T2.BASE_VALUE

[0101] FROM RULE_EXECUTE_RESULT_INFO T1

[0102] LEFT JOIN RULE_BASE_VALUE_INFO T2 ON T1.RULE_ID=T2.RULE_ID

[0103] LEFT JOIN RULE_INFO T3 ON T1.RULE_ID=T3.RULE_ID

[0104] WHERE T1.EXECUTE_STATUS='SUCCESS'

[0105] AND T1.EXECUTE_RESULT<>T2.BASE_VALUE -- Analyze the execution records of successfully executed rules.

[0106] Since the result of this execution is 5, which is not equal to the baseline value of 0, the system determines that the execution result of this rule is abnormal.

[0107] Subsequently, the system integrates the rule information, execution result, baseline value, and other related information of this anomaly and writes them into the rule result anomaly registration table RULE_RESULT_ABNORMAL_INFO.

[0108] Step 4, Impact Analysis

[0109] The impact report generation module reads exception records from the RULE_RESULT_ABNORMAL_INFO table, obtaining the library name, table name, and field name (FDM library, FDM_T01_CUSTOMER_INFO table, CUSTOMER_ID field) pointed to by the exception results. It then performs a related query on the COLUMN_LINEAGE_INFO table to conduct lineage analysis, obtaining the upstream problem data source and the downstream impact scope. Specifically:

[0110] Problem Data Source Locator

[0111] SELECT

[0112] T1.RULE_ID, -- Rule number

[0113] T1.RESULT_ID, -- Unique ID of the execution result

[0114] T2.SERIAL_NO, -- Field sequence number

[0115] T1.TARGET_SCHEMA, -- The database name where the problem table is located.

[0116] T1.TARGET_TABLE, -- Problem table name

[0117] T1.TARGET_COLUMN, -- Question field name

[0118] T1.COLUMN_MAPPING, -- Field processing logic

[0119] T2.SOURCE_SCHEMA, -- The database name where the affected table is located.

[0120] T2.SOURCE_TABLE, -- Affected table name

[0121] T2.SOURCE_COLUMN -- Affected field name

[0122] FROM RULE_RESULT_ABNORMAL_INFO T1

[0123] LEFT JOIN COLUMN_LINEAGE_INFO T2

[0124] ON T1.TARGET_SHCEMA=T2.SCHEMA_NAME

[0125] AND T1.TARGET_TABLE=T2.TABLE_NAME

[0126] AND T1.TARGET_COLUMN=T2.COLUMN_NAME

[0127] Through the upstream analysis above, we traced the data source of the CUSTOMER_ID field. We discovered it originates from the CUS_NO field of the SDM_CUSTOMER_INFO table in the SDM database, with the processing logic being T1.CUS_NO.

[0128] --Downstream Impact Analysis

[0129] SELECT

[0130] T1.RULE_ID, -- Rule number

[0131] T1.RESULT_ID, -- Unique ID of the execution result

[0132] T2.SERIAL_NO, -- Field sequence number

[0133] T1.TARGET_SCHEMA, -- The database name where the problem table is located.

[0134] T1.TARGET_TABLE, -- Problem table name

[0135] T1.TARGET_COLUMN, -- Question field name

[0136] T1.COLUMN_MAPPING, -- Field processing logic

[0137] T2.SOURCE_SCHEMA, -- The database name where the affected table is located.

[0138] T2.SOURCE_TABLE, -- Affected table name

[0139] T2.SOURCE_COLUMN -- Affected field name

[0140] FROM RULE_RESULT_ABNORMAL_INFO T1

[0141] LEFT JOIN COLUMN_LINEAGE_INFO T2

[0142] ON T1.TARGET_SHCEMA=T2.SOURCE_SCHEMA_NAME

[0143] AND T1.TARGET_TABLE=T2.SOURCE_TABLE_NAME

[0144] AND T1.TARGET_COLUMN=T2.SOURCE_COLUMN_NAME

[0145] Based on the downstream analysis above, we searched for downstream tables that depend on the CUSTOMER_ID field. We found that the CUST_ID field in the ADM_T01_CUSTOMER_INFO table of the ADM library depends on it.

[0146] The above SQL statements are used to obtain the rule number, execution result unique number, database name of the problem table, problem table name, problem field name, field processing logic, database name of the affected table, source table name, and source field name, and an Excel document is generated.

[0147] Step 5: Generate a formatted impact report and remediation plan.

[0148] These structured kinship analysis results were compiled and then input into the large language model along with large model prompts (e.g., "Perform impact analysis on FDM_T01_CUSTOMER_INFO... Output in the following format...").

[0149] For example, prompt word input:

[0150] An impact analysis was performed on FDM_T01_CUSTOMER_INFO, analyzing the problem source upstream and the impact downstream, and the output was in the following format:

[0151] Rule number, unique execution result number, database name of the problem table, problem table name, problem field name, field processing logic, database name of the affected table, source table name, source field name, impact type (upstream data source or downstream impact), impact level (table-level impact or field-level impact), impact description (within 50 characters), rectification description (detailed handling plan within 100 characters).

[0152] The output of the large model is as follows:

[0153] 1001,100101,FDM,FDM_T01_CUSTOMER_INFO,CUST_ID,T1.CUS_NO,SDM,SDM_CUSTOMER_INFO,CUS_NO,Upstream data source,Field-level impact,Upstream CUS_NO null values ​​or incorrect format directly pollute CUST_ID,Establish SDM_CUS_NO uniqueness verification and null value interception,Abnormal data is not entered into FDM and an alarm is triggered;

[0154] 1001,100102,ADM,ADM_T01_CUSTOMER_INFO,CUST_ID,T1.CUST_ID,FDM,FD M_T01_CUSTOMER_INFO,CUST_ID,Downstream impact,Field-level impact,The abnormal CUST_ID of FDM is passed down, causing the primary key of the ADM customer table to become invalid. Add a CUST_ID validity check at the FDM layer. Illegal values ​​will not be distributed to downstream ADM and the batch acquisition for the day will be rolled back.

[0155] After parsing this output, the system stores it in a structured manner in the impact analysis results table.

[0156] The IMPACT_RESULT_ANALYSIS_INFO command completes the entire process.

[0157] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

[0158] It should be understood that, in order to simplify the present invention and help those skilled in the art understand its various aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes described in a single embodiment or with reference to a single figure. However, the present invention should not be construed as including all features in the exemplary embodiments as essential technical features of the claims of this patent.

[0159] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0160] It should be understood that the modules, units, components, etc., included in the device of one embodiment of the present invention can be adaptively changed to be placed in a device different from that embodiment. Different modules, units, or components included in the device of the embodiment can be combined into a single module, unit, or component, or they can be divided into multiple sub-modules, sub-units, or sub-components.

[0161] The modules, units, or components in the embodiments of the present invention can be implemented in hardware, in software running on one or more processors, or in a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement the embodiments of the present invention. The present invention can also be implemented on a computer program product or computer-readable medium for performing some or all of the methods described herein.

Claims

1. A data lake data quality control method based on data rule validation, characterized in that, The method includes the following steps: S1: By parsing business knowledge documents and technical specification documents through a large language model, and combining the preset rule templates with the metadata information of the target data table, data validation rules are automatically generated and the rule information is stored in the rule configuration table; S2: Execute the rule logic in the rule configuration table through the scheduler, obtain the rule execution result, and record the result in the rule execution result table; S3: Compare the rule execution result with the rule baseline value preset in the rule baseline value information table to determine whether the result is abnormal, and write the abnormal information into the rule result abnormal registration table; S4: Based on the metadata lineage, trace the abnormal records upstream to the problem data source and infer the downstream impact scope. Then, use the large language model again to summarize the lineage analysis results, generate an impact report containing the impact type and impact degree, and store it in the impact analysis result table. S5: Based on the aforementioned impact report, a targeted data rectification plan is generated for the third time using a large language model.

2. The method according to claim 1, characterized in that, In step S1, the metadata information includes the database name, table name, and field name; the generated data validation rules include the rule name, rule type, rule logic, rule description, and the associated target table and fields.

3. The method according to claim 1, characterized in that, In step S3, the rule baseline value information table pre-sets a baseline value for each rule type, and the data is judged to be abnormal by comparing the rule execution result with the baseline value.

4. The method according to claim 1, characterized in that, In step S4, metadata lineage analysis is performed based on the field-level lineage table; the upstream tracing is based on the target table and target field in the abnormal record to query its source information; The downstream reasoning involves querying information that is relied upon by downstream entities as a source, based on the target table and target fields in the abnormal records.

5. The method according to claim 1, characterized in that, In step S4, when generating an impact report using a large language model, the input includes a problem table, problem fields, an affected table, and affected field information obtained from lineage analysis. The output is an impact report in a standard format, and the report content includes the type and degree of impact.

6. The method according to claim 1, characterized in that, In step S5, the key elements in the impact analysis results table are used as prompt words and input into the large language model, which then outputs a description of specific rectification measures.

7. A data lake data quality control system based on data rule validation, characterized in that, The system includes: The rule generation module is used to parse business knowledge documents and technical specification documents through a large language model, combine preset rule templates and prompt words, automatically generate data verification rules, and store the rule information in the rule configuration table; The rule result acquisition module is used to execute the verification rule through the scheduler and collect the execution result, and store the result in the rule execution result table; The rule result analysis module is used to compare the rule execution result with the preset rule benchmark value. If the execution result does not match the benchmark value, it is judged as an anomaly, and the anomaly record and related information are written into the rule result anomaly registration form. The impact report generation module is used to trace the problem data source upstream and infer the scope of impact downstream based on the problem table or field pointed to by the anomaly rule, based on the metadata lineage. It also uses a large language model to summarize the analysis results, generate a standardized impact report, and store it in the impact analysis results table. The rectification plan generation module is used to generate detailed data problem rectification descriptions and handling plans based on the impact report in the impact analysis results table using a large language model.

8. The system according to claim 7, characterized in that, The data verification rules generated by the rule generation module include rule name, rule type, rule SQL logic, rule description, target database name, target table name, and target field name.

9. The system according to claim 7, characterized in that, The rule result analysis module obtains the benchmark value corresponding to the rule type by querying the rule benchmark value information table, and compares the execution result in the rule execution result table with the benchmark value.

10. The system according to claim 7, characterized in that, The impact report generation module obtains the upstream problem data source and downstream impact scope by querying the field-level lineage relationship table, and inputs the analyzed lineage path information into the large language model to generate a standardized impact report.

Images