Dynamic data quality rule intelligent generation and self-adaptive correction system

Abstract

The invention provides a dynamic data quality rule intelligent generation and self-adaptive correction system, which belongs to the field of data governance and data management, and comprises four modules: a quality demand acquisition module, a rule intelligent generation module, a rule execution monitoring module and a self-adaptive optimization module. By utilizing strong natural language understanding, code generation and logical reasoning capabilities of a large model, intelligent generation, dynamic evaluation and self-adaptive correction of a data quality rule are realized, a data quality control closed loop capable of self-evolution and continuous optimization is formed, and the efficiency and accuracy of data management are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data governance and data management, and particularly relates to a dynamic data quality rule intelligent generation and self-adaptive correction system. BACKGROUND

[0002] In data governance and data management, data quality is a core link, which is directly related to the accuracy and reliability of data analysis, machine learning and business decision-making. Traditional data quality management systems rely heavily on data engineers or data governance experts to manually write detection rules. This process has significant pain points: 1. High dependence on expert experience: Rule creation requires deep business understanding and technical implementation capabilities, with high labor costs and low efficiency.

[0003] 2. Difficulty in dealing with complex business logic: For non-explicit business rules hidden in data (such as "VIP customer discount rate should not be lower than 20%"), traditional rules based on pattern matching or simple statistics are difficult to discover and define, and are prone to omissions.

[0004] 3. Static and rigid: Once the rules are deployed, they usually remain unchanged for a long time. When the business changes (such as marketing activities leading to a temporary surge in transaction volume) or the data source structure changes, the original rules will produce a large number of false positives or false negatives, requiring manual intervention for analysis and adjustment, with high maintenance costs.

[0005] 4. Difficulty in cross-system consistency checking: Traditional tools are usually limited to a single data source or known physical foreign key association, and lack the ability to actively discover and verify deep semantic consistency between different systems.

[0006] Most of the existing technologies are traditional data quality tools based on templates or rule engines (such as commercial Informatica Data Quality and open source Great Expectations). These tools provide a framework for defining and executing data quality rules, but their core workflow is still manually driven: Users need to know the specific form of quality problems in advance.

[0007] Users configure rules through graphical methods or directly write code in the tool interface.

[0008] After rule execution, users need to manually analyze the results to determine whether the problem is with the data or the rule, and manually adjust the rule parameters or logic.

[0009] Although these tools have achieved "automated execution" of quality checks, they have not achieved automation and intelligence in the "generation" and "optimization" of rules. Their intelligence is limited, they cannot understand business semantics, and they do not have the ability to evolve. SUMMARY

[0010] To solve the above technical problems, the present application provides a dynamic data quality rule intelligent generation and adaptive correction system, which overcomes the defects of relying on manual and static rigidity in the prior art, realizes intelligent generation, dynamic evaluation and adaptive correction of data quality rules by using the powerful natural language understanding, code generation and logical reasoning ability of a large model, forms a data quality management and control closed loop that can evolve and continuously optimize, and significantly improves the efficiency and accuracy of data governance.

[0011] The technical solution of the present application is: A dynamic data quality rule intelligent generation and adaptive correction system, comprising A quality requirement acquisition module that receives and acquires data quality requirements described in natural language by a user through interactive dialogue of a large model, and is responsible for pushing potential quality rule suggestions discovered to the user; A rule intelligent generation module that understands the natural language quality detection requirements by using a large model, and generates executable data quality rules in combination with metadata and data samples; A rule execution monitoring module that uses a large model to combine detection resource data update frequency and usage frequency, intelligently formulates detection task priority and scheduling strategy, executes the data quality rules and records execution results and logs; An adaptive optimization module for analyzing the execution results and logs, diagnosing rule defects by a large model, and generating correction instructions to optimize the data quality rules; at the same time, actively scanning and analyzing global metadata, and analyzing and reasoning out potential quality rules by using a large model.

[0012] Further, The quality requirement acquisition module is specifically configured to collect data quality requirements input by a user, guide the user to confirm and audit quality rules in a dialogue manner, and support selection of data quality rules recommended by a system from an interface.

[0013] Further, The rule intelligent generation module is specifically configured to input the natural language requirements, metadata information of related tables and data samples as prompt words to a large model, output executable codes conforming to specific data quality tool syntax specifications by the large model, and save the executable codes to a rule library.

[0014] The rule execution monitoring module is specifically configured to use a large model to combine detection resource data update frequency and usage frequency, intelligently formulate detection task priority and scheduling strategy, execute data quality detection rules in the rule library, and record quality detection execution results to form quality monitoring logs.

[0015] The adaptive optimization module is specifically used for dynamic evaluation and adaptive correction of quality rule effectiveness: continuously monitoring the generated quality monitoring log, when the false positive rate or false negative rate of the rule exceeds the threshold, triggering the large model to analyze the false positive or false negative data samples, inferring the rule invalidation reason and outputting the optimization scheme, the optimization scheme including adjusting the rule parameters or reconstructing the rule logic, the optimization scheme can be directly pushed to the rule library for automatic deployment and update for adaptive correction, or can be pushed to the user side as a correction suggestion for manual confirmation and audit before update and correction.

[0016] Further, The adaptive optimization module is specifically used for active discovery and recommendation of quality rules: actively scanning and analyzing the global metadata, using a large model to identify potential semantic associations between different tables and different systems, and recommending and displaying the discovered rules in natural language.

[0017] Further, The rule intelligent generation process includes: 1) The user describes the data quality requirements in a colloquial way through the quality requirement collection module; 2) The large model in the rule intelligent generation module receives the natural language instruction; 3) The large model calls and analyzes the relevant metadata to understand the context of the data; 4) The large model compiles the natural language requirements into executable quality rule code or an equivalent SQL query script based on its learned programming knowledge and quality rule patterns; 5) The generated rule code is confirmed by the user or automatically deployed to the rule library.

[0018] The rule execution and effect monitoring process includes: 1) The rule execution monitoring module actively scans the quality rules in the rule library and actively obtains the data update frequency and usage frequency information of the detected resources from the data platform; 2) The rule execution monitoring module uses a large model to intelligently analyze the quality rules and the relevant information of the detected resources, outputs a reasonable quality detection task scheduling strategy, and calls the rule execution engine to execute the detection task; 3) The module records the execution results of each rule, including but not limited to: the amount of error data triggered, error examples, rule execution time, historical triggering trends, etc., and stores them in the monitoring log.

[0019] The rule effectiveness dynamic evaluation and adaptive correction process includes: 1) The adaptive optimization module periodically analyzes the monitoring log and evaluates the effectiveness of the rules through statistics and pattern recognition; 2) The large model analyzes the false positive root cause according to the original definition of the rule, the false positive data samples generated in the past week (the specific time interval can be specified by the user), and the business change information, and generates a correction suggestion or a corrected rule; 3) The correction suggestion is updated after being reviewed by the user, or in the scenario where the rule historical false positive rate is lower than the predetermined threshold and the business involved in the rule is a non-core process, the update is automatically deployed.

[0020] The quality rule active discovery and recommendation process includes: 1) The global metadata is actively scanned and analyzed, and the large model is used to identify potential semantic associations between different tables and different systems; 2) The potential business rules are inferred by analyzing the historical quality detection data; 3) Then the quality requirement collection module recommends this potential consistency rule to the user, and after the user confirms, it is transferred to the rule intelligent generation module to complete rule generation.

[0021] The beneficial effects of the present application are: Significantly reduce the technical threshold and labor cost: business personnel can create complex data quality rules without coding, freeing data engineers from tedious rule writing and maintenance work.

[0022] Improve rule coverage and accuracy: through semantic understanding, rules can be discovered and generated for complex and hidden business logic, reducing the blind area of quality inspection. Through adaptive correction, the rule can dynamically adapt to business changes, continuously maintain high accuracy, and reduce false positives and omissions.

[0023] Active and intelligent data quality management: from passive response to active discovery, the system can automatically recommend cross-system consistency rules and actively adjust the rules, so that the data quality management system has the ability to "self-evolve" and responds faster.

[0024] Enhance enterprise data reliability: through comprehensive, accurate and dynamic quality supervision, the overall quality and credibility of enterprise data are fundamentally improved, providing a solid guarantee for upper-level business decision-making.

[0025] The system uses the powerful natural language understanding, code generation and logical reasoning capabilities of the large model to realize intelligent generation, dynamic evaluation and adaptive correction of data quality rules, forming a data quality management and control closed loop that can self-evolve and continuously optimize, significantly improving the efficiency and accuracy of data governance. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 is the overall architecture schematic diagram of the present application; Figure 2It is a working process schematic diagram of the present application. DETAILED DESCRIPTION

[0027] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0028] The application discloses a dynamic data quality rule intelligent generation and adaptive correction system based on a large model, and a core architecture mainly includes four modules. The quality requirement acquisition module receives and acquires data quality requirements described in natural language by a user in an interactive dialogue form of a large model, and is responsible for pushing potential quality rule suggestions found to the user.

[0029] The rule intelligent generation module understands the natural language quality detection requirement by using a large model, and generates executable data quality rules in combination with metadata and data samples.

[0030] The rule execution monitoring module updates the frequency, usage frequency of detection resources in combination with a large model, intelligently formulates a detection task priority and a scheduling strategy, executes the data quality rules, and records execution results and logs.

[0031] The adaptive optimization module is used for analyzing the execution results and logs, diagnosing rule defects by a large model, and generating correction instructions to optimize the data quality rules; meanwhile, the module actively scans and analyzes global metadata, analyzes and infers potential quality rules by using a large model.

[0032] (1) Semantic-driven rule intelligent generation 1) A user (a data governance officer or a business personnel) describes data quality requirements in a colloquial way (for example: "check the age field in the customer table, is there unreasonable data less than 18 years old or more than 100 years old?") through the quality requirement acquisition module.

[0033] 2) The large model in the rule intelligent generation module receives the natural language instruction.

[0034] 3) The large model calls and analyzes related metadata (for example, the `customer` table has an `age` field, which is an integer type) and data samples (for example, the value of the `age` field is roughly distributed) to understand the context of the data.

[0035] 4) The large model compiles the natural language requirements into executable quality rule code based on its learned programming knowledge and quality rule patterns (e.g. generate an Expectation for Great Expectations: `expect_column_values_to_be_between(column="age", min_value=18, max_value=100)` or an equivalent SQL query script: select * from customer where age<18 or age>100).

[0036] 5) The generated rule code is confirmed by the user or automatically deployed to the rule library.

[0037] (2) Rule execution and effect monitoring 1) The rule execution monitoring module actively scans the quality rules in the rule library and actively obtains the data update frequency, usage frequency and other related information of the detected resources from the data platform.

[0038] 2) The rule execution monitoring module uses the large model to intelligently analyze the quality rules and the relevant information of the detected resources, outputs reasonable quality detection task scheduling strategies, and calls the rule execution engine to execute the detection tasks.

[0039] 3) The module records the execution results of each rule, including but not limited to: the amount of error data triggered, error examples, rule execution time, historical triggering trends, etc., and stores them in the monitoring log.

[0040] (3) Dynamic evaluation and adaptive correction of rule effectiveness 1) The adaptive optimization module periodically analyzes the monitoring log and evaluates the effectiveness of the rules through statistics and pattern recognition. For example, it is found that the false positive rate of a rule to "identify unusually high transaction amounts" has recently increased sharply.

[0041] 2) The module calls the large model, inputs the original definition of the rule, a large number of false positive data samples generated recently, and possible business change information (such as: there is a promotion activity announcement recently).

[0042] 3) The large model analyzes the root cause of the false positives (infer: due to the promotion activity, transaction amounts have generally increased, and the original threshold is no longer applicable), and generates correction suggestions or corrected rules (e.g. "suggest adjusting the abnormal threshold from 3 times the standard deviation of the average value to 5 times the standard deviation" or directly generate new rule code).

[0043] 4) The correction suggestion can be updated by the user after review or automatically deployed by the system in low-risk scenarios, thereby realizing self-optimization of the rules.

[0044] (4) Cross-data-source consistency rule discovery and recommendation 1) The adaptive optimization module actively scans and analyzes the global metadata, and uses the large model to identify potential semantic associations between different tables and different systems (for example: the `vip_type` field in the `crm.customer` table and the `orders.discount` field).

[0045] 2) By analyzing historical quality detection data, the large model can infer potential business rules (for example: "Customers with `vip_type` as 'Premium' should have `discount` greater than or equal to 20%").

[0046] 3) The large model then recommends this potential consistency rule to the user through the quality requirement collection module (for example: "System found a potential rule: Premium VIP customers' discount rate should not be lower than 20%, do you need to generate and enable this check rule?"), after user confirmation, it goes to step (1) to complete rule generation.

[0047] The above is only a preferred embodiment of the present application, which is used to illustrate the technical solutions of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application is included in the protection scope of the present application.

Claims

1. A dynamic data quality rule intelligent generation and adaptive correction system, characterized in that, include: The quality requirements collection module uses a large model interactive dialogue format to receive data quality requirements described by users in natural language, and is responsible for pushing potential quality rule suggestions to users. The rule intelligent generation module uses a large model to understand the data quality requirements described in natural language, and combines metadata and data samples to generate executable data quality rules. The rule execution monitoring module uses a large model combined with the update frequency and usage frequency of detection resource data to intelligently formulate detection task priorities and scheduling strategies, execute the data quality rules, and record the execution results and logs. The adaptive optimization module is used to analyze the execution results and logs, diagnose rule defects through a large model, and generate correction instructions to optimize the data quality rules; at the same time, it actively scans and analyzes the full-domain metadata and uses the large model to analyze and infer potential quality rules.

2. The system according to claim 1, characterized in that, The quality requirement collection module is specifically used to: collect data quality requirements input by users, guide users to confirm and review quality rules in a dialog-based manner, and support the selection of system-recommended data quality rules from the interface.

3. The system according to claim 1, characterized in that, The rule intelligent generation module is specifically used to: input the data quality requirements described in natural language, the metadata information of the relevant tables, and the data samples as prompt words into the large model, and the large model outputs executable code that conforms to the syntax specifications of specific data quality tools and saves it to the rule base.

4. The system according to claim 3, characterized in that, The rule execution monitoring module is specifically used to: intelligently formulate detection task priorities and scheduling strategies by combining the large model with the update frequency and usage frequency of detection resource data, execute data quality detection rules in the rule base, and record the quality detection execution results to form a quality monitoring log.

5. The system according to claim 4, characterized in that, The adaptive optimization module is specifically used for dynamic evaluation and adaptive correction of the effectiveness of quality rules: continuously monitoring the generated quality monitoring logs, when the false alarm rate or false negative rate of a rule exceeds the threshold, triggering a large model to analyze the false alarm or false negative data samples, inferring the cause of rule failure and outputting an optimization plan. The optimization plan includes adjusting rule parameters or reconstructing rule logic. The optimization plan can be directly pushed to the rule base for automatic deployment and update for adaptive correction, or it can be pushed to the user side as a correction suggestion for manual confirmation and review before being updated and corrected.

6. The system according to claim 1, characterized in that, The adaptive optimization module is specifically used for proactive discovery and recommendation of quality rules: it proactively scans and analyzes the metadata of the entire domain, uses a large model to identify potential semantic relationships between different tables and systems, and then converts the discovered rules into natural language for recommendation and display.

7. The system according to claim 3, characterized in that, The rule intelligent generation process includes: 1) Users describe their data quality requirements in a conversational manner through the quality requirements collection module; 2) The large model in the rule intelligent generation module receives the data quality requirement instruction; 3) The large model calls and analyzes relevant metadata to understand the context of the data; 4) Based on the programming knowledge and quality rule patterns it has learned, the large model compiles natural language requirements into executable quality rule code or an equivalent SQL query script. 5) The generated rule code is confirmed by the user or automatically deployed to the rule base.

8. The system according to claim 4, characterized in that, The rule enforcement and effect monitoring process includes: 1) The rule execution monitoring module actively scans the quality rules in the rule base and actively obtains relevant information such as the data update frequency and usage frequency of the monitored resources from the data platform; 2) The rule execution monitoring module uses a large model to intelligently analyze the information related to quality rules and detection resources, outputs a reasonable quality detection task scheduling strategy, and calls the rule execution engine to execute the detection task; 3) The module records the execution results of each rule, including: the amount of error data triggered, error examples, rule execution time, historical triggering trends, and stores them in the monitoring log.

9. The system according to claim 5, characterized in that, The process of dynamic evaluation and adaptive correction of rule effectiveness includes: 1) The adaptive optimization module periodically analyzes monitoring logs and evaluates the effectiveness of rules through statistics and pattern recognition; 2) The large model analyzes the root causes of false alarms based on the original definition of the rule, the false alarm data samples generated in the past week, and the business change information, and generates correction suggestions or corrected rules. 3) The revised suggestions will be updated after user review, or the rule will be automatically deployed and updated in scenarios where the historical false alarm rate of the rule is lower than the predetermined threshold and the business involved in the rule is a non-core process.

10. The system according to claim 6, characterized in that, The proactive discovery and recommendation process for quality rules includes: 1) Proactively scan and analyze the entire domain's metadata, and use large models to identify potential semantic relationships between different tables and systems; 2) By analyzing historical quality inspection data, potential business rules can be deduced; 3) Subsequently, the quality requirements collection module recommends this potential consistency rule to the user. After the user confirms it, the system will transfer to the rule intelligent generation module to complete the rule generation.

Images