Automated quality audit method for heterogeneous data governance

Abstract

The present application relates to the technical field of data quality management, and specifically relates to an automatic quality auditing method for heterogeneous data management, which realizes unified processing of structured and unstructured data by constructing a task pool, processing verification rules, and generating a scheduling task, selects an object to be audited, constructs a task pool, processes verification rules, generates a scheduling task, takes an auditing object from the task pool, requests data from a related service according to the type of the scheduling task, requests data from a heterogeneous data service if the data type is structured data, requests data from an unstructured data service if the data type is unstructured data, verifies the scheduling task and generates a verification report, outputs the verification report, and realizes unified processing of structured and unstructured data through the collaborative work of a heterogeneous data processing verification engine and an unstructured data processing verification engine, thereby expanding the coverage of data quality auditing.

Description

Technical Field

[0001] This invention relates to the field of data quality management technology, specifically to an automated quality auditing method for heterogeneous data governance. Background Technology

[0002] With the advent of the big data era, enterprises and organizations have accumulated a large amount of heterogeneous data, which is scattered across different systems and has different structures and formats. Data quality directly affects the accuracy of enterprise decision-making and the efficiency of business operations, making data quality governance particularly important.

[0003] Existing technologies are primarily designed for single-type data sources, lacking a unified architecture capable of handling both structured and unstructured data simultaneously. This leads to fragmented data processing and hinders integrated governance of heterogeneous data. Traditional methods typically employ fixed batch processing, lacking flexible and efficient task pools and intelligent scheduling mechanisms. They cannot dynamically schedule tasks based on priority and dependencies, resulting in low processing efficiency. Existing technologies generally only support full execution, failing to select appropriate execution frequencies based on data update characteristics and business needs, such as incremental or periodic execution, leading to wasted system resources and reduced processing efficiency. When processing unstructured data, existing technologies lack effective semantic analysis and extraction techniques, failing to transform unstructured data into a verifiable structured form, resulting in high costs and low efficiency for manual auditing.

[0004] Existing methods lack an effective SQL abstraction and parsing engine, failing to flexibly convert advanced verification rules into concrete, executable SQL queries, thus limiting the system's ability to handle complex verification rules. Traditional methods often employ single-level inspection strategies, lacking multi-level quality control strategies and rule systems, failing to achieve comprehensive and refined data quality verification, and affecting the accuracy and comprehensiveness of verification results. Existing verification reports are typically simplistic in content and format, lacking comprehensive, multi-dimensional quality analysis and trend visualization, making it difficult to support users in conducting in-depth data quality analysis and decision-making.

[0005] Currently, there is a lack of an automated quality auditing method that can simultaneously process structured and unstructured data based on a dual-engine parallel architecture, support multiple execution frequencies, have an intelligent task scheduling mechanism, implement multi-level quality inspection strategies, and automatically generate comprehensive verification reports. Summary of the Invention

[0006] The purpose of this invention is to provide an automated quality auditing method for heterogeneous data governance, aiming to solve the technical problems in the prior art, such as the inability to efficiently process structured and unstructured data simultaneously, the lack of intelligent task scheduling mechanisms, and the lack of support for multiple execution frequencies.

[0007] This invention proposes an automated quality auditing method for heterogeneous data governance, including:

[0008] Step 1: Select the object to be audited;

[0009] Step 2: Build a task pool, process verification rules, and generate scheduling tasks;

[0010] Step 3: Retrieve an audit object from the task pool;

[0011] Step 4: Request data from relevant services according to the type of the scheduled task. If the data type is structured, request data from the heterogeneous data service; if the data type is unstructured, request data from the unstructured data service.

[0012] Step 5: Verify the scheduled task and generate a verification report;

[0013] Step 6: Output the verification report.

[0014] Preferably, in step 1, the object type includes a library, a table, a domain, and a file.

[0015] Preferably, step 2, which involves building a task pool, processing verification rules, and generating scheduling tasks, includes:

[0016] Step 2.1: Construct a task pool. The scheduling system receives scheduling tasks issued by the rule engine of data governance, creates a configuration table for scheduling tasks in the database, and saves each scheduling task as a record in the configuration table. Each scheduling task is a rule executed after verifying one or more objects. The scheduling task consists of a scheduling verification rule table and a table of objects to be verified. The scheduling verification rule table contains the ID of the verification rule and the scheduling execution frequency of the verification. The object table contains the ID of the object to be verified and the ID of the corresponding verification rule.

[0017] Step 2.2, Processing verification rules: The scheduling system extracts each core task according to the configuration table of the scheduling tasks and assigns it to different verification rule processing engines according to the type of the scheduling task. The verification rule processing engine reads the task and generates a scheduling task instance. The verification rule processing engine includes a heterogeneous data processing verification rule engine and an unstructured data processing verification rule engine. The execution frequency of the scheduling task includes full execution, incremental execution, and periodic execution. Incremental execution is used to perform periodic incremental processing on the data of the object being verified. Only newly added data is verified and a new verification report is generated and output.

[0018] Step 2.3: The task scheduling module generates scheduling tasks according to the execution frequency of the scheduled tasks. The heterogeneous data verification rule processing engine creates multiple scheduling tasks to be processed according to the database and database table dimensions. When each scheduling task is created, the scheduling task is generated according to the execution frequency of the scheduling task.

[0019] Preferably, in step 4, if the scheduled task is heterogeneous data, data is requested from the heterogeneous data service, including: starting the heterogeneous data processing verification engine; the heterogeneous data processing verification engine generates a query task based on the name of the database table and the SQL query statement in the verification rules; the heterogeneous data processing task obtains the query data of the scheduled task based on the name of the object being verified and the verification rules; the corresponding data is queried from the corresponding database according to the query task; and the queried data is processed and output to the heterogeneous data verification engine.

[0020] Preferably, when generating a query task, the following are also included:

[0021] Step 10: After the heterogeneous data processing verification engine obtains the scheduled task, it determines whether the name of the object to be verified and the verification rules are consistent based on the name of the object to be verified and the verification rules. If they are consistent, step 20 is executed; otherwise, the process ends.

[0022] Step 20: Determine whether the object to be verified has been verified in the previous execution cycle. If the task has been executed, proceed to step 30; otherwise, proceed to steps 40-60.

[0023] Step 30: Determine whether the frequency of the scheduled task execution is full execution;

[0024] Step 40: Request all data of the query object from the data resource service;

[0025] Step 50: Determine whether the task has obtained all the data. If all the data has been obtained, execute the heterogeneous data structured data verification engine. If no data has been obtained, it means that the object did not have corresponding data in the heterogeneous data service before. Then execute step 60 and do not execute the task.

[0026] Step 60: Filter the query results, keeping only the results obtained during the last verification.

[0027] Step 70: Execute the structured data verification engine for heterogeneous data. After execution, proceed to step 80.

[0028] Step 80: Insert the results generated in this cycle of verification into the audit result data table;

[0029] Step 90: Clear the verification records for the object being verified.

[0030] Preferably, in step 4, if the scheduled task is unstructured data, data is requested from the unstructured data service, including: starting the unstructured data processing and verification engine; the unstructured data processing and verification engine requests information from the structured data service based on the object name and the verification rules of the object; requests information on the query object from the data resource service; obtains the query results; and calls the semantic analysis and semantic extraction services based on the query results to transform the data into structured data.

[0031] Preferably, the unstructured data information includes: document content, web page text, natural language text, images, and videos.

[0032] Preferably, step 5 also includes:

[0033] Step 110: The data resource service receives query requests for structured data and pushes query information to the scheduling task;

[0034] Step 120: Based on the data query results obtained from the structured data, start the structured data verification engine for heterogeneous data to verify each table to be verified.

[0035] Step 130: The structured data verification engine for heterogeneous data performs quality inspection analysis based on the verification rules and data sets of each table.

[0036] Step 140: Write the quality problem results into the database;

[0037] Step 150: Clear the data from the last verification.

[0038] Preferably, in step 130, the quality inspection analysis includes:

[0039] Step 131: Start quality verification. The quality verification process iterates through all the strategies that need to be verified in each table and starts the quality inspection strategy verification engine.

[0040] Step 132: After starting the quality inspection strategy verification engine, first perform strategy verification parameter verification. If the verification passes, perform rule verification and parameter verification on the verification rules, and then start the rule verification engine for quality inspection analysis. If the verification fails, do not start the rule verification engine.

[0041] Step 133: Start the rule validation engine, obtain the data table and rules of the policy and the specific parameters of the rules, and then parse the policy into multiple executable SQL statements according to the SQL abstract syntax analysis engine.

[0042] Step 134: Execute the strategy, execute SQL statements to query data, generate the verification results of the rules corresponding to the strategy, and insert the verification result data of the rules into the verification result table;

[0043] Step 135: Perform the above policy verification process on all rules until all rules have completed quality inspection analysis or all policy verifications have been completed, then end this policy detection.

[0044] Preferably, in step 134, the verification results include: duplicate values ​​of the field, field uniqueness, field format, field length, field value type, field value range, illegal character values, illegal decimal places, garbled Chinese characters, missing data, and dirty data.

[0045] This invention employs a dual-engine parallel processing architecture, designing dedicated processing and verification engines for both structured and unstructured data to achieve unified processing of heterogeneous data. It improves system processing efficiency by constructing a task pool and an intelligent scheduling mechanism; supports three execution frequencies—full execution, incremental execution, and periodic execution—to optimize system resource utilization; utilizes semantic analysis technology to process unstructured data, expanding the system's applicability; converts advanced rules into specific SQL query statements through an SQL abstract syntax analysis engine; implements a multi-level quality inspection strategy and rule system; and provides a complete verification process and report generation mechanism to achieve comprehensive data quality control.

[0046] The present invention has the following beneficial effects:

[0047] 1. Dual-engine collaborative processing of heterogeneous data: By working together with the heterogeneous data processing verification engine and the unstructured data processing verification engine, unified processing of structured and unstructured data is achieved, solving the problem of the lack of heterogeneous data processing architecture in existing technologies and expanding the coverage of data quality audit.

[0048] 2. Intelligent scheduling of task pool improves efficiency: Through task pool and priority-based intelligent scheduling mechanism, efficient allocation and execution of tasks are achieved. Taking into account the dependencies between tasks, the problem of imperfect task scheduling mechanism in traditional methods is solved, and the system processing efficiency is significantly improved.

[0049] 3. Multi-frequency execution optimizes resource utilization: It supports three execution frequencies: full execution, incremental execution, and periodic execution. In particular, the incremental execution mode only processes newly added data, which solves the problem of single execution frequency in existing technologies, significantly reduces system resource consumption, and optimizes resource utilization efficiency.

[0050] 4. Semantic analysis expands the scope of application: By processing unstructured data through semantic analysis and semantic extraction services, the system solves the problem of weak unstructured data processing capabilities in traditional methods, expands the scope of application of the system, and can process various unstructured data such as file content, web page text, natural language text, images, and videos.

[0051] 5. Enhanced Flexibility through SQL Abstract Syntax Analysis: By converting advanced verification rules into specific SQL query statements through the SQL abstract syntax analysis engine, the limited verification rule conversion capability in existing technologies is solved, and the flexibility and adaptability of the system in handling complex verification rules are improved.

[0052] 6. Enhanced comprehensiveness through multi-level quality inspection strategies: By implementing a multi-level quality inspection strategy and rule system, the problem of single-level quality inspection strategies in traditional methods is solved. This enables comprehensive and accurate detection of various data quality issues, improving the accuracy and comprehensiveness of verification results.

[0053] 7. Comprehensive verification reports support in-depth analysis: Through a complete verification process and a multi-dimensional report generation mechanism, it solves the problem of simple verification report generation mechanisms in existing technologies, realizes comprehensive monitoring and management of data quality, and provides comprehensive reports including overall quality status, indicators of various dimensions, problem details and trend analysis, supporting users to conduct in-depth data quality analysis and decision-making. Attached Figure Description

[0054] Figure 1 This is a flowchart of the automated quality auditing method for heterogeneous data governance according to the present invention.

[0055] Figure 2 The flowchart for constructing a task pool, processing verification rules, and generating scheduling tasks in this invention;

[0056] Figure 3 This is a flowchart of the heterogeneous data processing verification engine of the present invention.

[0057] Figure 4 This is a flowchart of the unstructured data processing verification engine of the present invention;

[0058] Figure 5 This is a flowchart of the quality verification process of the present invention;

[0059] Figure 6 This is a flowchart of the quality inspection and analysis process of the present invention. Detailed Implementation

[0060] Please refer to Figure 1 - Figure 6 The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0061] like Figure 1 As shown, this invention provides an automated quality auditing method for heterogeneous data governance, comprising the following steps:

[0062] Step 1: Select the object to be audited.

[0063] In a preferred embodiment of the present invention, the types of objects to be audited include databases, tables, domains, files, etc. The system administrator can select the specific objects to be audited through a configuration interface, such as a specific table in a database or a specific file in a file system. These selected objects will serve as the processing targets for subsequent audit tasks.

[0064] Step 2: Build a task pool, process verification rules, and generate scheduling tasks.

[0065] like Figure 2 As shown, this step specifically includes:

[0066] Step 2.1, Construct the Task Pool. The scheduling system receives scheduling tasks from the data governance rule engine and creates a configuration table for the scheduling tasks in the database. Each scheduling task is stored as a record in the configuration table, where each scheduling task represents a rule executed after verifying one or more objects. A scheduling task consists of a scheduling verification rule table and a table of objects to be verified. The scheduling verification rule table contains the ID of the verification rule and the scheduling execution frequency. The object table contains the ID of the object to be verified and the ID of the corresponding verification rule.

[0067] Step 2.2, Processing Verification Rules. The scheduling system extracts each core task from the scheduling task configuration table and assigns it to different verification rule processing engines according to its type. The verification rule processing engine reads the task and generates a scheduling task instance. These engines include a heterogeneous data processing verification rule engine and an unstructured data processing verification rule engine. The execution frequency of this scheduling task includes full execution, incremental execution, and periodic execution. Incremental execution is used to periodically incrementally process the data of the object being verified, verifying only newly added data and generating a new verification report before outputting it.

[0068] In a preferred embodiment of the present invention, the full execution mode is suitable for scenarios where the data quality is verified for the first time or where a comprehensive inspection of historical data is required; the incremental execution mode is suitable for daily data quality monitoring, which only processes newly added or changed data, thus significantly reducing system resource consumption; and the periodic execution mode is suitable for scenarios where data quality verification is performed at fixed time intervals (such as daily, weekly, or monthly).

[0069] Step 2.3: The task scheduling module generates scheduling tasks according to their execution frequency. The heterogeneous data verification rule processing engine creates multiple pending scheduling tasks based on the database and database table dimensions, generating a new scheduling task based on its execution frequency at the time of creation.

[0070] In one specific embodiment, the task scheduling module can set different scheduling strategies based on the execution frequency:

[0071] For tasks that are executed in full, the system will create a scheduled task that is executed only once.

[0072] For incremental execution tasks, the system will set an appropriate execution interval based on the data update frequency, such as once per hour or once per day;

[0073] For tasks that are executed periodically, the system will create scheduled tasks that are executed at set intervals (such as every Monday at 2:00 AM).

[0074] Step 3: Retrieve an audit object from the task pool.

[0075] The task scheduling module selects an audit object from the task pool for processing based on task priority, resource availability, and dependencies. Preferably, the system employs a priority-based task scheduling algorithm to ensure that important tasks are executed first. Furthermore, the system considers dependencies between tasks to ensure that dependent tasks execute after the tasks they depend on.

[0076] Step 4: Request data from relevant services based on the type of the scheduled task. If the data type is structured, request data from the heterogeneous data service; if the data type is unstructured, request data from the unstructured data service.

[0077] like Figure 3 As shown, if the scheduled task involves heterogeneous data, data is requested from the heterogeneous data service, including: starting the heterogeneous data processing verification engine. The heterogeneous data processing verification engine generates a query task based on the database table name and the SQL query statement in the verification rules. Based on the name of the object being verified and the verification rules, the heterogeneous data processing task obtains the query data for the scheduled task. It then queries the corresponding data from the relevant database according to the query task. Finally, the retrieved data is processed and output to the heterogeneous data verification engine.

[0078] In a preferred embodiment, the heterogeneous data processing verification engine can handle various types of structured data sources, including relational databases (such as MySQL, Oracle, SQL Server, etc.), NoSQL databases (such as MongoDB, Cassandra, etc.), and data warehouses (such as Hive, Greenplum, etc.). For different types of data sources, the system uses corresponding connectors and adapters for connection and data acquisition.

[0079] like Figure 3 As shown, generating a query task also includes:

[0080] Step 10: After the heterogeneous data processing verification engine obtains the scheduled task, it determines whether the name of the object to be verified and the verification rules are consistent. If they are consistent, proceed to step 20; otherwise, the process ends.

[0081] In practical applications, the system checks whether the name of the object being verified matches the verification rule, such as checking whether the rule applies to a specified type of data object. For example, if there is a verification rule for a numeric field but it is specified to apply to a text field, the system will determine that it does not match and terminate the process.

[0082] Step 20: Determine whether the object to be verified has already been verified in the previous execution cycle. If the task has been executed, proceed to step 30; otherwise, proceed to steps 40-60.

[0083] The system determines whether the object being verified was verified in the previous execution cycle by querying the historical verification record table. The historical verification record table contains information such as object ID, last verification time, and verification result.

[0084] Step 30: Determine whether the frequency of the scheduled task execution is full execution.

[0085] If the execution frequency of the scheduled task is set to full execution, the system will process all data of the object; if it is not full execution (i.e., incremental execution or periodic execution), the system will adopt the corresponding processing strategy according to the different execution frequencies.

[0086] Step 40: Request all data of the query object from the data resource service.

[0087] For objects being verified for the first time or tasks that are executed at a full frequency, the system needs to request all data of the object from the data resource service. The data resource service is responsible for managing connections to various data sources and data acquisition.

[0088] Step 50: Determine if all data has been retrieved for this task. If all data has been retrieved, execute the heterogeneous data structured data verification engine. If no data has been retrieved, it means that there was no corresponding data for this object in the heterogeneous data service before, then proceed to step 60 and do not execute the task.

[0089] In practical applications, the system will set a timeout threshold (e.g., 30 seconds) and a retry mechanism (e.g., a maximum of 3 retries). If data is not obtained within the timeout period, or if the maximum number of retries is reached without obtaining data, the system will determine that no data has been obtained.

[0090] Step 60: Filter the query results, keeping only the results obtained during the last verification.

[0091] For incremental execution mode, the system will record the timestamp of the last verification and add the time condition to the current query to retrieve only the newly added or changed data.

[0092] Step 70: Execute the structured data verification engine for heterogeneous data. After completion, proceed to step 80.

[0093] The structured data verification engine for heterogeneous data performs quality checks on the data according to verification rules, including checks on multiple dimensions such as field integrity, data format, value range, and data consistency.

[0094] Step 80: Insert the results generated in this cycle of verification into the audit result data table.

[0095] The verification results contain various information, such as object ID, rule ID, verification time, verification result (pass / fail), problem description, and problem severity. This information is structured and stored in the audit results data table for easy subsequent analysis and report generation.

[0096] Step 90: Clear the verification records for the object being verified.

[0097] To prevent historical records from consuming excessive storage space, the system periodically clears expired verification records. Typically, verification records from the most recent period (e.g., 3 months) are retained; records exceeding this retention period are deleted. The clearing strategy can be configured according to actual needs; for example, retention periods can be set for different types of objects.

[0098] like Figure 4 As shown, if the scheduled task involves unstructured data, data is requested from the unstructured data service, including: starting the unstructured data processing and verification engine; the unstructured data processing and verification engine requests information from the structured data service based on the object name and the verification rules for that object; requesting information about the query object from the data resource service; obtaining the query results; and based on the query results, calling the semantic analysis and semantic extraction services to transform the data into structured data.

[0099] In a preferred embodiment of the present invention, unstructured data information includes various types such as document content, webpage text, natural language text, images, and videos. The system employs different processing strategies for different types of unstructured data.

[0100] For text-based data (such as file content, webpage text, and natural language text), the system uses natural language processing technology to extract key information;

[0101] For image data, the system uses image recognition technology to extract image features and content;

[0102] For video data, the system first extracts keyframes, and then performs image processing and content recognition on the keyframes.

[0103] Semantic analysis and extraction services are one of the core innovations of this invention. They can transform unstructured data into a structured form, enabling the application of structured data verification rules for quality checks. Based on machine learning and natural language processing technologies, this service can identify entities, relationships, and semantic information in text, transforming them into a structured data representation.

[0104] Step 5: Verify the scheduled task and generate a verification report.

[0105] like Figure 5 As shown, this step also includes:

[0106] Step 110: The data resource service receives query requests for structured data and pushes query information to the scheduling task.

[0107] The data resource service is responsible for managing connections to various data sources and data acquisition. When a query request is received, the service connects to the corresponding data source based on the request parameters, executes the query operation, and pushes the query results to the scheduling task.

[0108] Step 120: Based on the data query results obtained from the structured data, start the structured data verification engine of the heterogeneous data to verify each table to be verified.

[0109] The heterogeneous data structured data verification engine is a quality verification component specifically designed for structured data. It can perform quality checks on data in various dimensions according to predefined rules.

[0110] Step 130: The structured data verification engine for heterogeneous data performs quality inspection analysis based on the verification rules and data sets of each table.

[0111] like Figure 6 As shown, the quality inspection analysis specifically includes:

[0112] Step 131, Initiate quality verification. Quality verification iterates through all strategies requiring verification in each table. Start the quality inspection strategy verification engine.

[0113] A quality control strategy is a set of related verification rules designed for specific types of data quality issues. For example, a data integrity strategy contains multiple rules to check whether the data is complete, while a data consistency strategy contains multiple rules to check whether the data is consistent.

[0114] Step 132: After starting the quality inspection strategy verification engine, first perform strategy verification parameter verification. If the verification passes, perform rule verification and parameter verification on the verification rules, and then start the rule verification engine for quality inspection analysis. If the verification fails, do not start the rule verification engine.

[0115] Parameter validation is a crucial step in ensuring that validation rules are executed correctly. The system checks whether the rule parameters are complete, conform to the expected type, and are within the valid range. For example, for a rule that checks string length, the parameters should include a minimum length and a maximum length, and must be positive integers.

[0116] Step 133: Start the rule validation engine. After obtaining the data table, rules, and specific parameters of the rules for the strategy, the SQL abstract syntax analysis engine parses the strategy into multiple executable SQL statements.

[0117] The SQL abstract syntax analysis engine is an important component of this invention, which can convert high-level verification rules into specific SQL query statements.

[0118] Step 134: Execute the strategy by executing SQL statements to query data. Generate the verification results for the corresponding rules of the strategy and insert the verification result data of the rule into the verification result table.

[0119] After the rule is executed, the system will determine whether the data meets the quality requirements based on the execution result. For example, if the SQL query that checks for non-null fields returns a count of 0, it means there are no null values ​​and the rule check passes; if the return value is greater than 0, it means there are null values ​​and the rule check fails.

[0120] Step 135: Perform the above policy verification process on all rules until all rules have completed quality inspection analysis or all policy verifications have been completed, then end this policy detection.

[0121] The system will execute all rules in the strategy sequentially and record the execution result of each rule. The quality inspection analysis ends when all rules have been executed or all strategy checks have been completed.

[0122] Step 140: Write the quality problem results into the database.

[0123] Quality issues discovered during the verification process are written to the database, including information such as issue type, severity, scope of impact, and discovery time. In a preferred embodiment of the present invention, the verification results include various types such as: duplicate values ​​in fields, field uniqueness, field format, field length, field value type, field value range, invalid character values, invalid decimal places, garbled Chinese characters, missing data, and dirty data.

[0124] For example, for field formatting issues, the system will record the specific values ​​that do not meet the format requirements and the number of times they occur; for value range issues, the system will record the specific values ​​that exceed the value range and the number of times they occur; for data missing issues, the system will record the proportion of missing data and the scope of its impact.

[0125] Step 150: Clear the data from the last verification.

[0126] To avoid data redundancy and confusion, the system clears intermediate data from the previous verification after each verification, retaining only the final verification result. This helps reduce storage space usage and improve system performance.

[0127] Step 6: Output the verification report.

[0128] The system generates a detailed verification report based on the verification results, including an overall data quality overview, quality indicators for each dimension, details of specific issues, and trend analysis. The report can be presented in various formats, such as PDF documents, Excel spreadsheets, and HTML web pages, for easy viewing and analysis by users.

[0129] Reports typically include the following:

[0130] Verification summary: Overall quality score, number of issues, severity distribution, etc.;

[0131] Issue details: Displays specific issues by type, severity, scope of impact, etc.

[0132] Trend Analysis: Shows the trend of data quality changes over time;

[0133] Improvement suggestions: Provide specific improvement suggestions for the problems identified.

[0134] In another embodiment of the present invention, the automated quality auditing method for heterogeneous data governance further includes a quality problem analysis and remediation function. This function analyzes the root causes of quality problems based on the verification results and generates corresponding remediation plans.

[0135] Specifically, the system first categorizes and summarizes the verification results, identifying common and specific issues. Then, based on pre-defined rules and machine learning algorithms, the system analyzes the root causes of the problems. For example, inconsistent field formats may stem from lax validation during data input; missing data may result from data loss during inter-system integration; and duplicate data may be due to repeated imports or flawed primary key design.

[0136] Based on the root cause analysis results, the system will generate corresponding remediation plans, including data-level repair and process-level optimization. Data-level repair includes data cleaning, data completion, and data deduplication; process-level optimization includes enhanced input validation, system integration optimization, and primary key design improvements.

[0137] In addition, the system supports the automatic execution of repair solutions, including automatic generation of repair scripts, automatic scheduling of execution, and automatic verification of results. After the repair is completed, the system will perform a quality check again to confirm whether the problem has been resolved.

[0138] Through its quality problem analysis and repair functions, this invention can not only identify data quality problems, but also proactively solve them, forming a closed-loop management system and further improving the automation level and efficiency of data quality governance.

[0139] In another embodiment of the present invention, the automated quality auditing method for heterogeneous data governance supports the collaborative operation of multiple execution modes, including batch processing mode and stream processing mode.

[0140] Batch processing mode is suitable for scenarios requiring comprehensive quality verification of historical data. In this mode, the system processes a large amount of data at once, performing in-depth and comprehensive quality verification. Batch processing is typically executed during periods of low system resource load (such as at night or on weekends) to minimize the impact on daily business operations.

[0141] Stream processing is suitable for scenarios requiring real-time quality verification of data. In this mode, the system verifies newly generated or changed data in real time, promptly identifying and resolving quality issues. Stream processing is typically tightly integrated with business systems, triggering the verification process immediately upon data generation or modification.

[0142] The two modes can work together to complement each other: stream processing ensures that new data meets quality requirements, while batch processing periodically performs a comprehensive check on all data to ensure that no quality issues are overlooked. The system will automatically select the most suitable execution mode or combine the two modes based on business needs and data characteristics.

[0143] By working in concert with multiple execution modes, this invention can improve system response speed while ensuring comprehensive verification, and meet the data quality governance needs in different scenarios.

[0144] This invention provides an automated quality auditing method for heterogeneous data governance. Through innovative technologies such as task pool construction, intelligent task scheduling, dual-engine parallel processing, and multi-dimensional quality verification, it achieves unified and efficient processing of structured and unstructured data, significantly improving the automation level and accuracy of data quality governance. This invention supports multiple execution frequencies and modes, optimizes system resource utilization, expands its applicability, and forms comprehensive data quality control. Through this invention, enterprises and organizations can manage data quality more efficiently and comprehensively, providing reliable assurance for data-driven decision-making and business operations.

[0145] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An automated quality auditing method for heterogeneous data governance, characterized in that, Includes the following steps: Step 1: Select the object to be audited; Step 2: Build a task pool, process verification rules, and generate scheduling tasks; Step 3: Retrieve an audit object from the task pool; Step 4: Request data from relevant services according to the type of the scheduled task. If the data type is structured, request data from the heterogeneous data service; if the data type is unstructured, request data from the unstructured data service. Step 5: Verify the scheduled task and generate a verification report; Step 6: Output the verification report; Step 2, which involves building a task pool, processing verification rules, and generating scheduled tasks, includes: Step 2.1: Construct a task pool. The scheduling system receives scheduling tasks issued by the rule engine of data governance, creates a configuration table for scheduling tasks in the database, and saves each scheduling task as a record in the configuration table. Each scheduling task is a rule executed after verifying one or more objects. The scheduling task consists of a scheduling verification rule table and a table of objects to be verified. The scheduling verification rule table contains the ID of the verification rule and the scheduling execution frequency of the verification. The object table contains the ID of the object to be verified and the ID of the corresponding verification rule. Step 2.2, Processing verification rules: The scheduling system extracts each core task according to the configuration table of the scheduling tasks and assigns it to different verification rule processing engines according to the type of the scheduling task. The verification rule processing engine reads the task and generates a scheduling task instance. The verification rule processing engine includes a heterogeneous data processing verification rule engine and an unstructured data processing verification rule engine. The execution frequency of the scheduling task includes full execution, incremental execution, and periodic execution. Incremental execution is used to perform periodic incremental processing on the data of the object being verified. Only newly added data is verified and a new verification report is generated and output. Step 2.3: The task scheduling module generates scheduling tasks according to the execution frequency of the scheduled tasks. The heterogeneous data verification rule processing engine creates multiple scheduling tasks to be processed according to the database and database table dimensions. When each scheduling task is created, the scheduling task is generated according to the execution frequency of the scheduling task.

2. The automated quality auditing method for heterogeneous data governance according to claim 1, characterized in that, In step 1, the object types include libraries, tables, domains, and files.

3. The automated quality auditing method for heterogeneous data governance according to claim 1, characterized in that, In step 4, if the scheduled task is heterogeneous data, data is requested from the heterogeneous data service, including: starting the heterogeneous data processing verification engine; the heterogeneous data processing verification engine generates a query task based on the name of the database table and the SQL query statement in the verification rules; the heterogeneous data processing task obtains the query data of the scheduled task based on the name of the object being verified and the verification rules; the heterogeneous data processing task queries the corresponding data from the corresponding database according to the query task; and the queried data is processed and output to the heterogeneous data verification engine.

4. The automated quality auditing method for heterogeneous data governance according to claim 3, characterized in that, When generating a query task, it also includes: Step 10: After the heterogeneous data processing verification engine obtains the scheduled task, it determines whether the name of the object to be verified and the verification rules are consistent based on the name of the object to be verified and the verification rules. If they are consistent, step 20 is executed; otherwise, the process ends. Step 20: Determine whether the object to be verified has been verified in the previous execution cycle. If the task has been executed, proceed to step 30; otherwise, proceed to steps 40-60. Step 30: Determine whether the frequency of the scheduled task execution is full execution; Step 40: Request all data of the query object from the data resource service; Step 50: Determine whether the task has obtained all the data. If all the data has been obtained, execute the heterogeneous data structured data verification engine. If no data has been obtained, it means that the object did not have corresponding data in the heterogeneous data service before. Then execute step 60 and do not execute the task. Step 60: Filter the query results, keeping only the results obtained during the last verification. Step 70: Execute the structured data verification engine for heterogeneous data. After execution, proceed to step 80. Step 80: Insert the results generated in this cycle of verification into the audit result data table; Step 90: Clear the verification records for the object being verified.

5. The automated quality auditing method for heterogeneous data governance according to claim 1, characterized in that, In step 4, if the scheduled task is unstructured data, data is requested from the unstructured data service, including: starting the unstructured data processing and verification engine; the unstructured data processing and verification engine requests information from the structured data service based on the object name and the verification rules of the object; requests information on the query object from the data resource service; obtains the query results; and calls the semantic analysis and semantic extraction services based on the query results to transform the data into structured data.

6. The automated quality auditing method for heterogeneous data governance according to claim 5, characterized in that, The unstructured data information includes: file content, web page text, natural language text, images, and videos.

7. The automated quality auditing method for heterogeneous data governance according to claim 1, characterized in that, Step 5 also includes: Step 110: The data resource service receives query requests for structured data and pushes query information to the scheduling task; Step 120: Based on the data query results obtained from the structured data, start the structured data verification engine for heterogeneous data to verify each table to be verified. Step 130: The structured data verification engine for heterogeneous data performs quality inspection analysis based on the verification rules and data sets of each table; Step 140: Write the quality problem results into the database; Step 150: Clear the data from the last verification.

8. The automated quality auditing method for heterogeneous data governance according to claim 7, characterized in that, In step 130, the quality inspection analysis includes: Step 131: Start quality verification. The quality verification process iterates through all the strategies that need to be verified in each table and starts the quality inspection strategy verification engine. Step 132: After starting the quality inspection strategy verification engine, first perform strategy verification parameter verification. If the verification passes, perform rule verification and parameter verification on the verification rules, and then start the rule verification engine for quality inspection analysis. If the verification fails, do not start the rule verification engine. Step 133: Start the rule validation engine, obtain the data table and rules of the policy and the specific parameters of the rules, and then parse the policy into multiple executable SQL statements according to the SQL abstract syntax analysis engine. Step 134: Execute the strategy, execute SQL statements to query data, generate the verification results of the rules corresponding to the strategy, and insert the verification result data of the rules into the verification result table; Step 135: Perform the above policy verification process on all rules until all rules have completed quality inspection analysis or all policy verifications have been completed, then end this policy detection.

9. The automated quality auditing method for heterogeneous data governance according to claim 8, characterized in that, In step 134, the verification results include: duplicate values ​​of the field, field uniqueness, field format, field length, field value type, field value range, illegal character values, illegal decimal places, garbled Chinese characters, missing data, and dirty data.

Images