Intelligent quality gate access automatic checking and closed-loop rectification method and system before entering a lake

Abstract

This invention relates to a method and system for automatic verification and closed-loop rectification of intelligent quality access control before data enters the data platform, belonging to the field of data governance and data middle platform. An independent quality access control zone is set up between the source system and the data storage layer of the data middle platform. A three-layer progressive verification process—integrity pre-check, standardization pre-check, and business logic pre-check—is performed on the data entering the data platform. Data that fails all checks is intercepted, and a quality anomaly record is generated and pushed to the responsible party for rectification. Only data that passes automatic re-verification can enter the data platform. The subsequent verification strategy is adaptively adjusted based on the anomaly distribution characteristics of the verification results. The rule thresholds, verification severity levels, and field verification priorities of the quality verification rule base are dynamically updated based on the execution effect parameters, achieving adaptive evolution of the rules. This prevents low-quality data from entering the data middle platform, reduces subsequent data cleaning costs, improves data credibility, and enables the quality verification rules to have adaptive optimization capabilities. It can be applied to various enterprise data middle platform and data lake construction scenarios.

Description

Technical Field

[0001] This invention relates to the field of data governance and data platform technology, specifically to a method and system for automatic verification and closed-loop rectification of intelligent quality access control before lake entry. Background Technology

[0002] With the deepening of enterprise digital transformation, data platforms, as a key infrastructure for enterprise data aggregation, governance, and service, have been widely deployed in industries such as finance, manufacturing, and energy. Currently, over 70% of large and medium-sized enterprises in China have completed the construction of data platform infrastructure, and components such as data lakes and data warehouses have been fully deployed. However, data platform construction generally faces the industry problem of "emphasizing construction while neglecting governance." After completing data aggregation, many enterprises have encountered problems such as data silos, inconsistent standards, uneven data quality, and difficulty in implementing compliance management. Research shows that although 40% of enterprises have built a data platform infrastructure, data specifications remain on paper, and even after large-scale data governance, there is still a situation of "polluting while governing." The common industry dilemma is: data comes in, but standards are inconsistent; reports are running, but interpretations are inconsistent; quality rules are configured, but anomalies go unanswered.

[0003] In the data access phase, traditional data platforms typically adopt a "first into the lake, then clean up" construction model—data from source systems such as ERP, MES, financial systems, procurement systems, and inventory systems is directly extracted and written into the operational data store (ODS layer) of the data platform, and then cleaned and standardized in the subsequent data warehouse detail (DWD layer). While this model is relatively simple to implement, it has significant shortcomings from a data quality management perspective: quality issues such as missing values, duplicate records, format errors, inconsistent coding, and business logic conflicts in the source systems are not effectively intercepted during data entry and directly enter the ODS layer, causing the ODS layer to gradually become a data dumping ground with uncontrollable quality. This low-quality data, after entering the subsequent summary data layer (DWS layer) and application data store (ADS layer), directly affects the data credibility of BI reports, business analysis, and AI-powered intelligent question answering.

[0004] Existing technologies already include several solutions for data quality management. Chinese patent application CN120336302A discloses a data quality and security processing system and method based on automatic matching rules. This system achieves data quality checks through a dynamic rule configuration module, multi-source data acquisition, data feature recognition, and a distributed computing engine. Its technical solution focuses on the parallel execution of quality check tasks and the optimization of rule thresholds. Chinese patent application CN119337418A discloses a data asset management system. This system verifies input data using a pre-collection database and a security gateway, generates record information, and then stores the data in a data lake or returns it to the client. However, the security gateway in this solution primarily addresses the verification of data source legality and access permissions. Its verification purpose differs fundamentally from data quality verification and does not involve quality checks related to data integrity, format standardization, or business logic consistency. Furthermore, China Merchants Financial Technology Co., Ltd.'s invention patent CN115543198B discloses a method for ingesting unstructured data into a data lake. This method generates extended metadata and performs data deduplication checks during data writing. However, its verification dimension is singular, primarily targeting data duplication issues, and it fails to cover various data quality problems such as missing fields, format errors, and cross-table business logic conflicts. In existing data governance solutions, quality verification rules are typically statically configured. Once set, rule thresholds and verification strategies remain fixed for a long period, making it difficult to adaptively adjust to changes in data characteristics and the accumulation of governance experience.

[0005] It is evident that existing technologies lack an automatic quality gatekeeping method that performs integrity, standardization, and business logic verification on data before it formally enters the ODS layer, and systematically intercepts, provides feedback on, rectifys, and re-verifies non-compliant data. Furthermore, there is a lack of an adaptive governance method that can dynamically optimize verification strategies and rule thresholds based on verification results and governance effectiveness. Summary of the Invention

[0006] To address the problems of existing technologies, this invention provides a method and system for automatic verification and closed-loop rectification of intelligent quality access control before lake entry.

[0007] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution: Firstly, a method for automatic verification and closed-loop rectification of intelligent quality access control before lake entry, comprising: The data acquisition step involves obtaining source data to be added to the lake from at least one source system. The data writing gate restricted area step involves writing the source data into an independent data cleaning and verification area set between the source system and the data middle platform operation data storage layer; The steps for establishing a quality verification rule base include configuring the quality verification rule base, which contains integrity verification rules, standardization verification rules, and business logic verification rules for each field. Each rule is configured with a corresponding rule threshold, verification strictness level, and field verification priority. The quality verification step involves performing integrity verification, normalization verification, and business logic verification on the source data within the independent data cleaning and verification area according to the quality verification rule base, generating verification results and anomaly distribution characteristics for each field; the integrity verification is used to check whether a field is empty and the field fill rate; the normalization verification is used to check whether the field format conforms to a preset format template; the business logic verification is used to check whether the business logic relationship between cross fields and / or cross tables is valid; The adaptive processing step involves performing at least one of the following adaptive actions based on the verification result and the anomaly distribution characteristics: (a) When the integrity check detects that the null value rate of the target field exceeds the first preset threshold, the strictness level of the check of the target field in the normative check is reduced to skip check, and the target field is marked as pending supplementation. (b) When the normative verification detects that the frequency of format errors in the target field is lower than the second preset threshold and the error type is a preset repairable type, the format conversion adapter matching the error type is called to perform batch format cleaning on the target field, generate repaired data, and use the repaired data to replace the original abnormal data in subsequent verification, while recording the repair log; when the frequency of format errors is higher than or equal to the second preset threshold, the target field is marked as pending manual rectification. The feedback update step collects the execution effect parameters of the adaptive processing step, which include at least one of the following: field fill rate, automatic repair success rate, single batch verification time, and anomaly detection rate; and dynamically updates the rule threshold, verification strictness level, and field verification priority of the corresponding field in the quality verification rule base according to the execution effect parameters. The lake entry control step involves writing data that passes the quality check to the operational data storage layer, intercepting data that fails the quality check and generating an anomaly record. The iterative governance step uses the updated quality verification rule base from the feedback update step as the basis for quality verification of the next batch of source data, and repeats the quality verification step until the feedback update step.

[0008] In one specific implementation of the first aspect, in the adaptive processing step, the preset repairable types include date format separator inconsistency type and encoding prefix missing type; the format conversion adapter includes date format unification adapter and encoding prefix completion adapter; When the normative check detects that the date format of the target field is a mixture of "YYYY / MM / DD" and "YYYY-MM-DD", the date format unification adapter is called to convert all dates to "YYYY-MM-DD" format. When the normative verification detects that the target field is missing a preset prefix, the encoding prefix completion adapter is invoked to automatically complete the preset prefix for the encoding field that is missing the preset prefix; The adaptive processing step also includes a post-repair consistency verification sub-step: performing consistency verification on the data repaired by the format conversion adapter, and replacing the original abnormal data with the repaired data that passes the verification in subsequent verifications, while marking the data that fails the verification as pending manual review.

[0009] In one specific implementation of the first aspect, the feedback update step dynamically updates the field validation priority based on the execution effect parameter, specifically including: The failure rate of each field in the historical re-inspection process was statistically analyzed. Fields with a failure rate higher than the third preset threshold are marked as high-risk fields and prioritized for quality verification at the front of the verification queue in the next batch of verification. Fields with a failure rate lower than the fourth preset threshold in consecutive preset batches are marked as low-risk fields, and are moved to the end of the verification queue in the next batch of verification, while the verification sampling rate of the low-risk fields is reduced to the preset sampling ratio.

[0010] In one specific implementation of the first aspect, the feedback update step dynamically updates the quality verification rule base according to the execution effect parameters, specifically including: When the automatic repair success rate is higher than the fifth preset threshold in N consecutive batches, the format conversion rule corresponding to the automatic repair success rate is upgraded from "online repair rule" in the quality verification rule library to "pre-conversion rule", so that the source data of subsequent batches will be automatically converted by the pre-conversion engine before entering the independent data cleaning and verification area; N is a positive integer greater than or equal to 3; When the field fill rate is higher than the sixth preset threshold in M ​​consecutive batches, the strictness level of the field integrity check is reduced, the field integrity check rule is changed from "mandatory check" to "sampling check", and the sampling check ratio is set; M is a positive integer greater than or equal to 3.

[0011] In one specific implementation of the first aspect, the business logic verification includes at least one of the following business logic relationships: The consistency between order amount and the calculated results of quantity, unit price, discount, and tax rate; The chronological relationship between the date of warehousing and the date of production completion; The existence relationship of supplier codes in purchase orders in the master data; The relationship between the quantity of materials requisitioned and the planned quantity of materials to be used; The existence of customer codes in customer master data.

[0012] In one specific implementation of the first aspect, the anomaly record includes the following information items: data batch number, source system identifier, source table identifier, target table identifier, anomaly field name, anomaly type identifier, hit anomaly rule number, number of anomaly records, anomaly sample data, responsible department identifier, and processing status identifier; The method also includes an audit trail step, which records the execution log of the quality verification step, the repair log and action log of the adaptive processing step, the rule change log of the feedback update step, and the lake entry status log of the lake entry control step, forming a traceable quality management audit chain. The method also includes a data quality health scoring step: calculating a comprehensive quality score for each data table based on the abnormal records and the execution effect parameters, and using the comprehensive quality score as a weighting factor for adjusting the field validation priority in subsequent batches.

[0013] In one specific implementation of the first aspect, a closed-loop rectification step is also included: Based on the responsible department identifier in the anomaly record, an anomaly rectification notice is sent to the corresponding responsible party. Receive the rectified data, rewrite the rectified data into the independent data cleaning and verification area, and execute the quality verification steps again according to the quality verification rule base; If the verification passes, the data is written to the operation data storage layer by the lake entry control step; if the verification fails, the system continues to intercept and push the rectification notice again until the rectification is completed or the preset rectification limit is reached.

[0014] Secondly, a governance system for quality gatekeeping before data enters the lake includes: The data extraction module is used to obtain source data to be added to the lake from at least one source system; The quality gate restricted area module is located between the source system and the data middle platform operation data storage layer, and is used to receive and temporarily store the source data; The quality verification rule base module is used to store and manage integrity verification rules, normative verification rules, and business logic verification rules for each field, as well as the rule threshold, verification severity level, and field verification priority for each rule; The quality verification module is used to perform integrity verification, standardization verification, and business logic verification on the source data within the quality gate restricted area module according to the rules in the quality verification rule base module, and generate the verification results and anomaly distribution characteristics of each field; An adaptive execution module is used to adjust the verification strictness level of the corresponding field in the quality verification rule base module according to the verification result and the abnormal distribution characteristics, and / or perform online format cleaning on abnormal data; The effect feedback module is used to collect the execution effect parameters of the adaptive execution module and send a rule update instruction to the quality verification rule base module according to the execution effect parameters. The execution effect parameters include at least one of field filling rate, automatic repair success rate, single batch verification time and anomaly detection rate. The lake entry control module is used to write data that passes the quality check into the operation data storage layer, intercept data that fails the quality check, and generate an anomaly record; The iteration control module is used to use the updated rules from the quality verification rule base module as the verification basis for the next batch of source data.

[0015] In one specific implementation of the second aspect, it further includes: The anomaly management module is used to push anomaly rectification notices to the corresponding responsible parties based on the responsible department identifier in the anomaly record, and to receive the data after rectification. The lake entry re-inspection module is used to rewrite the rectified data into the quality gate restricted area module, and the quality verification module performs quality verification again. After the verification is passed, the lake entry control module writes the data into the operation data storage layer. The audit trail module is used to record the execution log of the quality verification module, the repair log of the adaptive execution module, the rule change log of the effect feedback module, and the lake entry status log of the lake entry control module.

[0016] Thirdly, a governance device for quality gate control before data enters the lake includes: One or more processors; A storage device is used to store one or more programs and a preset quality verification rule base. The quality verification rule base includes integrity verification rules, normative verification rules and business logic verification rules for each field. Each rule is configured with a corresponding rule threshold, verification strictness level and field verification priority. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 7.

[0017] The beneficial effects of this invention are as follows: 1. This invention establishes an independent quality gate between the source system and the data middleware ODS layer, separating data quality verification from the internal governance process of the data middleware and placing it upstream. This achieves a "governance before data entry" data access model at the architectural level, avoiding the subsequent increase in governance costs and the decline in the credibility of downstream applications caused by low-quality data entering the ODS layer. Based on the above architecture, this invention further adopts a three-layer progressive quality verification structure: integrity pre-check, standardization pre-check, and business logic pre-check. It automatically audits data layer by layer from three dimensions: field-level completeness, format standardization, and consistency of cross-field / cross-table business logic. This achieves in-depth verification from "whether data has been filled in" and "whether the filling is standardized" to "whether the business relationship is valid," systematically covering various data quality defects in enterprise data entry scenarios, such as missing fields, incorrect formats, inconsistent coding, inconsistent cross-table relationships, and invalid business facts. Based on this, the present invention establishes a complete closed-loop management mechanism for the interception, feedback, rectification, re-verification, and entry into the data lake for non-compliant data. Each intercepted non-compliant data item has a clear anomaly type identifier, responsibility attribution, and processing status tracking. After rectification, data undergoes automatic re-verification before being entered into the data lake. Data that fails re-verification continues to be intercepted, and the rectification process is re-triggered until it becomes compliant or meets preset upgrade conditions. This closed-loop mechanism transforms data quality management from a passive mode relying on manual spot checks and post-event remediation to a proactive governance mode characterized by automatic system execution, clearly defined responsibilities, and traceable processes. The discovery, processing, verification, and closure of data quality issues form a complete management closed loop, avoiding information loss caused by simply discarding non-compliant data and preventing secondary quality risks from directly entering the data lake without verification. 2. Building upon the aforementioned basic verification and closed-loop rectification architecture, this invention further establishes an adaptive verification strategy adjustment mechanism based on verification results and a rule self-evolution mechanism based on execution effect feedback. During the verification process, the system adaptively adjusts the verification strictness of the normative pre-inspection for that field based on the field fill rate output by the integrity pre-inspection, avoiding invalid verification due to data sparsity. Based on the format error type and error frequency output by the normative pre-inspection, it adaptively diverts occasional format errors to the automatic repair channel and systemic format deviations to the manual rectification channel, achieving differentiated configuration of processing resources under different anomaly modes. Furthermore, the system collects execution effect data of the adaptive processing process, including automatic repair success rate, field fill rate, and anomaly detection rate, and dynamically updates the rule thresholds, verification strictness levels, and field verification priorities in the quality verification rule base based on the multi-batch statistical results of the above execution effect data. Mature repair rules that have been verified through multiple batches are solidified into pre-conversion rules, low-risk fields with long-term stable performance are downgraded from mandatory verification to sampling verification, and fields with frequent historical problems are adjusted to the front of the verification queue for priority execution. The aforementioned mechanism enables the quality verification rule base to adaptively optimize based on historical execution results. Verification strategies can continuously adjust themselves as data characteristics change and governance experience accumulates, forming a positive self-evolutionary closed loop of "verification execution → effect evaluation → rule optimization → guiding subsequent verification." Therefore, this invention fundamentally changes the limitations of existing data governance solutions where quality verification rules are statically configured and cannot dynamically evolve with data characteristics. It allows the data quality governance system to adaptively configure verification resources and optimize governance strategies when facing different source systems, different data structures, and constantly changing business data, achieving the technical effect of continuously improving data governance efficiency as the number of system operation batches increases. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the overall process of the present invention.

[0019] Figure 2 This is a schematic diagram of the closed-loop rectification process for non-conforming data in this invention.

[0020] Figure 3 This is a schematic diagram of the closed-loop rectification process for non-conforming data in this invention. Detailed Implementation

[0021] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0022] like Figures 1 to 3 The method and system for automatic verification and closed-loop rectification of intelligent quality access control before entering the lake are shown.

[0023] (I) System Architecture This invention establishes an independent data cleaning and verification zone (i.e., a quality gate restricted area) between the source system and the data middleware ODS layer. The source system includes, but is not limited to, ERP, MES, PHM, financial, procurement, and inventory systems. After data from the source system is extracted using data extraction tools (such as DataX, Sqoop, or Kafka data pipelines), it is not directly written to the ODS layer but first enters the quality gate restricted area. Only data that passes all quality verifications can be written to the ODS layer; data that fails verification is intercepted and enters the rectification process.

[0024] The layered architecture of the data platform, from bottom to top, consists of: the ODS layer (Operational Data Storage Layer, storing raw data), the DWD layer (Detailed Data Layer, after cleaning and standardization), the DWS layer (Summary Data Layer, summarizing by topic), and the ADS layer (Application Data Layer, oriented towards specific application scenarios). The quality gate isolation zone of this invention is independent of the above layers. Physically, it can be deployed on independent server nodes or container clusters, and logically, it is connected to the source system and the ODS layer respectively through data pipelines.

[0025] (ii) Configuration of the quality verification rule base This invention transforms data standards, coding rules, field specifications, master data rules, and business logic rules developed during the data governance consulting phase into automatically executable quality audit rules. Rule types include required field rules, field format rules, coding specification rules, value range rules, uniqueness rules, primary and foreign key consistency rules, cross-field logic rules, cross-table business logic rules, and indicator caliber verification rules.

[0026] In this specific implementation, the quality verification rule base is stored using a structured configuration. Each rule contains the following fields: Rule ID: Uniquely identifies a rule; Rule Name: A Chinese description of the rule; Rule type (Rule_Type): Completeness, normativity, or business logic; Applicable Table Name (Table_Name): The data table to which the rule applies; Applicable Field Name (Field_Name): The field to which the rule applies; Rule expression: Executable conditional judgment logic, written using SQL-like syntax or a scripting language; Rule_Threshold: The threshold value that triggers interception; Strict Level: Forced verification, sampling verification, or skip verification; Field validation priority: determines the order in which fields are validated in the validation queue; Department ID (Dept_ID): The department responsible for the data quality issue; Is it enabled (Is_Active): The start / stop status of the rule; Version number: The version information of the rule.

[0027] The rules support a visual configuration interface, allowing users to create, modify, start, stop, and manage versions of rules via a web interface. After rule changes are implemented, online testing is supported, allowing users to upload sample data to verify the correctness of the rules before publishing and taking effect.

[0028] (III) Data writing restricted area After the data from the source system is extracted using the data extraction tool, the system performs the following operations: Input: Raw data extracted from the source system, including data content, source system identifier, source table identifier, and extraction timestamp.

[0029] Processing flow: The system generates a unique data batch number (Batch_ID) for the target data, in the format of "source system abbreviation_date_serial number", such as "ERP_20260624_001".

[0030] The system writes the raw data into a temporary storage table in the quality gate restricted area. The structure of the temporary storage table is consistent with the structure of the target ODS table, and additional management fields such as batch number, entry time into the restricted area, and processing status are added.

[0031] The system writes information such as data batch number, source system identifier, source table identifier, target table identifier, total number of records, and entry time into restricted areas into the batch management log.

[0032] Output: The source data written to the temporary storage table of the quality gate restricted area, and the corresponding batch management records.

[0033] (iv) Three-stage progressive quality pre-inspection After the data enters the quality gate restricted area, the system performs three pre-checks in sequence: 1. Integrity Pre-check The integrity pre-check checks whether the required fields in the source data are empty and calculates the fill rate of each field.

[0034] Input: Source data in the temporary storage table of the quality gate restricted area, and integrity verification rule base (including the required field identifier and fill rate threshold for each field).

[0035] Processing flow: The system iterates through each field of the target table. For fields marked as "required", it checks whether the field in each record is NULL or an empty string.

[0036] For non-required fields, calculate the fill rate of the field: fill rate = (number of non-empty records / total number of records) × 100%.

[0037] When a key field is empty, the record is marked as incomplete.

[0038] When the fill rate of a field is lower than the first preset threshold, the field is marked as a low fill rate field.

[0039] The preferred range for the first preset threshold is 60% to 80%. When the fill rate is lower than this threshold, it indicates that the data in the field is severely missing, and performing subsequent format validation on it is meaningless. For example, if the fill rate of a "Remarks" field is only 45%, which is lower than the 70% threshold, the system will mark this field as a low fill rate field and skip the format validation of this field in subsequent standardization checks.

[0040] Output: Fill rate statistics for each field, a list of records with incomplete data, and a list of fields with low fill rates.

[0041] 2. Standardized pre-inspection The pre-inspection check ensures that the field format conforms to the company's data standards.

[0042] Input: Output of integrity pre-check (especially the list of low fill rate field markers), and normative validation rule base (containing format templates for each field).

[0043] Processing flow: For fields marked as having a low fill rate (null value rate exceeding the first preset threshold), the system skips the standardization check of the field and directly marks it as "pending supplementation".

[0044] For other fields, the system checks whether the field format of each record conforms to the preset format template.

[0045] Format templates include, but are not limited to: date format (e.g., YYYY-MM-DD), code length (e.g., 8-digit fixed-length code), amount precision (e.g., retaining two decimal places), mobile phone number format (e.g., 11-digit number), unified social credit code format (e.g., 18-digit alphanumeric combination), etc.

[0046] For records with incorrect formatting, the system further determines the type and frequency of the formatting error. Formatting error frequency = (number of records with formatting errors / total number of records) × 100%.

[0047] The preferred range for the second preset threshold (formatting error frequency threshold) is 5% to 15%. When the formatting error frequency of a field is lower than this threshold (for example, the formatting error frequency of a date field is 8%, which is lower than the 10% threshold), it indicates that the error is an occasional formatting problem, and the system classifies the formatting error type of the field as "preset repairable type" and calls the corresponding format conversion adapter to perform automatic repair. When the formatting error frequency is higher than or equal to this threshold (for example, the formatting error frequency of an encoding field is 35%, which is higher than the 10% threshold), it indicates that there is a systematic formatting deviation, and the system marks the field as "awaiting manual rectification".

[0048] Preset repairable types include, but are not limited to: Inconsistent date format separators: For example, mixing "YYYY / MM / DD", "YYYY-MM-DD", and "YYYYMMDD" formats in the same field; Missing encoding prefix type: For example, the encoded field should start with "CN-", but some records are missing this prefix; Inconsistent amount precision: For example, some amount fields retain one decimal place, some retain two decimal places, and some are integers; Types of inconsistent case in enumeration values: For example, a status field should be "ACTIVE" but some records are "active" or "Active"; Inconsistent phone number format: For example, some phone numbers contain country codes while others do not.

[0049] Output: Statistical results of the frequency of format errors for each field, a list of records that do not meet the standardization requirements, records that can be automatically corrected and error type markings, and a list of fields marked for manual rectification.

[0050] 3. Business logic pre-check The business logic pre-check checks whether the business relationships across fields and tables are valid.

[0051] Input: Output results of integrity pre-check and standardization pre-check, and business logic verification rule base.

[0052] Processing flow: The system verifies the following business logic relationships one by one: (a) Order Amount Consistency Verification: Check whether the order amount is equal to the calculated results of quantity, unit price, discount, and tax rate, i.e., order amount = quantity × unit price × (1 - discount rate) × (1 + tax rate), with an allowable error range of ±0.01.

[0053] (b) Warehouse entry date time sequence verification: Check whether the warehouse entry date is later than the production completion date, that is, the warehouse entry date is greater than or equal to the production completion date.

[0054] (c) Supplier code validity check: Check whether the supplier code in the purchase order exists in the supplier master data table.

[0055] (d) Material requisition quantity limit verification: Check whether the material requisition quantity exceeds the planned material input quantity, i.e., the material requisition quantity ≤ the planned material input quantity.

[0056] (e) Customer code validity check: Check if the customer code in the order exists in the customer master data table.

[0057] When any of the above business logic relationships fails, the corresponding record is marked as having invalid business logic. The system also records the specific rule number involved and a detailed description of the business logic conflict.

[0058] Output: A list of records where the business logic is invalid, with each record accompanied by the matching rule number and conflict description.

[0059] 4. Comprehensive Judgment The system integrates the results of the three pre-checks and makes a final judgment on each record: If a record passes all three checks, it is marked as "Check Passed"; If a record fails any validation, it is marked as "Validation failed" and the specific exception type (integrity exception, normativity exception, or business logic exception) and the hit rule number are recorded.

[0060] (v) Adaptive processing This invention introduces an adaptive processing mechanism during the verification process, which dynamically adjusts subsequent verification behavior based on the verification results and anomaly distribution characteristics.

[0061] 1. Adjustment of verification strategy based on null value rate When the integrity pre-check detects that the null value rate of a certain field exceeds the first preset threshold (preferably 60%~80%): Triggering condition: Field null value rate > first preset threshold (e.g., 70%).

[0062] Adaptive Actions: The system reduces the stringency level of this field in the normative validation from "mandatory validation" to "skip validation"; The system marks this field as "pending data entry" and prompts the responsible department to supplement the data in the anomaly report; This field is directly entered into the business logic pre-check stage (if the business logic verification involves this field).

[0063] Technical benefits: It avoids unnecessary calculations during the format validation stage due to a large number of null values. For example, if the null value rate of a certain "Equipment Remarks" field is 72%, the system skips the format validation of this field and directly proceeds to the business logic validation, saving approximately 70% of the field-level validation calculation resources.

[0064] 2. Automatic correction based on format error type When the pre-detection of normative data detects that the frequency of format errors in a certain field is lower than the second preset threshold (preferably 5%~15%), and the error type is a preset repairable type: Triggering conditions: The frequency of formatting errors is less than the second preset threshold (e.g., 10%), and the error type belongs to the preset repairable type.

[0065] Adaptive Actions: (a) Calling the date format unification adapter: When the system detects that the date format of the target field is a mixture of "YYYY / MM / DD" and "YYYY-MM-DD", it invokes the date format unification adapter. This adapter internally performs the following operations: Step 1: Use regular expressions to identify the current format of the date field in each record; Step 2: Based on the recognition results, convert non-standard date strings into "YYYY-MM-DD" format; Step 3: Verify the validity of the converted results (e.g., whether the month is between 1 and 12, and whether the date is between 1 and 31).

[0066] (b) Calling the encoding prefix completion adapter: When a missing predefined prefix (such as "CN-") is detected in the target field, the system invokes the encoding prefix completion adapter. This adapter internally performs the following operations: Step 1: Check if the encoded field starts with a preset prefix; Step 2: For records with missing prefixes, automatically add a preset prefix before encoding; Step 3: Verify the length and format of the completed encoding.

[0067] Post-repair consistency verification sub-step: The system performs a consistency verification on the data after it has been repaired by the format conversion adapter: Verify that the repaired data conforms to the target format template; Verify whether the business meaning of the data remains unchanged after the repair (e.g., the year, month and day values ​​remain unchanged after date conversion). The corrected data, once verified, replaces the original abnormal data in subsequent verifications. Data that fails verification is marked as "awaiting manual review" and will not be included in automatic repair.

[0068] Technical benefits: Occasional formatting errors are automatically corrected without manual intervention, reducing the workload of manual rectification. For example, in a batch of 10,000 records, 800 records had date formats of "YYYY / MM / DD" instead of the standard format "YYYY-MM-DD". The system automatically completed the format conversion, achieving a 96% success rate in correcting the errors. Only 32 records were marked as requiring manual review due to invalid dates (such as February 30th).

[0069] 3. Rectification and triage based on the frequency of formatting errors When the pre-check detects a formatting error frequency in a field that is higher than or equal to the second preset threshold: Triggering condition: The frequency of formatting errors is greater than or equal to the second preset threshold (e.g., 10%).

[0070] Adaptive Actions: The system marks this field as "awaiting manual rectification"; Instead of performing automatic repairs, the system pushes all abnormal data to the rectification process. The system specifically noted in the anomaly report that there was a "systematic format deviation, and it is recommended to standardize the source data."

[0071] Technical benefits: Avoids performing invalid automatic repairs in scenarios with large-scale format errors, reduces waste of computing resources, and prompts data governance personnel to solve problems at the source.

[0072] (vi) Interception and abnormal recording of non-compliant data When data fails any verification rule, the system does not write it to the ODS layer, but instead generates a quality anomaly record.

[0073] Input: Records that failed the validation and their exception types, and the rule number that was matched.

[0074] Processing flow: The system generates a unique exception record number (Exception_ID) for each exception record.

[0075] The system summarizes anomaly information according to dimensions such as data batch number, source system, source table, and target table.

[0076] The system generates structured quality anomaly records, which include the following information items: Data batch number (Batch_ID); Source system identifier (Source_System); Source table identifier (Source_Table); Target table identifier (Target_Table); Exception field name (Field_Name); Exception type identifier (Exception_Type): Integrity exception, normative exception, or business logic exception; The exception rule number (Rule_ID) that was hit; Exception count; Abnormal Sample Data (Sample_Data): Retain the complete content of up to the first 10 abnormal records; Department ID (Dept_ID): Determined based on the mapping of data asset owner, business domain, and source system; Processing status (Status): Pending, In progress, Rectified, or Closed; Creation time (Create_Time).

[0077] Output: Structured quality anomaly records are written to the anomaly management database.

[0078] (vii) Feedback and Rectification The system pushes the anomaly report to the corresponding data governance personnel or business team based on the responsible department identifier in the quality anomaly record.

[0079] Input: Quality anomaly records, responsible department mapping table.

[0080] Processing flow: The system automatically matches the responsible department and person based on the data asset owner, business domain, and source system information.

[0081] The system generates an anomaly rectification notification, which includes: data batch number, anomaly summary (distribution of anomaly types, number of affected records), anomaly sample data, rectification deadline, and rectification operation instructions.

[0082] The system pushes notifications for abnormal rectification via email, corporate instant messaging, or in-system messages.

[0083] Business personnel can view abnormal data in the rectification interface and make corrections according to the rules.

[0084] Correction methods include, but are not limited to: Fill in missing fields: For any missing required fields, fill in the correct data. Modify format: Correct data that does not meet the format requirements to the correct format; Adjust the encoding: Change the incorrect encoding to an encoding that conforms to the specifications; Confirm business logic: For data with conflicting business logic, modify the data after confirming the correct business facts; Resubmit data: For batches with serious issues, re-export and resubmit from the source system.

[0085] Maximum number of remediation attempts: The system has a preset maximum number of remediation attempts, preferably 3-5. When a batch of data reaches the maximum number of remediation attempts and still fails verification, the system automatically escalates the process and notifies the data governance committee to intervene, preventing an infinite loop.

[0086] Output: Rectified data and rectification operation log.

[0087] (viii) Automatic re-verification and entry into the lake The rectified data was resubmitted to the quality control zone for another round of checks on completeness, standardization, and business logic.

[0088] Input: Data after rectification, and the original quality verification rule base (which may have been modified in the feedback update process).

[0089] Processing flow: After rectification, the data was rewritten to the temporary storage table in the quality gate restricted area, generating a new data batch number (with suffixes such as "-R1" or "-R2" added to the original batch number to indicate the number of re-inspections).

[0090] The system re-performs all verifications on the rectified data according to the three progressive pre-inspection process described in Section (IV).

[0091] If all passes, the system will automatically trigger the lake entry task, write the data to the ODS layer, and record the lake entry log (lake entry time, final batch number, verification pass rate, etc.).

[0092] If the verification still fails, the system will continue to block and send a rectification notice again according to the procedure in Section (VII) until the rectification is completed or the preset rectification limit is reached.

[0093] Output: Qualified data written to the ODS layer, lake entry logs, or continued interception of abnormal records.

[0094] (ix) Feedback updates and rule self-evolution Based on the execution effect data, this invention dynamically updates the quality verification rule base to achieve adaptive evolution of the rules.

[0095] 1. Collection of execution effect parameters The system collects execution effect parameters for the adaptive processing steps, including: Field_Fill_Rate: The percentage of non-empty records for each field in the current batch, generated by the integrity pre-check.

[0096] Auto_Repair_Success_Rate: The percentage of records that pass the consistency verification after being repaired by the format conversion adapter out of the total number of records that were attempted to be repaired.

[0097] Batch_Check_Duration: The total time taken from writing data to the gate restricted area to completing the three checks.

[0098] Exception detection rate (Exception_Detection_Rate): The proportion of records that failed verification out of the total number of records.

[0099] 2. Rule solidification based on repair success rate When the automatic repair success rate is higher than the fifth preset threshold for N consecutive batches: Triggering condition: The automatic repair success rate of N consecutive batches (N is a positive integer greater than or equal to 3, preferably 3 to 5 batches) is greater than the fifth preset threshold (preferably 85% to 95%).

[0100] Adaptive Actions – Rule Consolidation and Upgrade: The system upgrades the corresponding format conversion rules from "online repair rules" in the quality verification rule base to "pre-conversion rules". After the upgrade: Before subsequent batches of source data enter the independent data cleaning and verification area, the front-end conversion engine automatically performs this format conversion; Once the data enters the quality gate restricted area, no more format error detection and online repair will be performed on that field; it will directly proceed to the subsequent verification stage. This rule has a lower priority in the verification queue, reducing unnecessary verification overhead.

[0101] Technical Effects: Mature repair rules are executed in the pre-processing stage, avoiding repeated verification and repair after data enters the restricted area, thus reducing the overall verification time. For example, a certain date format conversion rule had a repair success rate of over 92% in four consecutive batches. The system upgraded it to a pre-processing conversion rule, and the format verification time for this field in subsequent batches decreased from an average of 8 seconds to 0.5 seconds.

[0102] 3. Validation level downgrade based on fill rate When the field fill rate is higher than the sixth preset threshold for M consecutive batches: Triggering condition: The field fill rate of M consecutive batches (M is a positive integer greater than or equal to 3, preferably 3 to 5 batches) is greater than the sixth preset threshold (preferably 95% to 99%).

[0103] Adaptive Action – Verification Level Degradation: The system lowers the strictness level of the integrity check for this field, changes the integrity check rule for this field from "mandatory check" to "sampling check", and sets the sampling check ratio (preferably 5%~20%).

[0104] In sampling verification mode, the system no longer checks whether each record in this field is empty, but instead randomly selects a preset proportion of records for verification. If the pass rate of the sampling inspection is lower than the preset standard (such as 95%), the system will automatically revert to the forced verification mode.

[0105] Technical effects: The validation intensity of low-risk fields is reduced, and the overall validation efficiency is improved. For example, if the fill rate of a certain "customer name" field is higher than 98% in five consecutive batches, the system will reduce its validation from mandatory validation to 10% sampling validation, which will reduce the validation time of this field by 90%.

[0106] 4. Prioritization of verification based on historical non-conformance rate The system dynamically adjusts the field validation priority based on the failure rate of each field in the historical re-inspection process: Triggering condition: Based on statistical analysis of historical batch verification data, updated after each batch is completed.

[0107] Adaptive Actions – Priority Ranking: Calculate the non-compliance rate (number of non-compliant records / total number of records) for each field in the most recent 10 batches. Fields with a failure rate higher than the third preset threshold (preferably 25%~35%) are marked as high-risk fields and will be prioritized for execution at the front of the verification queue in the next batch of verification. Fields with a failure rate lower than the fourth preset threshold (preferably 3% to 8%) in consecutive preset batches (preferably 5 to 10 batches) are marked as low-risk fields, and are moved to the end of the verification queue in the next batch verification, and the verification sampling rate is reduced to the preset sampling ratio (preferably 5% to 20%).

[0108] Technical benefits: High-risk issues are exposed first, low-risk fields consume less validation resources, and the overall problem detection time is shortened. For example, a certain "supplier code" field has a historical non-compliance rate of 32%, so it is moved to the top of the validation queue, and the problem can be detected and reported within the first 10% of batch processing time.

[0109] (x) Quality process audit and health score The system records logs of the entire process, including execution of quality rules, discovery of anomalies, message push, manual rectification, re-inspection results, and lake entry status, forming a traceable quality management closed loop.

[0110] Audit log types include: Quality verification execution log: Records the execution time, number of verification records, number of passes, and number of failures for each batch of three verifications; Adaptive processing action log: Records the triggering conditions, executed actions, and results of adaptive actions; Rule change log: Records every change to rule thresholds, validation strictness levels, and field validation priorities, including the value before the change, the value after the change, the time of the change, and the reason for the change; Lake entry status log: Records the time, batch number, number of entry records, and verification pass rate of data entering the lake.

[0111] Data quality health score: The system calculates a comprehensive quality score for each data table based on anomaly records and execution effect parameters. The score calculation formula is as follows: Overall Quality Score = Completeness Score × Weight 1 + Standardization Score × Weight 2 + Business Logic Score × Weight 3 in: Integrity score = (1 - null value rate) × 100; Standardization score = (1 - format error rate) × 100; Business logic score = (1 - business logic conflict rate) × 100; Weight 1, Weight 2, and Weight 3 can be configured according to business scenarios, and the default weights are all 1 / 3.

[0112] The overall quality score serves as a weighting factor for adjusting field validation priority in subsequent batches. When the overall quality score of a data table consistently exceeds 90 points, the system can reduce the overall validation intensity of that table; when the score consistently falls below 60 points, the system automatically increases the validation priority of that table and notifies the data governance manager.

[0113] VI. Example 1: Governance of production data from an equipment manufacturing enterprise in a lake.

[0114] Scene description: A manufacturing company is building a data platform and needs to integrate production completion data from its MES (Manufacturing Execution System) into the ODS (Operational Data Center) layer for subsequent production statistics and business analysis. The data source is the MES system's "Production Completion Table" (PROD_COMPLETE), which adds approximately 5,000 records daily, containing the following key fields: Completion Order Number (COMPLETE_NO), Product Code (PROD_CODE), Product Name (PROD_NAME), Completion Quantity (COMPLETE_QTY), Completion Date (COMPLETE_DATE), Production Workshop (WORKSHOP), Equipment Number (EQUIP_NO), Operator (OPERATOR), Quality Inspection Result (QC_RESULT), and Remarks (REMARK).

[0115] Implementation steps: Step 1: Configure the quality verification rule base The system administrator configures the following quality verification rules through the visual configuration interface: ; Step 2: Data Extraction and Entry into Restricted Areas On June 24, 2026, the system extracted production completion data from the MES system via a Kafka data pipeline, totaling 4987 records. The system generated a batch number "MES_20260624_001" and wrote the data to the temporary storage table TMP_PROD_COMPLETE in the quality gate restricted area.

[0116] Step 3: Three-step progressive quality pre-inspection Integrity pre-check: The system checks the following fields: COMPLETE_NO (0% null value rate), PROD_CODE (0% null value rate), COMPLETE_QTY (0.8% null value rate, 40 records are null), COMPLETE_DATE (0% null value rate).

[0117] Non-required field fill rate: The fill rate of the REMARK field is only 35% (1745 records have values, 3242 records are empty), which is lower than the first preset threshold of 70%, and is marked as a low fill rate field.

[0118] Forty records with empty COMPLETE_QTY were marked as incomplete.

[0119] Standardized pre-inspection: The REMARK field was skipped from format validation due to low fill rate and marked as "to be supplemented".

[0120] The COMPLETE_DATE field: Of the 4987 records, 63 records have a date format of "2026 / 06 / 24" (slash-separated) instead of the standard format "2026-06-24". The format error frequency is 1.26% (63 / 4987), which is lower than the second preset threshold of 10%. The error type is "inconsistent date format separator", which is a preset repairable type.

[0121] The QC_RESULT field indicates that the quality inspection result for 5 records is "pass" (lowercase), instead of the standard enumeration value "qualified". The format error frequency is 0.1%, which is lower than the second preset threshold of 10%, and belongs to the preset repairable type (inconsistent case of enumeration values).

[0122] All other field formats are correct.

[0123] Adaptive processing – automatic repair: The system invoked the date format unification adapter: automatically converting 63 records with the date format "2026 / 06 / 24" to "2026-06-24". All consistency verifications passed, achieving a 100% success rate in the repair.

[0124] The system calls the enumeration value normalization adapter: for the 5 records where QC_RESULT is "pass", it automatically converts them to "qualified". All consistency verifications pass.

[0125] The formatting errors in 68 records were automatically corrected without manual intervention.

[0126] Business logic pre-check: The system verified the existence of device numbers in the device master data table for 4987 records. It found that 12 records had device numbers that did not exist in the device master data table (possibly due to newly added devices not being updated in a timely manner), and these were marked as having non-compliant business logic.

[0127] The system verifies the chronological relationship between the entry date (STORE_DATE, this field comes from subsequent entry records) and the completion date. Since the current batch is the first entry into the lake, no entry records have been generated yet, so this rule is not verified at this time.

[0128] Step 4: Non-compliant interception and anomaly feedback Overall judgment result: Qualified records: 4935 (4987-40-12=4935), pass rate: 98.96%.

[0129] Non-compliant records: 52, of which: 40 entries were hit due to COMPLETE_QTY being empty (integrity exception), triggering rule R003; 12 items were found to have a non-existent device number (business logic anomaly), triggering rule R009.

[0130] The system generates quality anomaly records, including batch number "MES_20260624_001", anomaly type distribution, and anomaly sample data. Based on the business affiliation of the equipment number, the system pushes 12 records with abnormal equipment numbers to the Equipment Management Department; and pushes 40 records with empty completed quantities to the Production Department.

[0131] Step 5: Rectification and Re-inspection After receiving the rectification notice, the Production Department checked and found that 40 records with empty COMPLETE_QTY were due to omissions in the MES system data collection. After supplementing the completed quantities, the records were resubmitted. The Equipment Management Department checked and found that 12 equipment numbers were not updated in the master data in a timely manner. After confirming the existence of the equipment, the master data table was updated, and the original records were resubmitted (no data modification was required because the master data had been supplemented).

[0132] The rectified data has been re-entered into the restricted area, batch number "MES_20260624_001-R1". The system is re-performing three verifications: Integrity pre-check: COMPLETE_QTY for 40 records has been added and all have passed.

[0133] Standardized pre-inspection: All passed.

[0134] Business logic pre-check: The device numbers of the 12 records already exist in the updated master data table, and all have passed.

[0135] The re-inspection pass rate was 100%. The system automatically triggered the lake entry task, and 4,935 qualified data (Note: Of the original 4,987 data, 52 were abnormal records. After rectification, 52 were qualified records. However, during the actual rectification process, 5 were submitted repeatedly, and finally 4,982 data were entered into the lake. The slight difference in data volume is normal business operation) were written to the ODS layer.

[0136] Step 6: Feedback Updates and Rule Evolution The system collects the execution effect parameters for this batch: Date format automatic repair success rate: 100% (63 / 63), higher than the fifth preset threshold of 90%; Automatic repair success rate of enumerated values: 100% (5 / 5), higher than the fifth preset threshold of 90%; REMARK field fill rate: 35%, lower than the sixth preset threshold of 95%; COMPLETE_QTY field fill rate: 100% after rectification in this batch.

[0137] Because the date format conversion rule and enumeration value standardization rule achieved a repair success rate of over 92% in four consecutive batches, the system upgraded these two rules from "online repair rules" to "pre-conversion rules." Starting with the next batch, the production completion data of the MES system will undergo date format unification and enumeration value standardization conversion automatically by the pre-conversion engine before entering the quality gate restricted area.

[0138] The system updates the field validation priority based on the historical failure rate: the historical failure rate of the device number field is 8.5%, which is lower than the third preset threshold of 30%, so it maintains the normal priority; the COMPLETE_QTY field has been temporarily promoted to high priority because there are null values ​​in this batch.

[0139] Step 7: Quality Audit The system records a complete audit log: Verification execution log: Verification started at 09:30:00 on 2026-06-24 and completed at 09:32:15, taking 135 seconds; Adaptive log processing: 63 log entries were fixed by calling the date format unification adapter, and 5 log entries were fixed by calling the enumeration value standardization adapter; Rule Change Log: Date format conversion rule upgraded to pre-conversion rule, effective 2026-06-25 00:00:00; Lake entry status log: 2026-06-24 15:20:00 completed lake entry, 4982 entry records.

[0140] In this embodiment, 52 non-compliant data entries were intercepted through a three-tiered quality gate check, preventing low-quality data from entering the ODS layer. Through adaptive processing, 68 format errors were automatically corrected without manual intervention, reducing manual rectification workload by approximately 56.7% (68 / 120). Through rule self-evolution, date format conversion and enumeration value standardization were upgraded from "online repair per batch" to "automatic pre-conversion," reducing the time for these two checks in subsequent batches from an average of 12 seconds to 0.8 seconds, improving efficiency by 93%. The overall data quality entering the lake improved from a 98.96% pass rate in the initial check to 100% after the second check.

[0141] This invention can be widely applied to data platform construction, data lake construction, and data governance projects in various enterprises, and is particularly suitable for large manufacturing, financial, and energy companies that need to connect data from multiple source systems to a unified data platform. This invention can be integrated and deployed with existing data extraction tools (such as DataX, Sqoop, and Kafka), data platform products (such as Alibaba Cloud DataWorks and Huawei Cloud DGC), and data governance platforms, demonstrating broad industrial applicability.

[0142] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for automatic verification and closed-loop rectification of intelligent quality access control before entering the lake, characterized in that: include: The data acquisition step involves obtaining source data to be added to the lake from at least one source system. The data writing gate restricted area step involves writing the source data into an independent data cleaning and verification area set between the source system and the data middle platform operation data storage layer; The steps for establishing a quality verification rule base include configuring the quality verification rule base, which contains integrity verification rules, standardization verification rules, and business logic verification rules for each field. Each rule is configured with a corresponding rule threshold, verification strictness level, and field verification priority. The quality verification step involves performing integrity verification, standardization verification, and business logic verification on the source data within the independent data cleaning and verification area according to the quality verification rule base, generating verification results and anomaly distribution characteristics for each field; The integrity check is used to check whether the field is empty and the field fill rate; the standardization check is used to check whether the field format conforms to the preset format template. The business logic validation is used to check whether the business logic relationship between cross fields and / or cross tables is valid; The adaptive processing step involves performing at least one of the following adaptive actions based on the verification result and the anomaly distribution characteristics: (a) When the integrity check detects that the null value rate of the target field exceeds the first preset threshold, the strictness level of the check of the target field in the normative check is reduced to skip check, and the target field is marked as pending supplementation. (b) When the normative verification detects that the frequency of format errors in the target field is lower than the second preset threshold and the error type is a preset repairable type, the format conversion adapter matching the error type is called to perform batch format cleaning on the target field, generate repaired data, and use the repaired data to replace the original abnormal data in subsequent verification, while recording the repair log; when the frequency of format errors is higher than or equal to the second preset threshold, the target field is marked as pending manual rectification. The feedback update step collects the execution effect parameters of the adaptive processing step, which include at least one of the following: field fill rate, automatic repair success rate, single batch verification time, and anomaly detection rate; and dynamically updates the rule threshold, verification strictness level, and field verification priority of the corresponding field in the quality verification rule base according to the execution effect parameters. The lake entry control step involves writing data that passes the quality check to the operational data storage layer, intercepting data that fails the quality check and generating an anomaly record. The iterative governance step uses the updated quality verification rule base from the feedback update step as the basis for quality verification of the next batch of source data, and repeats the quality verification step until the feedback update step.

2. The method according to claim 1, characterized in that, In the adaptive processing step, the preset repairable types include date format separator inconsistency type and encoding prefix missing type; the format conversion adapter includes date format unification adapter and encoding prefix completion adapter; When the normative check detects that the date format of the target field is a mixture of "YYYY / MM / DD" and "YYYY-MM-DD", the date format unification adapter is called to convert all dates to "YYYY-MM-DD" format. When the normative verification detects that the target field is missing a preset prefix, the encoding prefix completion adapter is invoked to automatically complete the preset prefix for the encoding field that is missing the preset prefix; The adaptive processing step also includes a post-repair consistency verification sub-step: performing consistency verification on the data repaired by the format conversion adapter, and replacing the original abnormal data with the repaired data that passes the verification in subsequent verifications, while marking the data that fails the verification as pending manual review.

3. The method according to claim 1, characterized in that, The feedback update step dynamically updates the field validation priority based on the execution effect parameters, specifically including: The failure rate of each field in the historical re-inspection process was statistically analyzed. Fields with a failure rate higher than the third preset threshold are marked as high-risk fields and prioritized for quality verification at the front of the verification queue in the next batch of verification. Fields with a failure rate lower than the fourth preset threshold in consecutive preset batches are marked as low-risk fields, and are moved to the end of the verification queue in the next batch of verification, while the verification sampling rate of the low-risk fields is reduced to the preset sampling ratio.

4. The method according to claim 1, characterized in that, The feedback update step dynamically updates the quality verification rule base based on the execution effect parameters, specifically including: When the automatic repair success rate is higher than the fifth preset threshold in N consecutive batches, the format conversion rule corresponding to the automatic repair success rate is upgraded from "online repair rule" in the quality verification rule library to "pre-conversion rule", so that the source data of subsequent batches will be automatically converted by the pre-conversion engine before entering the independent data cleaning and verification area; N is a positive integer greater than or equal to 3; When the field fill rate is higher than the sixth preset threshold in M ​​consecutive batches, the strictness level of the field integrity check is reduced, the field integrity check rule is changed from "mandatory check" to "sampling check", and the sampling check ratio is set; M is a positive integer greater than or equal to 3.

5. The method according to claim 1, characterized in that, The business logic validation includes at least one of the following business logic relationships: The consistency between order amount and the calculated results of quantity, unit price, discount, and tax rate; The chronological relationship between the date of warehousing and the date of production completion; The existence relationship of supplier codes in purchase orders in the master data; The relationship between the quantity of materials requisitioned and the planned quantity of materials to be used; The existence of customer codes in customer master data.

6. The method according to claim 1, characterized in that, The anomaly record includes the following information items: data batch number, source system identifier, source table identifier, target table identifier, anomaly field name, anomaly type identifier, hit anomaly rule number, number of anomaly records, anomaly sample data, responsible department identifier, and processing status identifier; The method also includes an audit trail step, which records the execution log of the quality verification step, the repair log and action log of the adaptive processing step, the rule change log of the feedback update step, and the lake entry status log of the lake entry control step, forming a traceable quality management audit chain. The method also includes a data quality health scoring step: calculating a comprehensive quality score for each data table based on the abnormal records and the execution effect parameters, and using the comprehensive quality score as a weighting factor for adjusting the field validation priority in subsequent batches.

7. The method according to claim 1, characterized in that, It also includes closed-loop rectification steps: Based on the responsible department identifier in the anomaly record, an anomaly rectification notice is sent to the corresponding responsible party. Receive the rectified data, rewrite the rectified data into the independent data cleaning and verification area, and execute the quality verification steps again according to the quality verification rule base; If the verification passes, the data is written to the operation data storage layer by the lake entry control step; if the verification fails, the system continues to intercept and push the rectification notice again until the rectification is completed or the preset rectification limit is reached.

8. A governance system for quality access control before data enters a lake, characterized in that, include: The data extraction module is used to obtain source data to be added to the lake from at least one source system; The quality gate restricted area module is located between the source system and the data middle platform operation data storage layer, and is used to receive and temporarily store the source data; The quality verification rule base module is used to store and manage integrity verification rules, normative verification rules, and business logic verification rules for each field, as well as the rule threshold, verification severity level, and field verification priority for each rule; The quality verification module is used to perform integrity verification, standardization verification, and business logic verification on the source data within the quality gate restricted area module according to the rules in the quality verification rule base module, and generate the verification results and anomaly distribution characteristics of each field; An adaptive execution module is used to adjust the verification strictness level of the corresponding field in the quality verification rule base module according to the verification result and the abnormal distribution characteristics, and / or perform online format cleaning on abnormal data; The effect feedback module is used to collect the execution effect parameters of the adaptive execution module and send a rule update instruction to the quality verification rule base module according to the execution effect parameters. The execution effect parameters include at least one of field filling rate, automatic repair success rate, single batch verification time and anomaly detection rate. The lake entry control module is used to write data that passes the quality check into the operation data storage layer, intercept data that fails the quality check, and generate an anomaly record; The iteration control module is used to use the updated rules from the quality verification rule base module as the verification basis for the next batch of source data.

9. The system according to claim 8, characterized in that, Also includes: The anomaly management module is used to push anomaly rectification notices to the corresponding responsible parties based on the responsible department identifier in the anomaly record, and to receive the data after rectification. The lake entry re-inspection module is used to rewrite the rectified data into the quality gate restricted area module, and the quality verification module performs quality verification again. After the verification is passed, the lake entry control module writes the data into the operation data storage layer. The audit trail module is used to record the execution log of the quality verification module, the repair log of the adaptive execution module, the rule change log of the effect feedback module, and the lake entry status log of the lake entry control module.

10. A governance device for quality access control of data before it enters a lake, characterized in that, include: One or more processors; A storage device is used to store one or more programs and a preset quality verification rule base. The quality verification rule base includes integrity verification rules, normative verification rules and business logic verification rules for each field. Each rule is configured with a corresponding rule threshold, verification strictness level and field verification priority. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 7.

Images