Data quality monitoring method and system, electronic equipment and storage medium

Abstract

The invention discloses a data quality monitoring method and system, electronic equipment and a storage medium, and relates to the technical field of data processing. The method comprises the following steps: acquiring a data flow fluctuation curve of a target data table, and constructing a data quality baseline library according to the data flow fluctuation curve; extracting abnormal data features based on the data quality baseline library, and generating a feature defect matrix corresponding to the abnormal data features; screening rule atoms from a preset rule pool according to the characteristic defect matrix, and constructing a rule atom link corresponding to the rule atoms; performing service scene adaptation on the rule atomic link to obtain a corresponding quality monitoring rule set; and performing data scanning on the target data table based on the quality monitoring rule set to obtain an abnormal data scanning result, and sending the abnormal data scanning result to a corresponding associated person. By implementing the technical scheme provided by the invention, the effect of improving the accuracy of data quality monitoring is achieved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data quality monitoring method, system, electronic device and storage medium. Background Art

[0002] With the advent of the big data era, the scale of enterprise data has exploded, and data has become a core asset for enterprises. Data quality directly affects an enterprise's operational efficiency and decision-making accuracy, making data quality monitoring particularly important.

[0003] Currently, the data quality monitoring method commonly used in the industry is mainly based on pre-set rules for data detection. This method generally establishes a series of fixed monitoring rules and then regularly executes these fixed monitoring rules on the data to identify abnormal data. However, in actual applications, due to the complexity and variability of the data environment, existing data quality monitoring methods suffer from insufficient flexibility. When only fixed rules are used to monitor data anomalies in different scenarios, errors are often included, which reduces the accuracy of data quality monitoring. Summary of the Invention

[0004] The present application provides a data quality monitoring method, system, electronic device and storage medium, which have the effect of improving the accuracy of data quality monitoring.

[0005] In a first aspect, the present application provides a data quality monitoring method, comprising: Obtaining a data flow fluctuation curve of a target data table, and building a data quality baseline library based on the data flow fluctuation curve; Extracting abnormal data features based on the data quality baseline library, and generating a feature defect matrix corresponding to the abnormal data features; Screening rule atoms from a preset rule pool according to the characteristic defect matrix, and constructing rule atom links corresponding to the rule atoms; Adapting the rule atomic link to the business scenario to obtain a corresponding quality monitoring rule set; The target data table is scanned based on the quality monitoring rule set to obtain abnormal data scanning results, and the abnormal data scanning results are sent to corresponding associated personnel.

[0006] In a second aspect of the present application, a data quality monitoring system is provided, the system comprising: A baseline library determination module is used to obtain a data flow fluctuation curve of a target data table and construct a data quality baseline library based on the data flow fluctuation curve; A link determination module is configured to extract abnormal data features based on the data quality baseline library and generate a feature defect matrix corresponding to the abnormal data features; screen rule atoms from a preset rule pool according to the feature defect matrix, and construct rule atom links corresponding to the rule atoms; A rule set determination module is used to adapt the rule atomic link to the business scenario to obtain a corresponding quality monitoring rule set; The quality monitoring module is used to perform data scanning on the target data table based on the quality monitoring rule set, obtain abnormal data scanning results, and send the abnormal data scanning results to corresponding associated personnel.

[0007] In a third aspect of the present application, an electronic device is provided, comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein the program can implement a data quality monitoring method when loaded and executed by the processor.

[0008] In a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements a data quality monitoring method.

[0009] In summary, one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: By adopting the above technical solution, the data flow fluctuation curve of the target data table is obtained, and a data quality baseline library is constructed based on the fluctuation curve, so that the data change characteristics can be dynamically reflected, providing a scientific judgment standard for subsequent monitoring; the constructed data quality baseline library is used to extract abnormal data features, and further generate the corresponding feature defect matrix, so as to achieve a quantitative measurement of the deviation between the actual quality of the data and the expected quality, ensuring more accurate identification of abnormal data; on this basis, by screening rule atoms from the preset rule pool and constructing rule atom links according to the feature defect matrix, each rule atom can be organically combined according to its trigger relationship and execution order, thereby improving the monitoring rules. This process makes the monitoring rules highly matched with the specific business data characteristics, thereby further improving the flexibility and adaptability of the monitoring process; finally, based on the quality monitoring rule set, the target data table is scanned, and the abnormal data scanning results obtained can not only comprehensively cover the full amount of data detection, but also realize the effective classification and timely feedback of abnormal data. By sending the abnormal results to the corresponding related personnel, it ensures that the related personnel can respond to the data quality anomalies in the shortest time, thereby improving the accuracy of data quality monitoring as a whole. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a flow chart of a data quality monitoring method provided in an embodiment of the present application; Figure 2 This is a structural diagram of a data quality monitoring system provided in an embodiment of the present application; Figure 3 This is a structural diagram of an electronic device provided in an embodiment of the present application.

[0011] Description of reference numerals: 300, electronic device; 301, processor; 302, communication bus; 303, user interface; 304, network interface; 305, memory. DETAILED DESCRIPTION

[0012] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0013] In the description of the embodiments of this application, words such as "for example" or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "for example" or "for instance" in the embodiments of this application should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "for example" or "for instance" is intended to present the relevant concepts in a concrete manner.

[0014] In the description of the embodiments of the present application, the term "multiple" means two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals. In addition, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0015] This application embodiment provides a data quality monitoring method. In one embodiment, please refer to Figure 1 , Figure 1 This is a flow chart of a data quality monitoring method provided in an embodiment of the present application. This method can be implemented using a computer program, which can be integrated into an application or run as a standalone tool application. This method can also be implemented using a single-chip microcomputer or run in a data quality monitoring system based on a von Neumann architecture. Specifically, this method can include the following steps: Step 101: Obtain a data flow fluctuation curve of a target data table, and build a data quality baseline library based on the data flow fluctuation curve.

[0016] In this embodiment of the present application, the target data table specifically refers to a data table that requires data quality monitoring. For example, in an enterprise data environment, this may be a business data table such as an order transaction table, a user information table, or a product inventory table. These data tables typically carry the enterprise's core business data, and their data quality directly affects the accuracy of business operations and decision-making analysis.

[0017] In the present embodiment, a data flow fluctuation curve refers to a curve that reflects the changing trend of the data volume of a target data table over a certain period of time. Specifically, it can be represented as a two-dimensional curve graph with time as the horizontal axis and data volume as the vertical axis. Data volume can include indicators such as the number of data records and data size. For example, the data flow fluctuation curve of an order transaction table may show periodic characteristics such as higher transaction volume on weekdays and significant fluctuations on holidays. This curve can intuitively demonstrate the usage and changing patterns of the data table.

[0018] In the embodiment of the present application, the data quality baseline library refers to a set of quality assessment benchmarks obtained by analyzing and optimizing the data flow fluctuation curve.

[0019] Specifically, since the key to data quality monitoring lies in accurately identifying data anomalies, and determining data anomalies requires a reasonable benchmark as a reference, it is first necessary to build a data quality baseline library for the target data table to provide a basis for subsequent anomaly detection. Specifically, by continuously monitoring the target data table, the changes in its data flow at different time points are collected to form a data flow fluctuation curve. This curve can intuitively reflect characteristics such as the data volume, update frequency, and fluctuation patterns of the data table. After obtaining the data flow fluctuation curve, the upstream and downstream data dependencies of the target data table are further analyzed, and the data flow process between different systems or modules is analyzed. Based on this, a data lineage map representing data associations is generated. Based on the data lineage map, key data nodes that have a significant impact on the quality of the target data table can be identified and the data flow characteristics of these key data nodes can be extracted. By analyzing the data flow characteristics of key data nodes, the node impact factor reflecting the degree of influence of each node on the target data table is calculated.

[0020] Subsequently, the node impact factor was used to calibrate the previously acquired data flow fluctuation curve. The quality attenuation parameter of the node impact factor was extracted, and the data flow fluctuation curve was weighted based on the quality attenuation parameter to obtain a weighted data curve. After determining the fluctuation range of the weighted data curve, a quality baseline threshold was determined based on the fluctuation range. The weighted data curve was segmented using this threshold to obtain multiple segmented quality baselines.

[0021] Finally, the segmented quality baselines are integrated and mapped to create a data quality baseline library. This quality baseline library, built based on data flow characteristics and data dependencies, accurately reflects the normal data characteristics of the target data table in different scenarios, providing a reliable reference standard for subsequent anomaly detection and effectively improving the accuracy of data quality monitoring.

[0022] Based on the above embodiment, as an optional embodiment, in step 101: constructing a data quality baseline library according to the data flow fluctuation curve, this step may also include the following steps: Step 201: Obtain the upstream and downstream data dependency relationships of the target data table, and generate a data lineage graph corresponding to the upstream and downstream data dependency relationships.

[0023] Specifically, to accurately assess data quality, a deep understanding of the relationships between the target data table and other data tables is necessary. To implement this, the database metadata is first analyzed to identify the target data table's data source tables (upstream tables) and data destination tables (downstream tables). For example, the upstream of an order transaction table might include user information tables and product information tables, while the downstream might include order payment tables and logistics distribution tables. Subsequently, by tracking field mapping relationships, data conversion rules, and other information during data processing, the data transfer paths and dependencies between related tables are determined. Based on the obtained dependency information, a data lineage graph is constructed using a graph-structured approach. The graph's nodes represent individual data tables, and the edges represent the dependencies between tables. Edge attributes can include information such as dependency type and associated fields. This visual data lineage graph can intuitively demonstrate the data flow path between different tables, helping to understand the scope and impact of data quality issues.

[0024] Step 202: Identify key data nodes based on the data lineage graph and extract data flow characteristics corresponding to the key data nodes.

[0025] Specifically, in order to determine the key monitoring targets, it is necessary to identify the key nodes that have a greater impact on the quality of the target data table from the data lineage map. In the specific implementation, the criticality indicators of each node are first calculated based on the graph analysis algorithm, including the node's in-degree, out-degree, centrality and other characteristic values. For example, the data quality problems of nodes with more downstream dependencies may have a wider range of impacts. At the same time, combined with the business importance assessment, data nodes with important business value are determined. For the identified key data nodes, their data traffic characteristics are further extracted, including data update frequency, data magnitude, data distribution characteristics, etc. For example, the data processing volume, data increment ratio, abnormal fluctuation and other indicators of the nodes in different time periods are statistically analyzed. These characteristics can reflect the data processing mode and quality change law of key nodes, and provide a basis for the subsequent calculation of influencing factors.

[0026] Step 203: Calculate the node impact factor according to the data traffic characteristics, and use the node impact factor to calibrate the data traffic fluctuation curve to generate a data quality baseline library.

[0027] Specifically, to accurately assess the impact of key nodes on the target data table, it is necessary to calculate node impact factors based on extracted data traffic characteristics. In this implementation, an impact factor calculation model is first constructed, which comprehensively considers characteristic indicators such as the node's data volume, update frequency, and degree of abnormal fluctuation. By assigning weights to different characteristic indicators, a quantitative value reflecting the strength of each node's influence, namely the node impact factor, is calculated. The calculated node impact factor is then used to calibrate the data traffic fluctuation curve. During the calibration process, the data points in the original data traffic fluctuation curve are weighted and adjusted according to the node impact factor, so that the curve more accurately reflects the changing patterns of data quality. For example, data fluctuations during periods with larger impact factors may require stricter monitoring standards. Finally, based on the calibrated fluctuation curve, combined with quality decay parameters and fluctuation range thresholds, a baseline library that accurately reflects data quality characteristics is constructed. This baseline library, which accounts for data dependencies and node influence, can provide a more reliable evaluation standard for data quality monitoring.

[0028] Based on the above embodiment, as an optional embodiment, in step 203: using the node impact factor to calibrate the data flow fluctuation curve to generate a data quality baseline library, this step may also include the following steps: Step 213: extracting the quality attenuation parameter of the node influence factor; performing weighted calculation on the data flow fluctuation curve according to the quality attenuation parameter to obtain a weighted data curve.

[0029] Specifically, to accurately reflect the decay effect of node influence factors over time, it is necessary to extract quality decay parameters and perform weighted calculations. In practice, the timeliness characteristics of node influence factors are first analyzed based on historical data to construct a decay function model. For example, an exponential decay function can be used to calculate the decay coefficients for different time periods, gradually reducing the impact of data from earlier time periods. The extracted quality decay parameters include indicators such as decay rate and periodic factor, which reflect the pattern of data quality changes over time. Subsequently, the quality decay parameters are applied to the weighted calculation of the data flow fluctuation curve, dynamically adjusting the data values ​​at each time point on the curve. Specifically, the original value of the data point is multiplied by the decay coefficient for the corresponding time period to obtain a weighted data value that takes timeliness into account, ultimately forming a weighted data curve that reflects the actual data quality status. This weighted processing method based on quality decay characteristics can more accurately characterize the dynamic changes in data quality.

[0030] Step 223: Determine the fluctuation range of the weighted data curve, and determine the quality baseline threshold according to the fluctuation range.

[0031] Specifically, to establish reasonable quality monitoring standards, it is necessary to analyze the fluctuation characteristics of the weighted data curve and determine the corresponding threshold range. In practice, the weighted data curve is first statistically analyzed to calculate the statistical characteristics of data fluctuations, such as the mean, standard deviation, and coefficient of variation. Based on these statistical indicators, the normal fluctuation range of the data is determined. For example, the mean plus or minus a certain number of standard deviations can be used as the boundaries of the fluctuation range. After determining the fluctuation range, further consideration is given to business tolerance and risk level to set different levels of quality baseline thresholds. For example, warning thresholds and severe anomaly thresholds can be set to correspond to different degrees of data quality issues. This threshold determination method based on data statistical characteristics and business needs can provide scientific and reasonable judgment criteria for anomaly detection.

[0032] Step 233: Segment the weighted data curve using the quality baseline threshold to obtain multiple segmented quality baselines; integrate and map the segmented quality baselines to obtain a data quality baseline library.

[0033] Specifically, to adapt to data quality characteristics in different scenarios, the weighted data curve needs to be segmented and integrated to form a complete baseline library. In practice, the weighted data curve is first segmented based on the established quality baseline threshold. For example, segmentation can be based on factors such as time characteristics (e.g., weekdays / holidays) and business characteristics (e.g., peak / low trading periods), ensuring that each segment has relatively stable data characteristics. For each segment, characteristic parameters (e.g., mean, fluctuation range, and periodicity) are extracted to form a corresponding segmented quality baseline. Finally, the segmented quality baselines are integrated and mapped to establish a hierarchical data quality baseline library. During the integration process, the continuity and transition characteristics between segments need to be considered to ensure that the baseline library smoothly describes data quality changes. This segmented processing and integrated mapping approach enables the construction of a more flexible and accurate data quality assessment benchmark, effectively improving the adaptability and accuracy of monitoring.

[0034] Step 102: Extract abnormal data features based on the data quality baseline library, and generate a feature defect matrix corresponding to the abnormal data features.

[0035] In this embodiment, abnormal data characteristics refer to data performance characteristics that deviate from the baseline quality indicators in the data quality baseline library. Specifically, they may include abnormal performance in dimensions such as field type, data length, value range, and association relationships. These characteristics reflect the specific manifestations of data quality issues. By analyzing these characteristics, the type and source of data quality issues can be accurately identified.

[0036] In the embodiments of this application, the feature defect matrix refers to a multi-dimensional scoring matrix obtained after quantitatively evaluating the abnormal data features. The rows of the matrix represent different abnormal data records or fields, the columns represent different feature dimensions, and the matrix element values ​​represent the abnormality scores of the corresponding features. By mapping these scores with preset quality standards, the severity and impact of data quality issues can be comprehensively assessed. The feature defect matrix can not only quantitatively express data quality issues, but also provide a basis for subsequent rule screening and monitoring strategy optimization.

[0037] Specifically, to accurately identify data quality issues, it is necessary to extract abnormal data features based on a pre-established data quality baseline database and quantitatively assess the degree of abnormality. To implement this, baseline quality indicators are first obtained from the database. These indicators include data quality assessment dimensions such as null value rate, duplicate value rate, value range distribution, and update frequency. For example, for an order transaction table, the baseline quality indicators may specify the normal value range for the order amount field and the allowable null value rate for the order status field. Subsequently, the actual quality indicators for each field in the target data table are collected and calculated, and a difference analysis is performed between these actual indicators and the baseline quality indicators. By setting up a difference assessment model and calculating the degree of deviation of each quality indicator, abnormal fields and their degree of abnormality are identified. For each identified abnormal field, its data features are further extracted as abnormal data features, including feature dimensions such as field type, data length, value range, and relationships with other fields. For example, for the abnormal order amount field, its value distribution characteristics and change trend characteristics need to be extracted. To quantitatively represent the abnormal features, the extracted abnormal data features are quantitatively calculated and a feature scoring matrix is ​​constructed.

[0038] During the quantification process, corresponding scoring rules are applied to different types of features. For example, numerical features can be used to calculate the degree of outliers, while categorical features can be used to calculate the degree of distribution deviation. Finally, the feature scoring matrix is ​​mapped and compared with the preset quality standards to generate a feature defect matrix that reflects the extent and type of data quality defects. This anomaly identification method based on multi-dimensional feature analysis can comprehensively and objectively assess data quality, providing a basis for subsequent rule screening and monitoring strategy formulation. The construction of the feature defect matrix not only reflects the specific manifestations of abnormal data but also quantifies the severity of the anomaly, effectively improving the accuracy and operability of data quality monitoring.

[0039] Based on the above embodiment, as an optional embodiment, in step 102: extracting abnormal data features based on the data quality baseline library and generating a feature defect matrix corresponding to the abnormal data features, this step may also include the following steps: Step 301: Obtain the benchmark quality indicators in the data quality baseline library, which include the null value rate, duplicate value rate, value domain distribution range and update frequency; and perform difference analysis between the actual quality indicators of each field in the target data table and the benchmark quality indicators.

[0040] Specifically, to accurately assess data quality, it is necessary to obtain standardized quality assessment benchmarks based on a data quality baseline library. This approach begins by extracting baseline quality indicators from the database. These indicators include null value rate, duplicate value rate, value range distribution, and update frequency. For example, for an order transaction table, the baseline quality indicators might specify that the null value rate of the order amount field should be less than 0.1%, the duplicate value rate of the customer ID field should be 0, the value range distribution of the order amount should be between 0 and 1 million, and the update frequency of transaction records should be between 100 and 1000 per minute. Subsequently, through data scanning and statistical analysis, the actual quality indicators for each field in the target data table are calculated. During the variance analysis process, statistical methods are used to calculate the degree of deviation between the actual and baseline indicators. For example, this method, based on multi-dimensional indicators, can comprehensively reveal the specific manifestations of data quality issues.

[0041] Step 302: Identify abnormal fields based on the difference analysis results, and extract data features of the abnormal fields as abnormal data features. The abnormal data features include field type, data length, value range, and association relationship.

[0042] Specifically, to accurately locate data quality issues, it is necessary to identify abnormal fields and extract their features based on the results of difference analysis. During implementation, a difference threshold standard is first set, and fields that exceed the threshold are marked as abnormal fields. For example, when the actual null value rate of a field exceeds 150% of the baseline value, it is identified as an abnormal field. For identified abnormal fields, their data features are further extracted, including field type (such as numeric type, character type), data length (such as the distribution of string length), value range (such as the upper and lower bounds of the value), and associations with other fields (such as foreign key associations, business logic associations, etc.). For example, for an abnormal order amount field, it is necessary to extract its value type, precision requirements, valid value range, and association constraints with fields such as order quantity and unit price. This multi-dimensional feature extraction method can accurately characterize the characteristic performance of abnormal data and provide a basis for subsequent quantitative evaluation.

[0043] Step 303: Quantitatively calculate the abnormal data features to obtain a feature scoring matrix; map the feature scoring matrix to a preset quality standard to generate a feature defect matrix.

[0044] Specifically, to achieve a quantitative assessment of the degree of anomaly, it is necessary to quantify the extracted abnormal data features and establish a scoring system. During implementation, a feature quantification model is first constructed, and corresponding calculation methods are designed for different types of features. For example, a standardized score can be calculated for numerical features, an abnormality probability score can be calculated for categorical features, and a violation degree score can be calculated for associated features. Through feature quantification calculation, a feature scoring matrix is ​​generated, and each element in the matrix represents the abnormality degree score of a specific abnormal record on a specific feature dimension. Subsequently, the feature scoring matrix is ​​mapped and compared with the preset quality standards, which define the quality levels and processing strategies corresponding to different score ranges. Through mapping and transformation, a feature defect matrix is ​​finally generated. This matrix contains not only the quantitative score of the degree of anomaly, but also the type label and processing suggestions of quality issues. This method based on quantitative scoring and standard mapping can transform qualitative abnormal features into measurable and comparable quantitative indicators, providing a decision-making basis for precise data quality management.

[0045] Step 103: Filter rule atoms from a preset rule pool according to the characteristic defect matrix, and construct rule atom links corresponding to the rule atoms.

[0046] Among them, the preset rule pool in the embodiment of this application refers to a collection library of pre-configured data quality check rules. This rule pool contains various types of data quality check rules for dealing with data quality issues in different scenarios. For example, the rule pool may contain data integrity rules (check whether required fields are empty), data accuracy rules (check whether values ​​are within a reasonable range), data consistency rules (check whether associated field values ​​match), etc. These rules are systematically organized and managed to facilitate rule screening and calling based on specific quality issues.

[0047] In the embodiments of this application, a rule atom refers to the smallest rule unit that can be executed independently in the rule pool. Each rule atom has a specific check function and applicable conditions. For example, a "null value check rule atom" is used to check whether a field is empty, a "format validation rule atom" is used to check whether the data format meets the requirements, and a "range check rule atom" is used to check whether the value is within a specified range. The design of rule atoms follows the "single responsibility" principle. Each rule atom is only responsible for a specific check task. This atomic design allows rules to be flexibly combined and reused.

[0048] In the embodiments of this application, a rule atom link refers to a rule execution path that organizes multiple rule atoms according to a specific execution order and logical relationship. The rule atom link defines the dependencies and execution order between rule atoms, forming a complete inspection process. For example, the inspection of order data may require executing multiple rule atoms in the order of "field integrity check → data format check → value range check → business rule check". Through the design of the rule atom link, the logic of rule execution is guaranteed and the efficiency of rule inspection is improved.

[0049] Specifically, to effectively address data quality issues, it's necessary to select appropriate quality rules based on the characteristic defect matrix and build a rule execution chain. During implementation, the characteristic defect matrix is ​​first analyzed for anomaly types and severity to identify the quality issues that require priority. For example, if the characteristic defect matrix reveals a serious value range anomaly in the order amount field, rules related to value range verification should be prioritized. Subsequently, rule atoms are selected from a pre-set rule pool, which contains the basic units of various data quality check rules. Rule atoms are the smallest independently executable rule units, such as null value check rule atoms, format validation rule atoms, and value range rule atoms. During rule selection, a rule matching algorithm is used to match the anomaly characteristics in the characteristic defect matrix with the applicable conditions of the rule atoms, selecting the set of rule atoms most suitable for addressing the current quality issue. For example, for anomalies in order amount, rule atoms related to "value range verification," "accuracy check," and "trend analysis" may need to be selected. After obtaining the rule atoms, the logical relationships and execution dependencies between the rule atoms are further analyzed to build a rule atom chain. Rule atomic links define the execution order and combination of rule atoms, ensuring the integrity and efficiency of rule checking. For example, checking order amounts might require executing rule atoms in the order of "null value check → format verification → range check → association verification." Building rule atomic links not only ensures the logical rationality of rule execution but also improves the efficiency of rule checking. This feature-based rule screening and link building approach enables precise matching and efficient organization of data quality rules, providing effective rule support for subsequent quality monitoring.

[0050] Based on the above embodiment, as an optional embodiment, in step 103: screening rule atoms from a preset rule pool according to the characteristic defect matrix and constructing rule atom links corresponding to the rule atoms, this step may further include the following steps: Step 401: Calculate the rule matching degree according to the defect features in the feature defect matrix. The rule matching degree is used to indicate the degree of adaptation between the rule atom and the defect feature.

[0051] Specifically, in order to accurately select rule atoms that are suitable for handling current data quality issues, it is necessary to quantitatively evaluate the degree of match between rule atoms and defect features. In specific implementation, defect feature information is first extracted from the feature defect matrix, including dimensions such as anomaly type, anomaly degree, and impact range. Then, a rule matching calculation model is constructed, which takes into account factors such as the applicable conditions, inspection scope, and expected effects of rule atoms. When calculating the rule matching degree, a weighted scoring method is used to quantitatively evaluate the matching conditions of different dimensions. For example, when the feature defect matrix shows that there is a value range anomaly in the order amount field, the system will calculate the processing ability score of each rule atom for this type of anomaly, the rule execution cost score, etc., and calculate the comprehensive matching degree according to the preset weights. This multi-dimensional matching calculation method can objectively evaluate the applicability of rule atoms and provide a quantitative basis for rule screening.

[0052] Step 402: Select target rule atoms whose rule matching degree exceeds a matching degree threshold from a preset rule pool; and construct a rule atom dependency graph based on the triggering relationship between the target rule atoms.

[0053] Specifically, to identify the most suitable rule atoms and clarify the relationships between rules, rule selection is performed based on rule matching and inter-rule dependency analysis. In this implementation, a matching threshold is first set. Rule atoms with matching degrees exceeding the threshold are selected from the preset rule pool as target rule atoms. For example, if the rule matching threshold is set to 0.8, the system selects all rule atoms with matching degrees greater than 0.8. For these selected target rule atoms, the system further analyzes their triggering relationships, including pre-dependencies (one rule must execute after another) and mutual exclusions (two rules cannot execute simultaneously). Based on these triggering relationships, a rule atom dependency graph is constructed, representing rule atoms as nodes and inter-rule dependencies as edges. For example, the "Numeric Format Check" rule atom must execute before the "Numeric Range Check" rule atom. This dependency is represented by a directed edge in the dependency graph. This dependency graph-based representation of rule relationships intuitively displays the constraints governing rule execution, providing a basis for determining the subsequent execution order.

[0054] Step 403: Determine the rule execution order in the rule atom dependency graph, and connect the rule atoms into rule atom links according to the rule execution order.

[0055] Specifically, to ensure the logic and efficiency of rule checking, it is necessary to determine the specific execution order and construct a rule atomic link based on the rule atomic dependency graph. In implementation, the rule atomic dependency graph is first topologically sorted to determine the rule execution sequence that satisfies all dependency constraints. During the sorting process, factors such as rule priority, execution efficiency, and resource consumption are considered. For example, basic rules with low computational cost are prioritized, while complex rules with strong dependencies are prioritized. After the execution order is determined, the rule atoms are connected in this order to form a complete rule atomic link. The rule atomic link not only includes the rule execution order but also the data transfer method and exception handling strategy between rules. For example, the execution result of a previous rule may affect the execution parameters of subsequent rules, and this data flow relationship is also reflected in the link. This rule link construction method based on dependency analysis ensures the correctness of rule execution and improves the efficiency of rule checking.

[0056] Step 104: Adapt the rule atomic link to the business scenario to obtain a corresponding quality monitoring rule set.

[0057] Among them, the quality monitoring rule set in the embodiment of the present application refers to a complete rule combination that can be directly applied to actual data quality monitoring after being adapted to the business scenario.

[0058] Specifically, to better align rule checking with actual business needs, the constructed rule atomic links need to be adapted to the business scenario, thereby forming a directly applicable quality monitoring rule set. During implementation, the business scenario characteristics of the target data table are first analyzed, including business constraints such as business process requirements, data processing timeliness, and quality tolerance. For example, for an order transaction table, it is necessary to consider the special processing requirements of different transaction types, performance requirements during peak trading periods, and data quality standards for different business processes. Subsequently, the rule atomic links are parameterized and their execution strategies optimized based on the business scenario characteristics. During parameter tuning, specific thresholds and judgment conditions for rule atoms are set based on the business scenario. For example, differentiated checking criteria are set for orders with different amount ranges, and different update frequency requirements are set for transactions in different time periods. During execution strategy optimization, the trigger conditions, execution frequency, and exception handling methods for rule execution are adjusted based on business characteristics. For example, an incremental rule checking strategy is configured for real-time transaction data, while a batch rule checking strategy is configured for historical inventory data. Through these adaptations and optimizations, the rule atomic links are transformed into a quality monitoring rule set that is deeply integrated with the business scenario. This rule set encompasses not only the rule checking logic but also execution strategies and processing mechanisms tailored to the business scenario. This ensures accuracy while meeting the performance and availability requirements of the business system. By adapting to business scenarios, the quality monitoring rule set can both accurately identify data quality issues and smoothly integrate into business systems, effectively enhancing the practicality of data quality monitoring.

[0059] Based on the above embodiment, as an optional embodiment, in step 104: adapting the rule atomic link to the business scenario to obtain the corresponding quality monitoring rule set, this step may also include the following steps: Step 501: Obtain preset business data feature tags, which include data source type, data update cycle, data sensitivity, and data application scenario.

[0060] Specifically, to accurately identify data features across different business scenarios, it's necessary to obtain pre-set business data feature labels as the basis for scenario classification. In practice, the system first extracts pre-set business data feature labels. These labels include dimensions such as data source type (e.g., real-time data collection, batch data import), data update cycle (e.g., real-time update, daily update, weekly update), data sensitivity (e.g., core transaction data, user privacy data), and data application scenario (e.g., transaction settlement, report statistics). For example, order transaction data might be labeled with feature labels such as "real-time collection, minute-level update, core sensitivity, real-time settlement." The acquisition of these labels comprehensively considers the actual operational characteristics and data processing requirements of the business system, providing a clear classification basis for subsequent scenario adaptation. This scenario description approach based on multi-dimensional feature labels comprehensively characterizes the data features of different business scenarios, providing a precise scenario profile for rule adaptation.

[0061] Step 502: categorize rule atomic links by scenario based on service data feature labels to obtain adaptation rule subsets for multiple types of scenarios.

[0062] Specifically, to better adapt rule checking to different business scenarios, rule atomic links need to be categorized based on business data feature tags. This involves first establishing a mapping between feature tags and scenario types, grouping data with similar feature tag combinations into the same scenario type. For example, data with "real-time collection, minute-level updates" might be categorized as a "real-time transaction scenario," while data with "batch import, daily updates" might be categorized as an "offline statistical scenario." Subsequently, for each scenario type, rule atomic links are filtered to identify appropriate rule combinations, forming a corresponding subset of adapted rules. During this screening process, the degree to which the rule execution characteristics match the scenario requirements is considered. For example, real-time scenarios may favor lightweight checking rules, while offline scenarios may include more complex statistical rules. This scenario-based rule adaptation approach enables rule checking to better meet the specific needs of different scenarios.

[0063] Step 503: sorting the rules priorities of the adaptation rule subsets, and determining the corresponding rule execution strategy according to the rule priorities of the adaptation rule subsets; merging the adaptation rule subsets with the same rule execution strategy to generate a quality monitoring rule set.

[0064] Specifically, to achieve rational scheduling and resource optimization for rule execution, adaptation rule subsets must be prioritized and execution strategies determined. During implementation, each adaptation rule subset is prioritized based on scenario importance and rule impact. Prioritization factors include data sensitivity, business importance, and rule complexity. For example, a rule subset involving core transaction data may be assigned a higher priority, while a statistical rule subset may be assigned a lower priority. Based on rule priority, a corresponding execution strategy is set for each adaptation rule subset, including configurations such as execution frequency, resource quotas, and timeout handling. For example, high-priority rules may adopt a real-time execution strategy, while low-priority rules may adopt a scheduled batch execution strategy. Finally, adaptation rule subsets with the same execution strategy are merged to generate the final quality monitoring rule set. During the merging process, conflicts between rules must be resolved and the execution order optimized to ensure the overall execution efficiency of the rule set. This priority-based rule organization approach ensures the timely execution of important rules while ensuring the efficient use of system resources. The quality monitoring rule set generated in this way exhibits excellent execution efficiency and resource adaptability, effectively supporting the data quality management requirements of business systems.

[0065] Step 105: Scan the target data table based on the quality monitoring rule set to obtain abnormal data scanning results, and send the abnormal data scanning results to corresponding related personnel.

[0066] Among them, the abnormal data scanning result in the embodiment of the present application refers to a detailed record of all data quality abnormalities found after scanning the target data table through the quality monitoring rule set.

[0067] In the embodiment of the present application, the associated personnel refers to the person responsible for handling the discovered data quality issues.

[0068] Specifically, to promptly identify and address data quality issues, a quality monitoring rule set is required to continuously monitor the target data tables and promptly notify relevant personnel of any identified issues. This implementation begins by initiating a data scan task based on the execution strategy defined in the quality monitoring rule set. The scan process utilizes a distributed computing framework to parallelize the target data tables, improving scanning efficiency. During the scan, the system applies quality monitoring rules sequentially based on rule priority, performing multi-dimensional quality checks on the data. For example, a scan of an order transaction table might include real-time checks on order amount validity and periodic checks on transaction status consistency. When abnormal data is detected, the system records detailed information, including the location of the abnormal data, the type of rule violated, and the specific manifestations of the abnormality. Furthermore, the system prioritizes abnormalities based on severity, categorizing them as "emergency" and "warning" for minor abnormalities that impact data accuracy. After generating abnormal data scan results, the system pushes the abnormality information to the relevant personnel based on pre-defined notification policies. Associated personnel are identified based on the exception type and handling responsibilities. For example, data quality anomalies are notified to the data governance team, and business rule violations are notified to the business owner. The pushed exception information includes a description of the issue, the scope of impact, and recommended actions, enabling relevant personnel to quickly understand and address the issue. This rule-based automated scanning and precise push notification mechanism enables timely detection and efficient handling of data quality issues, effectively improving the responsiveness and efficiency of data quality management. The timely detection and handling of abnormal data not only ensures the stable operation of business systems but also provides effective support for continuous improvement of data quality.

[0069] Based on the above embodiment, as an optional embodiment, in step 105: scanning the target data table based on the quality monitoring rule set to obtain abnormal data scanning results, this step may also include the following steps: Step 601: Construct a rule-based scanning task based on the quality monitoring rule set. The rule-based scanning task includes a scanning cycle, a scanning range, and a scanning depth.

[0070] Specifically, to achieve systematic monitoring of data quality, it is necessary to construct rule scanning tasks based on the quality monitoring rule set. During implementation, key parameters for the rule scanning task must first be set based on the execution strategy for each rule in the rule set. The scanning cycle defines the frequency of rule checks, such as once per minute for real-time check rules and once per day for batch check rules. The scanning scope determines the scope of data to be checked, such as a full table scan or incremental data scan. The scanning depth specifies the level of granularity of the check, such as field-level checks or record-level checks. For example, for an order transaction table, a scanning task might be set to "check the validity of the order amount once per minute, with the scope being the most recently generated order data and a depth of field-level checks." This refined configuration of task parameters ensures the targeted and efficient nature of data quality checks.

[0071] Step 602: Perform a full data scan on the target data table based on the rule-based scanning task to obtain an initial scanning result.

[0072] Specifically, to comprehensively identify data quality issues, a full scan of the target data table is required based on a rule-based scanning task. This scanning task is executed using a distributed computing framework, which divides large-scale data into shards and processes them in parallel to improve scanning efficiency. The scanning process strictly adheres to the execution parameters defined in the rule-based scanning task, applying quality monitoring rules sequentially according to their priority. During the scan, the system records the inspection results of each rule, including detailed information on abnormal data, rule triggering conditions, and execution time. This information constitutes the initial scan results, providing foundational data for subsequent anomaly analysis. This systematic scanning mechanism enables efficient quality inspection of large-scale data.

[0073] Step 603: Perform abnormality classification on the initial scan result to obtain abnormal data corresponding to multiple abnormality levels. The abnormality classification includes severe abnormality, general abnormality and slight abnormality.

[0074] Specifically, in order to facilitate the hierarchical processing of problems, the initial scan results need to be graded for anomalies. During specific implementation, the anomaly grading standards are first formulated, and anomalies are divided into three levels: severe anomalies, general anomalies, and minor anomalies. Severe anomalies refer to problems that may cause business interruption or significant losses, such as missing core fields and errors in key data; general anomalies refer to problems that affect data usage but do not cause business interruption, such as data update delays and non-critical field anomalies; minor anomalies refer to problems that have less impact on the business, such as irregular formats and redundant data. The system automatically classifies the abnormal data in the initial scan results according to these standards and generates abnormal data sets of different levels. This hierarchical processing mechanism provides priority guidance for subsequent problem handling.

[0075] Step 604: Determine the abnormality proportion and impact range of the abnormal data corresponding to each abnormality level, and generate an abnormal data scanning result including the abnormality level, abnormality proportion and impact range.

[0076] Specifically, to comprehensively assess the data quality status, it is necessary to conduct statistical analysis of abnormal data at all levels and generate a final scan report. During implementation, the first step is to calculate the abnormality ratio corresponding to each abnormality level, that is, the proportion of abnormal data at that level to the total data volume. Then, the impact scope of each type of abnormality is analyzed, including the affected business systems, the data tables and fields involved, the affected data processing flow, etc. Based on these analysis results, an abnormal data scan result is generated, which includes the abnormality level, abnormality ratio, and impact scope. For example, the scan result may display detailed information such as "Severe abnormalities account for 1%, affecting order processing and settlement processes; general abnormalities account for 5%, affecting the accuracy of statistical reports." This multi-dimensional abnormality analysis result can help relevant personnel accurately understand the severity and impact scope of data quality issues, providing a basis for formulating treatment plans.

[0077] Reference Figure 2 , is a data quality monitoring system provided by an embodiment of the present application, the system comprising: a baseline library determination module, a link determination module, a rule set determination module, and a quality monitoring module, wherein: A baseline library determination module is used to obtain a data flow fluctuation curve of a target data table and construct a data quality baseline library based on the data flow fluctuation curve; The link determination module is used to extract abnormal data features based on the data quality baseline library and generate a feature defect matrix corresponding to the abnormal data features; based on the feature defect matrix, rule atoms are screened from the preset rule pool and rule atom links corresponding to the rule atoms are constructed; The rule set determination module is used to adapt the rule atomic link to the business scenario and obtain the corresponding quality monitoring rule set; The quality monitoring module is used to scan the target data table based on the quality monitoring rule set, obtain abnormal data scanning results, and send the abnormal data scanning results to the corresponding related personnel.

[0078] Based on the above embodiments, the baseline library determination module is also used to obtain the upstream and downstream data dependencies of the target data table, and generate a data lineage map corresponding to the upstream and downstream data dependencies; identify key data nodes based on the data lineage map, and extract data flow characteristics corresponding to the key data nodes; calculate the node impact factor based on the data flow characteristics, and use the node impact factor to calibrate the data flow fluctuation curve to generate a data quality baseline library.

[0079] Based on the above embodiment, the baseline library determination module is also used to extract the quality attenuation parameters of the node influencing factors; perform weighted calculation on the data flow fluctuation curve according to the quality attenuation parameters to obtain a weighted data curve; determine the fluctuation range of the weighted data curve, and determine the quality baseline threshold according to the fluctuation range; segment the weighted data curve according to the quality baseline threshold to obtain multiple segmented quality baselines; integrate and map each segmented quality baseline to obtain a data quality baseline library.

[0080] Based on the above embodiment, the link determination module is also used to obtain benchmark quality indicators in the data quality baseline library, which include null value rate, duplicate value rate, value domain distribution range and update frequency; perform difference analysis on the actual quality indicators of each field in the target data table and the benchmark quality indicators; identify abnormal fields based on the difference analysis results, and extract data features of the abnormal fields as abnormal data features, which include field type, data length, value range and association relationship; perform quantitative calculation on the abnormal data features to obtain a feature scoring matrix; map the feature scoring matrix with the preset quality standard to generate a feature defect matrix.

[0081] Based on the above embodiment, the link determination module is also used to calculate the rule matching degree based on the defect characteristics in the feature defect matrix, and the rule matching degree is used to indicate the degree of adaptation between the rule atom and the defect characteristics; select the target rule atom whose rule matching degree exceeds the matching degree threshold from the preset rule pool; construct a rule atom dependency graph based on the triggering relationship between the target rule atoms; determine the rule execution order in the rule atom dependency graph, and connect the rule atoms into rule atom links according to the rule execution order.

[0082] Based on the above embodiment, the rule set determination module is also used to obtain preset business data feature labels, which include data source type, data update cycle, data sensitivity and data application scenarios; based on the business data feature labels, the rule atomic links are classified into scenarios to obtain adaptation rule subsets for multiple types of scenarios; each adaptation rule subset is sorted by rule priority, and the corresponding rule execution strategy is determined according to the rule priority of each adaptation rule subset; the adaptation rule subsets with the same rule execution strategy are merged to generate a quality monitoring rule set.

[0083] Based on the above embodiment, the quality monitoring module is also used to construct a rule scanning task according to the quality monitoring rule set, and the rule scanning task includes a scanning cycle, a scanning range and a scanning depth; based on the rule scanning task, a full data scan of the target data table is performed to obtain an initial scanning result; the initial scanning result is subjected to anomaly classification to obtain abnormal data corresponding to multiple abnormality levels, and the abnormality classification includes severe abnormalities, general abnormalities and minor abnormalities; the abnormality proportion and impact range of the abnormal data corresponding to each abnormality level are determined, and an abnormal data scanning result including the abnormality level, abnormality proportion and impact range is generated.

[0084] It should be noted that the above embodiments provide devices that implement their functions using only the division of the above functional modules as examples. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiments are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0085] This application also discloses an electronic device. Figure 3 , Figure 3 The electronic device 300 may include: at least one processor 301 , at least one network interface 304 , a user interface 303 , a memory 305 , and at least one communication bus 302 .

[0086] The communication bus 302 is used to implement the connection and communication between these components.

[0087] The user interface 303 may include a display interface and a camera interface. Optionally, the user interface 303 may also include a standard wired interface and a wireless interface.

[0088] The network interface 304 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0089] The processor 301 may include one or more processing cores. Using various interfaces and circuits, the processor 301 connects to various components within the server. It executes instructions, programs, code sets, or instruction sets stored in the memory 305, as well as accesses data stored in the memory 305, to perform various server functions and process data. Optionally, the processor 301 may be implemented using at least one of the following hardware forms: a digital signal processing (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA). The processor 301 may integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily processes the operating system, user interface graphics, and application programs; the GPU is responsible for rendering and drawing content displayed on the display screen; and the modem handles wireless communications. It is understood that the modem may also be implemented as a separate chip, rather than integrated into the processor 301.

[0090] Among them, the memory 305 may include a random access memory (RAM) or a read-only memory (Read-Only Memory). Optionally, the memory 305 includes a non-transitory computer-readable storage medium. The memory 305 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 305 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 305 may also optionally be at least one storage device located away from the aforementioned processor 301. Refer to Figure 3 , the memory 305 as a computer storage medium may include an operating system, a network communication module, a user interface module and an application program of a data quality monitoring method.

[0091] exist Figure 3In the electronic device 300 shown, the user interface 303 is mainly used to provide an input interface for the user and obtain the data input by the user; and the processor 301 can be used to call an application program storing a data quality monitoring method in the memory 305. When executed by one or more processors 301, the electronic device 300 executes one or more methods in the above-mentioned embodiments. It should be noted that for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should know that this application is not limited to the described order of actions, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for this application.

[0092] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0093] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are merely schematic, such as the division of units, which is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interface, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0094] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0095] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0096] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of this application, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the method of this application. The aforementioned memory includes various media that can store program code, such as USB flash drives, mobile hard drives, magnetic disks, or optical disks.

[0097] The above are merely exemplary embodiments of the present disclosure and are not intended to limit the scope of the present disclosure. In other words, any equivalent variations and modifications made in accordance with the teachings of the present disclosure are still within the scope of the present disclosure. Those skilled in the art will readily conceive of other embodiments of the present disclosure after considering the disclosure in this specification and practice.

[0098] This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not described in this disclosure. The description and examples are to be considered as illustrative only.

Claims

1. A data quality monitoring method, characterized in that: include: Obtaining a data flow fluctuation curve of a target data table, and building a data quality baseline library based on the data flow fluctuation curve; Extracting abnormal data features based on the data quality baseline library, and generating a feature defect matrix corresponding to the abnormal data features; Screening rule atoms from a preset rule pool according to the characteristic defect matrix, and constructing rule atom links corresponding to the rule atoms; Adapting the rule atomic link to the business scenario to obtain a corresponding quality monitoring rule set; The target data table is scanned based on the quality monitoring rule set to obtain abnormal data scanning results, and the abnormal data scanning results are sent to corresponding associated personnel.

2. The data quality monitoring method according to claim 1, characterized in that: The step of constructing a data quality baseline library according to the data flow fluctuation curve includes: Obtaining upstream and downstream data dependency relationships of the target data table, and generating a data lineage graph corresponding to the upstream and downstream data dependency relationships; Identifying key data nodes based on the data lineage graph and extracting data flow characteristics corresponding to the key data nodes; A node impact factor is calculated according to the data traffic characteristics, and the data traffic fluctuation curve is calibrated using the node impact factor to generate the data quality baseline library.

3. The data quality monitoring method according to claim 2, characterized in that: The step of calibrating the data flow fluctuation curve by using the node impact factor to generate the data quality baseline library includes: Extracting a quality attenuation parameter of the node influence factor; Performing weighted calculation on the data flow fluctuation curve according to the quality attenuation parameter to obtain a weighted data curve; determining a fluctuation range of the weighted data curve, and determining a quality baseline threshold value based on the fluctuation range; Segmenting the weighted data curve using the quality baseline threshold to obtain multiple segmented quality baselines; The segmented quality baselines are integrated and mapped to obtain the data quality baseline library.

4. The data quality monitoring method according to claim 1, characterized in that: The extracting of abnormal data features based on the data quality baseline library and generating a feature defect matrix corresponding to the abnormal data features includes: Obtaining benchmark quality indicators in the data quality baseline library, wherein the benchmark quality indicators include null value rate, duplicate value rate, value domain distribution range and update frequency; Performing a difference analysis between the actual quality index of each field in the target data table and the benchmark quality index; Identify abnormal fields based on the difference analysis results, and extract data features of the abnormal fields as abnormal data features, wherein the abnormal data features include field type, data length, value range, and association relationship; Quantitatively calculating the abnormal data features to obtain a feature scoring matrix; The feature scoring matrix is ​​mapped to the preset quality standard to generate a feature defect matrix.

5. The data quality monitoring method according to claim 1, characterized in that: The step of screening rule atoms from a preset rule pool according to the characteristic defect matrix and constructing rule atom links corresponding to the rule atoms includes: Calculating a rule matching degree according to the defect characteristics in the characteristic defect matrix, wherein the rule matching degree is used to indicate the degree of adaptation between the rule atom and the defect characteristics; Selecting a target rule atom whose rule matching degree exceeds a matching degree threshold from the preset rule pool; Based on the triggering relationship between the target rule atoms, a rule atom dependency graph is constructed; The rule execution order in the rule atom dependency graph is determined, and the rule atoms are connected into rule atom links according to the rule execution order.

6. The data quality monitoring method according to claim 1, characterized in that: Adapting the rule atomic link to the business scenario to obtain a corresponding quality monitoring rule set includes: Obtaining preset business data feature tags, which include data source type, data update cycle, data sensitivity, and data application scenario; Classifying the rule atomic links into scenarios based on the service data feature labels to obtain adaptation rule subsets for multiple types of scenarios; Sorting the rule priorities of each of the adaptation rule subsets, and determining corresponding rule execution strategies according to the rule priorities of each of the adaptation rule subsets; The adaptation rule subsets with the same rule execution strategy are merged to generate a quality monitoring rule set.

7. The data quality monitoring method according to claim 1, characterized in that: The step of scanning the target data table based on the quality monitoring rule set to obtain abnormal data scanning results includes: Constructing a rule scanning task according to the quality monitoring rule set, wherein the rule scanning task includes a scanning period, a scanning range, and a scanning depth; Perform a full data scan on the target data table based on the rule scanning task to obtain an initial scanning result; Performing abnormality grading on the initial scan result to obtain abnormal data corresponding to a plurality of abnormality grades, wherein the abnormality grades include severe abnormality, general abnormality, and slight abnormality; The abnormality proportion and the impact range of the abnormal data corresponding to each abnormality level are determined, and an abnormal data scanning result including the abnormality level, the abnormality proportion and the impact range is generated.

8. A data quality monitoring system, characterized in that: The system comprises: A baseline library determination module is used to obtain a data flow fluctuation curve of a target data table and construct a data quality baseline library based on the data flow fluctuation curve; A link determination module is configured to extract abnormal data features based on the data quality baseline library and generate a feature defect matrix corresponding to the abnormal data features; screen rule atoms from a preset rule pool according to the feature defect matrix, and construct rule atom links corresponding to the rule atoms; A rule set determination module is used to adapt the rule atomic link to the business scenario to obtain a corresponding quality monitoring rule set; The quality monitoring module is used to perform data scanning on the target data table based on the quality monitoring rule set, obtain abnormal data scanning results, and send the abnormal data scanning results to corresponding associated personnel.

9. An electronic device, characterized in that: It includes a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes the data quality monitoring method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores instructions, and when the instructions are executed, the data quality monitoring method according to any one of claims 1 to 7 is executed.

Images