Data quality monitoring method, system, equipment and medium

Abstract

The embodiment of the invention provides a data quality monitoring method, system and device and a medium, and belongs to the technical field of information. The method comprises the following steps: collecting multi-source heterogeneous data; performing multi-dimensional quality index verification on the multi-source heterogeneous data, and performing dynamic threshold adjustment and anomaly prediction on the multi-source heterogeneous data by adopting a pre-trained machine learning model to obtain a multi-dimensional quality index verification result and an anomaly prediction result; and by taking the data object, the time window and the core index as association keys, establishing an association mapping table of a multi-dimensional quality index verification result and an abnormal prediction result, and determining a quality monitoring result of the multi-source heterogeneous data by analyzing the association mapping table. Quality problems can be efficiently found and processed in a multi-source heterogeneous and mass real-time data environment, and the real-time performance, the accuracy and the operation and maintenance efficiency of data governance are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of information technology, and more specifically to a method, system, device, and medium for monitoring data quality. Background Technology

[0002] With the widespread application of cloud computing, the Internet of Things, and artificial intelligence, enterprises and institutions need to continuously collect and process heterogeneous data from multiple sources, including databases, data lakes, message queues, log files, and real-time streams. Throughout the entire lifecycle of data collection, transmission, transformation, and storage, quality issues such as missing data, duplication, format errors, time-series anomalies, and inconsistent correlations often arise due to factors like network fluctuations, interface changes, system failures, or human error. This decline in data quality directly impacts subsequent data analysis, modeling, and business decisions, and may even lead to financial losses or compliance risks.

[0003] Existing data quality management largely relies on manual rule configuration and periodic sampling inspections, and its main shortcomings include: 1. Insufficient real-time performance: Batch offline detection cannot detect anomalies in streaming or incremental data in a timely manner. 2. High maintenance costs: Rules need to be manually written and frequently adjusted, making them difficult to apply in the long term to dynamically changing business environments.

[0004] 3. Low level of intelligence: It lacks the ability to learn from historical quality data and adaptively adjust thresholds, which easily leads to false alarms and missed alarms. Therefore, in the context of big data and data platform construction, enterprises urgently need a technical solution that can automatically discover data quality problems, support dynamic rules and real-time monitoring, in order to reduce operation and maintenance costs and improve data governance. Summary of the Invention

[0005] The purpose of this invention is to provide a data quality monitoring method, system, device, and medium. By combining big data processing, real-time stream computing, and machine learning prediction technologies through automated data profiling, intelligent anomaly detection, and visual monitoring, it achieves continuous quality exploration of multi-source heterogeneous data.

[0006] To achieve the above objectives, embodiments of the present invention provide a data quality monitoring method, comprising: Collect multi-source heterogeneous data; The multi-source heterogeneous data is subjected to multi-dimensional quality index verification and a pre-trained machine learning model is used to dynamically adjust the threshold and predict anomalies in the multi-source heterogeneous data to obtain multi-dimensional quality index verification results and anomaly prediction results. The pre-trained machine learning model includes a time series prediction model for predicting the reasonable prediction range of multi-source heterogeneous data and an anomaly judgment model for judging anomalies based on the reasonable prediction range and real-time data. Using data objects, time windows, and core indicators as association keys, a correlation mapping table is established for the verification results of multi-dimensional quality indicators and the prediction results of anomalies. By analyzing the correlation mapping table, the quality monitoring results of multi-source heterogeneous data are determined.

[0007] Optionally, the multi-source heterogeneous data is subjected to multi-dimensional quality indicator verification, including: Field scanning is performed on multi-source heterogeneous data, and the missing status of key fields in each record is counted; the field integrity score of each record is calculated using the field integrity score calculation formula, and the average field integrity score of all records is counted. If the average field integrity score is less than the first preset standard value, it indicates that the field integrity does not meet the standard. Perform hash calculations on the key fields of each record to generate a hash fingerprint, and perform deduplication on the hash fingerprint. Count the number of deduplicated record fingerprints. Calculate the record duplication rate based on the number of deduplicated record fingerprints and the total number of record fingerprints. If the duplication rate is greater than the second preset standard value, the uniqueness of the record is not up to standard. Load the regular expressions for each field and perform regular expression matching on the target field of each record. Count the number of records with valid format. Calculate the format accuracy based on the number of valid format records and the total number of records. If the format accuracy is less than the third preset standard value, it means that the accuracy does not meet the standard. The range compliance rate and time compliance rate are calculated using the range compliance rate calculation formula and the time compliance rate calculation formula, respectively. If the range compliance rate and time compliance rate are within the preset range, it means that the range is valid. Load the list of relationships, count the number of records with related fields through the JOIN operation, and calculate the relationship existence rate based on the total number of records. If the relationship existence rate is less than the fourth preset standard value, it means that the relationship consistency does not meet the standard. The multi-dimensional quality indicator verification includes at least integrity, uniqueness, accuracy, range verification, and correlation consistency verification.

[0008] The formula for calculating field integrity score is as follows:

[0009] In the formula, m represents the total number of key fields. This represents the business weight of the i-th field. Indicates an indicator function.

[0010] Optionally, the formula for calculating the field repetition rate is as follows:

[0011] In the formula, This indicates the total number of records in the field. This indicates the number of records after deduplication. This indicates the importance coefficient of the field.

[0012] Optionally, the formula for calculating the compliance rate of numeric field ranges is as follows:

[0013] In the formula, Indicates the upper and lower limits of the reasonable range of the field. This represents the actual value of the field, where n represents the total number of records in the field. This indicates the percentage of records that exceeded the warning range. Indicates the warning coefficient; The formula for calculating the compliance rate of time-based fields is as follows:

[0014] In the formula, This indicates the actual value of the time field. Indicates the minimum legal time. This indicates the current time when the verification is performed, and n represents the total number of records in the time field. Indicates an indicator function.

[0015] Optionally, a pre-trained machine learning model can be used to dynamically adjust thresholds and predict anomalies in multi-source heterogeneous data, including: Extracting static and temporal features from multi-source heterogeneous data; The time-series features are input into the time-series prediction model, and the predicted values ​​are output. The architecture of the time-series prediction model consists of an input layer, a hidden layer, an output layer, and a fully connected layer. A new dynamic threshold is calculated based on the predicted value output by the time series prediction model, and the dynamic threshold is adjusted according to the new dynamic threshold. Extract the core indicator values ​​of the real-time window data and compare them with the dynamic threshold. If the core indicator value is less than the lower threshold or greater than the upper threshold, it is marked as a threshold anomaly candidate and enters the second verification. The real-time feature vector is concatenated with the predicted value and the standard deviation of the predicted value output by the time series prediction model to form a fused feature vector; The fused feature vector is input into the anomaly detection model. By randomly dividing the feature space, abnormal samples are isolated, and the degree of anomaly is determined by the length of the isolated path. The architecture of the anomaly detection model is a forest composed of multiple isolated trees, and the height of each tree is set to an upper limit. A voting mechanism is used to determine whether it is an anomaly.

[0016] Optionally, the dynamic threshold can be calculated using the following formula:

[0017] In the formula, Indicates the upper limit threshold. Indicates the lower limit threshold. This represents the predicted value output by the time series forecasting model. The standard deviation of the predicted values ​​is represented by the standard deviation of the predicted values. is the confidence coefficient.

[0018] Secondly, the present invention also provides a data quality monitoring system, comprising: The data access layer is used to collect heterogeneous data from multiple sources. An anomaly detection layer is used to perform multi-dimensional quality indicator verification on the multi-source heterogeneous data and to use a pre-trained machine learning model to dynamically adjust thresholds and predict anomalies in the multi-source heterogeneous data, thereby obtaining multi-dimensional quality indicator verification results and anomaly prediction results. The pre-trained machine learning model includes a time-series prediction model for predicting the reasonable range of multi-source heterogeneous data and an anomaly judgment model for determining anomalies based on the reasonable prediction range and real-time data. A correlation mapping table is established between the multi-dimensional quality indicator verification results and anomaly prediction results, using data objects, time windows, and core indicators as association keys. The quality monitoring results of the multi-source heterogeneous data are determined by analyzing the correlation mapping table. A visualization monitoring platform is used to display the quality monitoring results of multi-source heterogeneous data.

[0019] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the data quality monitoring method described above.

[0020] Fourthly, the present invention also provides a storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the data quality monitoring method described above.

[0021] The above technical solutions enable efficient discovery and handling of quality issues in multi-source heterogeneous, massive real-time data environments, significantly improving the real-time performance, accuracy, and operational efficiency of data governance.

[0022] Other features and advantages of the embodiments of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0023] The accompanying drawings are provided to further illustrate embodiments of the present invention and form part of the specification. They are used together with the following detailed description to explain the embodiments of the present invention, but do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a data quality monitoring method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a data quality monitoring system provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0024] Various embodiments of this disclosure will be described more fully in the following detailed description. This disclosure may have various embodiments, and adjustments and changes may be made therein. However, it should be understood that there is no intention to limit the various embodiments of this disclosure to the specific embodiments disclosed herein, but rather this disclosure should be understood to cover all adjustments, equivalents, and / or alternatives falling within the spirit and scope of the various embodiments of this disclosure.

[0025] In the following, the terms “comprising” or “may include”, which may be used in various embodiments of this disclosure, indicate the presence of the disclosed functions or operations and do not limit the addition of one or more functions or operations. Furthermore, as used in various embodiments of this disclosure, the terms “comprising,” “having,” and their cognates are intended only to indicate a specific feature, number, step, operation, or combination of the foregoing and should not be construed as primarily excluding the presence of one or more other features, numbers, steps, operations, or combinations of the foregoing, or the possibility of adding one or more features, numbers, steps, operations, or combinations of the foregoing.

[0026] In various embodiments of this disclosure, the expression "or" or "at least one of A and / or B" includes any combination or all combinations of the words listed simultaneously. For example, the expression "A or B" or "at least one of A and / or B" may include A, may include B, or may include both A and B.

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] See Figure 1 The diagram shows a flowchart of a data quality monitoring method in a specific embodiment, including the following execution steps: Step 100: Collect multi-source heterogeneous data.

[0029] Specifically, for streaming data: real-time business data is accessed through the Source interface of the stream computing engine, and the data format is unified as JSON / Parquet. The sampling frequency can be set according to business needs and is not limited here. For batch data: historical offline data is accessed through the batch processing interface for model training and threshold initialization.

[0030] In some implementations, the collected multi-source heterogeneous data is cleaned using the ProcessFunction operator of a stream computing engine. First, null values ​​are filtered out. Then, extreme values ​​are removed based on the 3σ principle; if |x-μ|>3σ, the data is marked as noise and removed. Here, μ is the data mean, and σ is the standard deviation. Finally, timestamps are standardized to UTC format, and numerical fields are standardized to the same unit. The preprocessed data is then divided into training and inference streams, with the split achieved through the Side Output mechanism of stream computing.

[0031] Step 101: Perform multi-dimensional quality index verification on the multi-source heterogeneous data and use a pre-trained machine learning model to dynamically adjust the threshold and predict anomalies in the multi-source heterogeneous data to obtain multi-dimensional quality index verification results and anomaly prediction results.

[0032] The pre-trained machine learning model includes a time-series prediction model for predicting the reasonable range of multi-source heterogeneous data and an anomaly detection model for determining anomalies based on the reasonable prediction range and real-time data.

[0033] In one implementation, the training process of the machine learning model is as follows: Offline training: Dataset construction: Historical data was extracted from the batch data and divided into training set, validation set and test set in a ratio of 7:2:1; the training set was labeled with normal (0) / abnormal (1) using unsupervised annotation plus manual correction.

[0034] Training of time series prediction models: Loss function: Root mean square error is used. ,in, For the true value of the i-th historical data, Let be the model prediction value for the i-th sample, and N represent the total number of samples.

[0035] Optimizer: The Adam optimizer is used, with an initial learning rate of 1e. -4 The learning rate is halved when the RMSE of the validation set does not decrease for 5 consecutive epochs.

[0036] Training termination conditions: The validation set RMSE reaches a preset threshold or the training epochs reach a set number, and the optimal model is saved.

[0037] Anomaly detection model training: Input: Real-time data feature vector x of the training set real Predicted value eigenvector y pred σ pred The fusion characteristics after splicing.

[0038] Training parameters: number of trees set to 100, maximum number of samples to 256, maximum number of features D+2; Model evaluation: Calculate the precision, recall, and F1 score of anomaly detection on the test set. The F1 score must be greater than 0.9. Otherwise, re-optimize the features or adjust the model parameters.

[0039] Online incremental training: Triggering condition: Incremental training is triggered when the distribution of real-time streaming data changes significantly; Data update: New data from the past week is extracted from the training stream of the real-time stream and merged with the latest data from the original training set for the past year to form a new training set.

[0040] Incremental training: Time series prediction model: Load the optimal model weights trained offline, continue training using a new training set, and set the training epoch to 10-20 to avoid the model forgetting historical knowledge; Anomaly detection model: Retrain directly on the new training set and replace the old online model; Model validation and deployment: Evaluate the model performance on the new validation set. If the F1-Score is ≥5% higher than the old model, switch to the online model to avoid service interruption.

[0041] In one specific implementation, when performing multi-dimensional quality index verification on the multi-source heterogeneous data in step 101, the following steps can be specifically performed: S1010: Perform field scanning on multi-source heterogeneous data and count the missing status of key fields in each record; calculate the field integrity score of each record using the field integrity score calculation formula, and count the average field integrity score of all records. If the average field integrity score is less than the first preset standard value, it indicates that the field integrity does not meet the standard.

[0042] Specifically, the formula for calculating the field completeness score is as follows:

[0043] In the formula, m represents the total number of key fields. This represents the business weight of the i-th field. Indicates an indicator function.

[0044] S1011: Perform hash calculation on the key fields of each record to generate a hash fingerprint, and perform deduplication on the hash fingerprint. Count the number of deduplicated record fingerprints. Calculate the record duplication rate based on the number of deduplicated record fingerprints and the total number of record fingerprints. If the duplication rate is greater than the second preset standard value, the uniqueness of the record is not up to standard.

[0045] Specifically, the formula for calculating the field repetition rate is as follows:

[0046] In the formula, This indicates the total number of records in the field. This indicates the number of records after deduplication. This indicates the importance coefficient of the field.

[0047] S1012: Load the regular expressions for each field, perform regular expression matching on the target field of each record, count the number of records with valid format, calculate the format accuracy rate based on the number of valid format records and the total number of records, and if the format accuracy rate is less than the third preset standard value, it means that the accuracy does not meet the standard.

[0048] In some implementations, data accuracy verification includes determining the results of format accuracy, semantic consistency score, and deviation rate calculation. The formula for calculating format accuracy is as follows:

[0049] In the formula, This indicates the number of records that match the regular expression. This indicates the total number of records in the field.

[0050] The formula for calculating semantic consistency score is as follows:

[0051] In the formula, n represents the total number of records in the field. This represents a set of fields that conform to business rules. Indicates an indicator function, This represents the semantic influence coefficient.

[0052] The formula for calculating the deviation rate is as follows:

[0053] In the formula, This represents the actual stored value of the derived field. This represents the theoretical value recalculated based on the original fields, where n represents the number of sampled records. Indicates an indicator function.

[0054] S1013: Calculate the range compliance rate and time compliance rate of numeric fields and time fields in multi-source heterogeneous data using the range compliance rate calculation formula and the time compliance rate calculation formula respectively. If the range compliance rate and time compliance rate are within the preset range, it means that the range is valid.

[0055] Specifically, the formula for calculating the compliance rate of numeric field ranges is as follows:

[0056] In the formula, Indicates the upper and lower limits of the reasonable range of the field. This represents the actual value of the field, where n represents the total number of records in the field. This indicates the percentage of records that exceeded the warning range. Indicates the warning coefficient; The formula for calculating the compliance rate of time-based fields is as follows:

[0057] In the formula, This indicates the actual value of the time field. Indicates the minimum legal time. This indicates the current time when the verification is performed, and n represents the total number of records in the time field. Indicates an indicator function.

[0058] S1014: Load the list of related relationships, count the number of records with related fields through the JOIN operation, and calculate the relationship existence rate based on the total number of records. If the relationship existence rate is less than the fourth preset standard value, it means that the relationship consistency does not meet the standard.

[0059] In some implementations, the association logic consistency score is calculated according to the following formula:

[0060] In the formula, k represents the number of associated attributes. This represents the weight of the i-th associated attribute. This represents the inconsistency rate of the i-th associated attribute.

[0061] The multi-dimensional quality indicator verification includes at least integrity, uniqueness, accuracy, range verification, and correlation consistency verification.

[0062] In some implementations, the overall quality score is calculated using the following formula:

[0063] In the formula, ω1, ω2, ω3, ω4, and ω5 represent the weights of field completeness, field duplication rate, semantic consistency score, numeric field range compliance rate, and relational logic consistency score, respectively. FCSavg Indicates the average score for field completeness, FDR avg This indicates the average score of the field repetition rate, SCS. avg Meaning of semantic consistency average score, RCR avg This represents the average score of compliance rate for numeric fields, ALCS. avg This represents the average score for consistency of the associated logic.

[0064] In one specific implementation, when performing dynamic threshold adjustment and anomaly prediction on multi-source heterogeneous data using a pre-trained machine learning model in step 101, the following steps can be performed: S1: Extract static and temporal features from multi-source heterogeneous data.

[0065] Specifically, for static features, device ID, interface type, user level, and region code are extracted from business metadata and converted into numerical features through embedding. For time-series features, a sliding window based on stream computing is used for extraction. The window size and sliding step are set according to the business cycle, such as a 20-minute window and a 5-minute step. Specific features include: statistical features: mean, variance, maximum, minimum, and median of the data within the window; trend features: first-order difference, second-order difference, and growth rate of the data within the window; and periodic features: the periodic components of the data are extracted through Fourier transform and used as input for periodic features.

[0066] S2: Input the time series features into the time series prediction model and output the predicted value. The architecture of the time series prediction model consists of an input layer, a hidden layer, an output layer, and a fully connected layer.

[0067] Specifically, the input layer consists of a time-series feature vector in the format [batch_size, time_steps, D]; the hidden layer consists of two LSTM units, with the first layer having an output dimension of 128 and the second layer having an output dimension of 64, using tanh as the activation function and Dropout to prevent overfitting; and the output layer consists of a fully connected layer with an output dimension of 1, representing the predicted normal data value for the next window, and also outputting the standard deviation of the predicted value for subsequent dynamic threshold calculation.

[0068] S3: Calculate a new dynamic threshold based on the predicted value output by the time series prediction model, and adjust the dynamic threshold according to the new dynamic threshold.

[0069] Specifically, the dynamic threshold is calculated using the following formula:

[0070] In the formula, Indicates the upper limit threshold. Indicates the lower limit threshold. This represents the predicted value output by the time series forecasting model. The standard deviation of the predicted values ​​is represented by the standard deviation of the predicted values. is the confidence coefficient.

[0071] In some implementations, real-time threshold updates can be achieved by triggering real-time inference of the time-series prediction model after each sliding window calculation completed by the streaming engine, outputting the normal data values ​​for the next window and the standard deviation of the predicted values; based on the new predicted normal data values ​​for the next window and the standard deviation of the predicted values, TH is recalculated. up and TH low Overrides the old threshold to achieve dynamic window-level adjustment; Threshold constraint: if the calculated TH low If <0, then force TH low = 0, to avoid logical errors.

[0072] S4: Extract the core indicator values ​​of the real-time window data and compare them with the dynamic threshold. If the core indicator value is less than the lower threshold or greater than the upper threshold, it is marked as a threshold anomaly candidate and enters the second verification.

[0073] It should be understood that the lower and upper thresholds can be set according to the specific application scenario, and are not restricted here.

[0074] S5: The real-time feature vector is concatenated with the predicted value and the standard deviation of the predicted value output by the time series prediction model to form a fused feature vector.

[0075] S6: Input the fused feature vector into the anomaly detection model, isolate the abnormal samples by randomly dividing the feature space, and judge the degree of anomaly by the length of the isolation path. The architecture of the anomaly detection model is a forest composed of multiple isolation trees, and the height of each tree is set with an upper limit. The voting mechanism determines whether it is abnormal.

[0076] For example, the real-time feature vector X real The predicted value y output by the time series prediction model pred and the standard deviation of the predicted value σ pred The features are concatenated to form a fused feature vector X. fuse = [X real , y pred , σ pred ]; X fuse Input the anomaly detection model and output the anomaly score and anomaly label. Anomaly score: the ratio of the average isolation path length of the sample to the average path length of the forest. The anomaly score ∈ [0,1]. The closer the score is to 1, the higher the probability of anomaly. Anomaly label: the anomaly detection model outputs 1 (abnormal) or 0 (normal) according to the preset threshold.

[0077] Step 102: Using data objects, time windows, and core indicators as association keys, establish an association mapping table for multi-dimensional quality indicator verification results and anomaly prediction results, and determine the quality monitoring results of multi-source heterogeneous data by analyzing the association mapping table.

[0078] For example, based on the association key, execute the query SELECT * FROM quality_result q JOIN anomaly_result a ON q.data_object=a.data_object AND q.time_window=a.time_window to filter out two types of results for the same data object and within the same time period; use association rules to statistically analyze the support and confidence of the quality problem → anomaly type in historical data, determine the strength of the association, and write the strength of the relationship into the knowledge base as root cause rules.

[0079] For example, if the support (frequency of missing payment time and abnormal payment amount / total number of records) = 0.75, the confidence (probability of missing payment time → abnormal payment amount) = 0.92, and the support ≥ 0.6 and the confidence ≥ 0.8, then it is determined to be a strong association.

[0080] In some implementations, detection results, anomaly patterns, and handling strategies are stored in a knowledge base to achieve experience accumulation and rule reuse. A visual monitoring platform displays data quality scores, trend analysis, anomaly details, and alarm information, supporting user operations and further rule optimization. The knowledge base includes: a rule metadata database, a historical anomaly database, a handling strategy database, and a data quality indicator database, used to store rule execution history, anomaly logs, alarm handling records, and quality scoring indicators, enabling knowledge reuse and closed-loop governance.

[0081] In some implementations, a multi-dimensional dashboard is built on the monitoring platform, displaying two types of results and their relationships in modules. The core pages include: a comprehensive overview page with core indicator cards displaying the average quality score (OQS), total number of abnormal events, and number of P0 / P1 / P2 alarms for all data objects, updated in real time; a dimensional trend chart displaying the score trends of dimensions such as completeness (FCS), uniqueness (FDR), and accuracy (SCS) for a data object by time window, allowing comparison with historical data from the same period; a problem details table listing quality issues for the data object, including the problem field, the number of affected records, and suggested remediation solutions; an anomaly type distribution pie chart showing the percentage of anomaly types within a certain time period; a list of associated quality issues displaying associated quality issues for a specific anomaly event; and a root cause diagram using a flowchart. The document demonstrates the chain from quality issues to intermediate impacts to abnormal results; it also compares historical root causes, querying historical root causes of the same anomaly type to aid in rapid identification. Root cause identification and closed-loop repair are implemented: after an alarm is triggered, the root cause must be quickly identified based on correlation analysis results and knowledge base experience, forming a closed loop of repair, verification, and consolidation to prevent recurrence. The root cause identification process involves: retrieving correlation results, obtaining quality issues associated with the abnormal event from the monitoring platform, and matching knowledge base rules. It queries the knowledge base for the root cause corresponding to missing payment time. If a synchronization delay in source system A is suspected, the synchronization logs of source system A are checked, and the data volume of the payment time field in the source and target systems is compared. If the verification is successful, the root cause is determined to be a synchronization timeout in source system A; otherwise, other knowledge base rules are verified sequentially until the root cause is identified.

[0082] In some implementations, the closed-loop repair process includes: performing repair: executing repair operations based on the root cause. After the repair, a quality check and anomaly detection are triggered to re-detect the data object. If the quality score recovers and the anomaly disappears, the repair is considered successful. If the standard is not met, the root cause is re-identified, and the root cause → repair solution → verification result is written into the knowledge base, and the quality problem-root cause-repair solution mapping table is updated.

[0083] In this embodiment, a closed-loop management system can be implemented, which includes automated data profiling, intelligent threshold calculation, anomaly alarms, and self-healing. This significantly reduces data governance and maintenance costs, improves the real-time performance and accuracy of data quality monitoring, and meets the high-reliability data governance needs of data platforms in finance, healthcare, government, and large enterprises.

[0084] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0085] Compared with existing methods that rely on manual sampling or single-item testing, the automated data quality inspection method of this application has significant advantages and technical effects in the following aspects: 1. Real-time and comprehensive: By combining streaming computing with batch processing architecture, continuous probing and real-time alerts can be provided for multi-source heterogeneous structured, semi-structured and unstructured data, ensuring that quality anomalies in each stage of data acquisition, transmission, transformation and storage can be quickly detected and handled. 2. Intelligence and Adaptability: By employing machine learning and time series prediction algorithms, historical quality indicators are modeled to achieve dynamic threshold adjustment and abnormal pattern recognition, reducing false alarms and false negatives caused by the rigidity of traditional rules and significantly improving detection accuracy. 3. Low maintenance and high scalability: Provides graphical rule configuration and domain-specific language, and can automatically recommend detection strategies based on metadata, greatly reducing manual writing and later operation and maintenance costs; modular design facilitates rapid deployment and expansion in cloud platforms, data middleware and various enterprise-level systems. 4. Closed-loop governance capability: By storing rules, detection results, and processing strategies in a knowledge base, it supports automated anomaly handling and experience accumulation, forming a closed-loop data governance system of investigation, alerting, repair, and self-healing, and continuously improving the reliability and value of data assets. In summary, this application can significantly improve the efficiency and stability of quality monitoring in large-scale data processing environments, ensure the data foundation for critical business decisions, and has broad industry application prospects and economic value.

[0086] like Figure 2 As shown, the following are embodiments of the data quality monitoring system provided in this disclosure. The data quality monitoring methods described above belong to the same inventive concept. For details not described in detail in the embodiments of the data quality monitoring system, please refer to the embodiments of the data quality monitoring methods described above.

[0087] Data quality monitoring systems include: The data access layer is used to collect heterogeneous data from multiple sources. An anomaly detection layer is used to perform multi-dimensional quality indicator verification on the multi-source heterogeneous data and to use a pre-trained machine learning model to dynamically adjust thresholds and predict anomalies in the multi-source heterogeneous data, thereby obtaining multi-dimensional quality indicator verification results and anomaly prediction results. The pre-trained machine learning model includes a time-series prediction model for predicting the reasonable range of multi-source heterogeneous data and an anomaly judgment model for determining anomalies based on the reasonable prediction range and real-time data. A correlation mapping table is established between the multi-dimensional quality indicator verification results and anomaly prediction results, using data objects, time windows, and core indicators as association keys. The quality monitoring results of the multi-source heterogeneous data are determined by analyzing the correlation mapping table. A visualization monitoring platform is used to display the quality monitoring results of multi-source heterogeneous data.

[0088] Figure 3 This is a schematic diagram of the hardware structure of an electronic device that implements various embodiments of the present invention.

[0089] The data quality monitoring method provided in this application can be applied to electronic devices. Those skilled in the art will understand that the electronic device structure involved in the embodiments of this invention does not constitute a limitation on the electronic device. An electronic device may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. In the embodiments of this invention, electronic devices include, but are not limited to, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of this application described and / or claimed herein.

[0090] Electronic devices may include processors, external memory interfaces, internal memory, universal serial bus (USB) interfaces, charging management modules, power management modules, batteries, wireless communication modules, audio modules, speakers, microphones, sensor modules, buttons, cameras, displays, and SIM card interfaces, etc.

[0091] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0092] A processor may include one or more processing units, such as: a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0093] The processor can serve as the nerve center and command center of an electronic device. The controller can generate operation control signals based on the instruction opcode and timing signals to control the fetching and execution of instructions.

[0094] The processor may also include memory for storing instructions and data. In some embodiments, the memory in the processor is a cache memory. This memory can store instructions or data that the processor has just used or that are used repeatedly. If the processor needs to use the instruction or data again, it can retrieve it directly from this memory. This avoids repeated accesses, reduces processor latency, and thus improves system efficiency.

[0095] An external storage interface (ESI) can be used to connect external memory cards, such as microSD cards, to expand the storage capacity of electronic devices. The external memory card communicates with the processor through the ESI to perform data storage functions, such as saving music and video files on the external memory card.

[0096] Internal memory can be used to store computer executable program code, which includes instructions. The processor executes various functional applications and data processing of electronic devices by running the instructions stored in internal memory. Internal memory can include a program storage area and a data storage area. Internal memory can include high-speed random access memory, and can also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0097] Wireless communication functionality in electronic devices can be achieved through antennas, wireless communication modules, modem processors, and baseband processors.

[0098] Wireless communication modules can provide solutions for wireless communication applications in electronic devices, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies.

[0099] Electronic devices can implement audio functions through audio modules, speakers, receivers, microphones, headphone jacks, and application processors.

[0100] Electronic devices can achieve shooting functions through ISPs, cameras, video codecs, GPUs, displays, and application processors.

[0101] Electronic devices can achieve display functions through GPUs, displays, and application processors.

[0102] A GPU is a microprocessor for image processing, connected to the display screen and application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering. A processor may include one or more GPUs, which execute program instructions to generate or modify display information.

[0103] A display screen is used to display images, videos, etc. A display screen includes a display panel.

[0104] The storage medium provided in this application stores a program product capable of implementing a data quality monitoring method.

[0105] The data quality monitoring method includes: collecting multi-source heterogeneous data; performing multi-dimensional quality indicator verification on the multi-source heterogeneous data and using a pre-trained machine learning model to dynamically adjust thresholds and predict anomalies in the multi-source heterogeneous data, thereby obtaining multi-dimensional quality indicator verification results and anomaly prediction results. The pre-trained machine learning model includes a time-series prediction model for predicting a reasonable prediction range for the multi-source heterogeneous data and an anomaly judgment model for determining anomalies based on the reasonable prediction range and real-time data. A correlation mapping table is established between the multi-dimensional quality indicator verification results and the anomaly prediction results, using data objects, time windows, and core indicators as association keys. The quality monitoring results of the multi-source heterogeneous data are determined by analyzing the correlation mapping table.

[0106] In some possible implementations, the subject matter of this disclosure, the method and system for monitoring data quality, can be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0107] The storage medium disclosed herein may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0108] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method of monitoring data quality, characterized by, The method comprises the following steps: Collecting multi-source heterogeneous data; Performing multi-dimensional quality index verification on the multi-source heterogeneous data and performing dynamic threshold adjustment and anomaly prediction on the multi-source heterogeneous data by using a pre-trained machine learning model to obtain multi-dimensional quality index verification results and anomaly prediction results, wherein the pre-trained machine learning model comprises a time series prediction model for predicting a reasonable prediction range of the multi-source heterogeneous data and an anomaly judgment model for judging anomalies based on the reasonable prediction range and real-time data; Establishing a correlation mapping table of the multi-dimensional quality index verification results and the anomaly prediction results by taking data objects, time windows and core indicators as correlation keys, and determining a quality monitoring result of the multi-source heterogeneous data by analyzing the correlation mapping table.

2. The method of claim 1, wherein, The multi-dimensional quality index verification on the multi-source heterogeneous data comprises the following steps: Performing field scanning on the multi-source heterogeneous data and counting the missing state of a key field in each record; calculating the field integrity score of each record by using a field integrity score calculation formula, and counting the average field integrity score of all records, if the average field integrity score is less than a first preset standard value, the field integrity is not up to standard; Performing hash calculation on the key field of each record to generate a hash fingerprint, and performing deduplication processing on the hash fingerprint to count the number of deduplicated record fingerprints, calculating the record duplication rate based on the number of deduplicated record fingerprints and the total number of record fingerprints, if the duplication rate is greater than a second preset standard value, the uniqueness is not up to standard; Loading regular expressions of each field, and performing regular matching on the target field of each record to count the number of format qualified records, calculating the format accuracy rate based on the number of format qualified records and the total number of records, if the format accuracy rate is less than a third preset standard value, the accuracy is not up to standard; Respectively using a range compliance rate calculation formula and a time compliance rate calculation formula to calculate the range compliance rate and the time compliance rate of the numerical fields and the time fields in the multi-source heterogeneous data, if the range compliance rate and the time compliance rate are within a preset interval, the range is valid; Loading the correlation relationship list, counting the number of associated field records by JOIN operation, and calculating the association existence rate based on the total number of records, if the association existence rate is less than a fourth preset standard value, the association consistency is not up to standard; The multi-dimensional quality index verification at least comprises integrity, uniqueness, accuracy, range verification and association consistency verification.

3. The method of monitoring data quality according to claim 2, wherein, The field integrity score calculation formula is as follows: In the formula, m represents the total number of key fields, represents the service weight of the i-th field, represents an indication function.

4. The method of claim 2, wherein, The field duplication rate calculation formula is as follows: In the formula, represents the total number of records of the field, represents the number of records after deduplication, represents the field importance coefficient.

5. The method of claim 2, wherein, The numerical field range compliance rate calculation formula is as follows: In the formula, represents the upper and lower limits of the reasonable range of the field, represents the actual value of the field, and n represents the total number of records of the field, represents the proportion of records exceeding the warning interval, represents the early warning coefficient; The time field compliance rate calculation formula is as follows: In the formula, represents the actual value of the time field, represents the minimum legal time, represents the current time when the check is performed, and n represents the total number of time field records, represents an indication function.

6. The method of claim 1, wherein, The dynamic threshold adjustment and anomaly prediction on the multi-source heterogeneous data by using the pre-trained machine learning model comprises the following steps: Extracting static features and time series features from the multi-source heterogeneous data; Inputting the time series features into the time series prediction model to output a predicted value, wherein the architecture of the time series prediction model comprises an input layer, a hidden layer, an output layer and a full connection layer; Calculating a new dynamic threshold value based on the predicted value output by the time series prediction model, and adjusting the dynamic threshold value according to the new dynamic threshold value; The core index value of the real-time window data is extracted and compared with a dynamic threshold value. If the core index value is less than the lower threshold value or the core index value is greater than the upper threshold value, the core index value is marked as a threshold abnormality candidate and enters a second re-verification; The real-time feature vector is spliced with the predicted value and the standard deviation of the predicted value output by the time series prediction model to form a fusion feature vector; The fusion feature vector is input into an anomaly judgment model. The anomaly judgment model is a forest composed of multiple isolation trees. The height of each tree is set with an upper limit. The voting mechanism is used to determine whether it is abnormal.

7. The method of monitoring data quality according to claim 6, wherein, The dynamic threshold value is calculated according to the following formula: wherein represents an upper threshold value, represents a lower threshold value, represents a prediction value output by the temporal prediction model, represents a standard deviation of the prediction values, is a confidence coefficient.

8. A data quality monitoring system, characterized by It comprises: a data access layer for collecting multi-source heterogeneous data; an anomaly detection layer for performing multi-dimensional quality index verification on the multi-source heterogeneous data and adjusting and predicting the anomaly of the multi-source heterogeneous data by using a pre-trained machine learning model to obtain a multi-dimensional quality index verification result and an anomaly prediction result, wherein the pre-trained machine learning model comprises a time series prediction model for predicting a reasonable prediction range of the multi-source heterogeneous data and an anomaly judgment model for judging the anomaly based on the reasonable prediction range and real-time data; a correlation mapping table of the multi-dimensional quality index verification result and the anomaly prediction result is established by taking data objects, time windows and core indexes as correlation keys, and the quality monitoring result of the multi-source heterogeneous data is determined by analyzing the correlation mapping table; a visual monitoring platform for displaying the quality monitoring result of the multi-source heterogeneous data.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to realize the steps of the data quality monitoring method according to any one of claims 1 to 7.

10. A storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to realize the steps of the data quality monitoring method according to any one of claims 1 to 7.

Images