A Data Governance Risk Early Warning Method and System Based on Big Data Mining
By using big data mining technology to collect and analyze data governance operation data, identify abnormal propagation paths, and perform local recalculation verification, the problem of low efficiency in judging and verifying abnormal propagation in data governance risk warning is solved, and more accurate risk warning and efficient data governance are achieved.
Patent Information
- Application Number
- CN202611347325.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-02
- Publication Date
- 2026-11-17
AI Technical Summary
Existing data governance risk warning technologies are insufficient in combining the judgment of abnormal propagation with the actual impact range of current data, resulting in low verification efficiency and reliance on large-scale data recalculation, which affects the accuracy of warnings.
By using big data mining technology, we collect data on data governance operations, establish data relationships, uncover abnormal propagation relationships, identify the current risk propagation status, assess data governance risks, determine risk objects to be verified, perform partial or complete recalculation verification, and generate the final risk warning result.
It improves the accuracy of identifying abnormal propagation paths, reduces the duplication of irrelevant data processing, and enhances the efficiency of risk verification and the reliability of early warning.
Smart Images

Figure CN123115116A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data governance technology, and in particular to a data governance risk early warning method and system based on big data mining. Background Technology
[0002] With the development of big data, cloud computing, and data platform technologies, enterprise data scales are constantly expanding, and the processing stages such as data collection, cleaning, transformation, aggregation, storage, exchange, and publication are becoming increasingly complex. Data governance risk warning has evolved from early data quality checks and manual maintenance to continuous monitoring and proactive early warning across the entire data processing process. Existing technologies typically monitor the data processing process by collecting data on the execution status of data processing tasks, processing time, input and output data volume, number of abnormal records, field integrity, data structure changes, and data distribution. This monitoring is combined with fixed thresholds, historical operational data, statistical analysis, or anomaly detection models to identify task execution anomalies and data quality anomalies. As the correlation between different data processing tasks increases, some data governance platforms also analyze the potential impact of abnormal tasks by considering task dependencies, data lineage, and upstream and downstream data flow relationships, thereby helping to identify risk targets and the scope of anomaly impact.
[0003] While current data governance risk warning technologies can identify situations such as task failures, abnormal data volumes, missing fields, structural changes, and data distribution shifts, issues still exist that affect the accuracy of warnings and the efficiency of verification. First, existing technologies typically rely on pre-established data lineages, task dependencies, or historical anomaly associations to determine anomaly propagation. They rarely consider the current deviation from the actual scope and quantity of data read by downstream tasks to determine if an anomaly has truly propagated. This can easily include governance objects with only dependencies but not currently affected by abnormal data within the risk scope, and it's difficult to accurately determine the source of the current anomaly. Second, after identifying a risky object, current technologies typically verify it by re-executing the entire task, rereading all input data, or manually checking logs. When the data processing scale is large and the task chain is long, it's necessary to repeatedly process a large amount of data unrelated to the current anomaly, resulting in a large amount of recalculated data and low risk verification efficiency. Therefore, current data governance risk warnings still suffer from insufficient integration of anomaly propagation judgments with the actual scope of current data impact, and risk verification relies on large-scale data recalculation, leading to low verification efficiency. Summary of the Invention
[0004] In view of the aforementioned existing problems, the present invention is proposed.
[0005] Therefore, this invention provides a data governance risk early warning method and system based on big data mining to solve the problems of insufficient combination between the judgment of abnormal propagation and the current actual data impact range, as well as the problems of risk verification relying on large-scale data recalculation and low verification efficiency.
[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:
[0007] In a first aspect, the present invention provides a data governance risk early warning method based on big data mining, which includes: collecting data governance operation data and establishing data correlation relationships, obtaining a basic dataset for data governance monitoring, establishing a data governance analysis benchmark based on the basic dataset for data governance monitoring, and obtaining data governance analysis benchmark data.
[0008] Based on the benchmark data of data governance analysis, anomaly propagation relationships are mined to obtain a risk propagation correlation graph. The current data processing anomalies are identified through the risk propagation correlation graph to obtain the current risk propagation status data.
[0009] Based on the current risk propagation status data, assess data governance risks, obtain data governance risk assessment results, determine risk objects to be verified based on data governance risk assessment results, and obtain a set of risk objects to be verified.
[0010] Anomaly recalculation verification is performed based on the set of risk objects to be verified, and the risk verification results are obtained. Data governance risk warnings are generated based on the risk verification results, and the final data governance risk warning results are obtained.
[0011] As a preferred embodiment of the data governance risk early warning method based on big data mining described in this invention, the step of collecting data governance operation data and establishing data association relationships to obtain the basic dataset for data governance monitoring refers to collecting data processing operation data and data status data containing input data version identifiers or input data snapshot identifiers, task versions and task execution parameters, establishing directed association relationships, data processing dependencies and input source records between governance objects, associating them according to data processing instances, and obtaining the basic dataset for data governance monitoring.
[0012] As a preferred embodiment of the data governance risk early warning method based on big data mining described in this invention, the step of establishing a data governance analysis benchmark based on the data governance monitoring basic dataset and obtaining the data governance analysis benchmark data refers to reading the data governance monitoring basic dataset, filtering historical normal processing instances, historical abnormal processing instances and current processing instances, establishing a normal monitoring benchmark based on the historical normal processing instances, and associating input source records, data processing dependencies and directed associations to obtain the data governance analysis basic data.
[0013] As a preferred embodiment of the data governance risk early warning method based on big data mining described in this invention, the step of obtaining a risk propagation correlation graph by mining abnormal propagation relationships based on data governance analysis benchmark data refers to reading basic data of data governance analysis, calculating the abnormal propagation correlation degree according to the actual data transmission relationship of historical abnormal handling instances, configuring the abnormal propagation correlation degree with governance objects as nodes and directed correlation relationships as edges, and associating normal monitoring benchmarks and current handling instances to obtain a risk propagation correlation graph.
[0014] As a preferred embodiment of the data governance risk early warning method based on big data mining described in this invention, the step of identifying current data processing anomalies through a risk propagation correlation graph and obtaining current risk propagation status data refers to reading the risk propagation correlation graph, comparing the current processing instance with the normal monitoring benchmark, obtaining the deviation of numerical features, structural deviation, field distribution deviation, and task status, calculating the current local anomaly degree, determining the current deviation data range, and obtaining the current risk propagation status data.
[0015] As a preferred embodiment of the data governance risk early warning method based on big data mining described in this invention, the step of assessing data governance risk based on the current risk propagation status data and obtaining the data governance risk assessment result refers to reading the current risk propagation status data, calculating the current deviation data impact coverage, the current abnormal propagation intensity, and the current propagation evidence, obtaining the data governance risk value and the abnormal source location value, determining candidate abnormal source data processing instances, determining the upper boundary of normal risk based on historical normal processing instances, and obtaining the data governance risk assessment result.
[0016] As a preferred embodiment of the data governance risk early warning method based on big data mining described in this invention, the step of determining the risk objects to be verified through the data governance risk assessment results and obtaining the set of risk objects to be verified refers to reading the data governance risk assessment results, comparing the current data governance risk value with the upper boundary of normal risk, determining the normal monitoring objects and the risk objects to be verified, and recording the candidate abnormal source data processing instances, the current deviation data range and the current propagation path to obtain the set of risk objects to be verified.
[0017] As a preferred embodiment of the data governance risk early warning method based on big data mining described in this invention, the step of performing anomaly recalculation verification based on the set of risk objects to be verified and obtaining the risk verification result refers to reading the set of risk objects to be verified and the basic dataset of data governance monitoring, constructing anomaly dependent data slices for the currently locatable deviation data range and recalculating them, and when partial recalculation is not possible, performing a complete recalculation or status verification, determining the verification status based on the recalculated data governance risk value and the upper boundary of normal risk, and obtaining the risk verification result.
[0018] As a preferred embodiment of the data governance risk early warning method based on big data mining described in this invention, the step of generating a data governance risk early warning through risk verification results and obtaining the final data governance risk early warning result refers to reading the risk verification results, not issuing warnings for monitoring error objects, generating recovery prompts for recoverable abnormal objects, determining the data governance risk level for confirmed risk objects based on the actual downstream governance object's operating status, updating the normal monitoring benchmark and abnormal propagation correlation, and obtaining the final data governance risk early warning result.
[0019] Secondly, the present invention provides a data governance risk early warning system based on big data mining, comprising:
[0020] The monitoring and acquisition module is used to collect data processing operation data and data status data, establish data processing dependencies, and form a basic dataset for data governance monitoring.
[0021] The benchmark analysis module is used to filter historical normal and abnormal handling instances, establish normal monitoring benchmarks, and form basic data for data governance analysis.
[0022] The propagation identification module is used to analyze the data transmission relationships of historical anomaly instances, calculate the anomaly propagation correlation degree, and construct a risk propagation correlation diagram.
[0023] The risk assessment module is used to identify the current abnormal state, calculate the intensity of abnormal propagation, evidence of propagation and data governance risk value, and determine the risk objects to be verified.
[0024] The verification and early warning module is used to perform partial or complete recalculation of the risk objects to be verified, determine the verification status, and generate the final data governance risk early warning result.
[0025] The beneficial effects of this invention are as follows: by combining the correlation of historical anomalies with the coverage of current deviation data, the current anomaly propagation intensity and propagation evidence are calculated based on the actual data transmission relationship, reducing misjudgments caused by relying solely on static lineage or historical correlation, improving the accuracy of anomaly propagation path and anomaly source identification, and by tracing data processing dependencies for the current deviation data range, forming anomaly dependent data slices and performing local recalculation, reducing the repetitive processing of irrelevant data, reducing the amount of recalculated data while ensuring the reliability of risk verification, and improving the efficiency of data governance risk verification and early warning. Attached Figure Description
[0026] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0027] Figure 1 This is a flowchart of a data governance risk early warning method based on big data mining.
[0028] Figure 2 This is a schematic diagram of a data governance risk early warning system based on big data mining. Detailed Implementation
[0029] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
[0030] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.
[0031] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.
[0032] Reference Figure 1 This is one embodiment of the present invention, which provides a data governance risk early warning method based on big data mining, including the following steps:
[0033] S1. Collect data on data governance operation and establish data correlation to obtain a basic dataset for data governance monitoring. Based on the basic dataset for data governance monitoring, establish a data governance analysis benchmark and obtain data governance analysis benchmark data.
[0034] S1.1: Collect data processing operation data and data status data containing input data version identifier or input data snapshot identifier, task version and task execution parameters, establish directed relationships between governance objects, data processing dependencies and input source records, associate by data processing instance, and obtain the basic dataset for data governance monitoring.
[0035] Specifically, data processing tasks included in data governance risk monitoring are identified as governance objects. Each governance object corresponds to a task identifier and task version. Each execution of a data processing task by a governance object forms a data processing instance and generates a corresponding output data batch. A single data processing instance serves as the basic unit for subsequent anomaly detection, risk assessment, and risk verification. When an output data batch generated by one governance object is read as an input data batch by another governance object, the former governance object is identified as the upstream governance object of the latter, and a directed association relationship between governance objects is established based on the actual reading relationship between input and output data batches. The data processing configuration of each governance object is read from the data governance configuration data, and a data processing dependency relationship is formed according to the execution order of each data processing operation. Data processing operation data and data status data of each data processing instance are collected according to the monitoring cycle. When a data processing instance reads one or more upstream data batches, the actual data partition or recording range read by each upstream data batch and its corresponding data are recorded. The system should read the number of records and associate them with the upstream governance object identifier, the upstream output data batch identifier, and the current input data batch identifier to form an input source record. For each data processing instance, the system should obtain the task processing duration based on the task start and end times, the abnormal record rate based on the number of abnormal records and the number of input records, and the key field null value rate based on the number of null values in key fields and the number of records involving the corresponding key field in the processing. The system should then concatenate the field number, field name, and field type in order to form a structure description string, and perform the same hash operation on the structure description string to obtain a structure fingerprint. For key fields requiring distributed monitoring, the system should count the number of valid records in each bin according to fixed bin boundaries, and divide the number of valid records in each bin by the total number of valid records for that key field to obtain the field binning probability. Finally, the system should associate the data processing runtime data, data status data, input source records, data processing dependencies, and directed relationships between governance objects according to the data processing instance to obtain the basic dataset for data governance monitoring.
[0036] It should be noted that the data processing configuration includes at least the associated fields, grouping fields, deduplication fields, window ranges, and input / output field mapping relationships;
[0037] The data processing operation data includes at least the governance object identifier, data processing instance identifier, task identifier, task version, task start time, task end time, task execution status, input data batch identifier, output data batch identifier, input data partition identifier, output data partition identifier, input data version identifier or input data snapshot identifier, task execution parameter identifier, number of input records, number of output records, and number of abnormal records;
[0038] Data status data should include at least field names, field types, field order, number of null values in key fields, and statistical results of data intervals for key fields that need to be monitored for distribution. Key fields and their corresponding fixed bin boundaries should be read from the data governance configuration data and kept consistent during the establishment of historical benchmarks and subsequent monitoring.
[0039] S1.2: Read the basic dataset for data governance monitoring, filter historical normal processing instances, historical abnormal processing instances, and current processing instances, establish a normal monitoring benchmark based on historical normal processing instances, and associate input source records, data processing dependencies, and directed relationships to obtain basic data for data governance analysis.
[0040] Specifically, the data governance monitoring basic dataset is read. Based on the runtime and verification status of each data processing instance, historical normal processing instances and historical abnormal processing instances are filtered from the completed historical processing data, and the current processing instance corresponding to the current monitoring period is retained. For data processing instances that have not yet completed status verification or whose normal or abnormal status cannot be determined, they are not included in the establishment of normal monitoring benchmarks and the statistics of abnormal propagation correlation, and the current processing instance corresponding to the current monitoring period is retained. For the same governance object, task processing time, number of input records, number of output records, abnormal record rate, and null value rate of key fields are extracted from historical normal processing instances as numerical monitoring features. For the numerical monitoring features of the governance object, the historical midpoint of the feature values corresponding to historical normal processing instances is calculated. The median and median absolute deviation are used to statistically analyze the structural fingerprints of the same governance object in historical normal processing instances, forming a normal structural fingerprint set. For key fields requiring distribution monitoring, the average binning probability of each binning field in historical normal processing instances is calculated to obtain the normal field binning probability and form a normal field distribution. The historical median, median absolute deviation, normal structural fingerprint set, and normal field distribution are jointly determined as the normal monitoring benchmark for the corresponding governance object. Historical normal processing instances, historical abnormal processing instances, historical median, median absolute deviation, normal structural fingerprint set, normal field distribution, current processing instance, input source records, data processing dependencies, and directed associations between governance objects are correlated to obtain basic data for data governance analysis.
[0041] The formula for calculating the median absolute deviation is:
[0042]
[0043] In the formula, It is the object of governance The The median absolute deviation of the numerical monitoring features. It is the object of governance The The historical median of the numerical monitoring features, It is the object of governance The The first historical normal processing instance corresponds to the first Numerical monitoring characteristic values.
[0044] It should be noted that data processing instances where the task is completed normally, the output data can be read normally, and the data is not identified as abnormal by the fault handling record or data integrity verification record are defined as historical normal processing instances; data processing instances that have been identified as abnormal by the task failure record, data integrity verification record, or fault handling record are defined as historical abnormal processing instances.
[0045] S2. Based on the data governance analysis benchmark data, mine the abnormal propagation relationship to obtain the risk propagation correlation diagram. Identify the current data processing anomalies through the risk propagation correlation diagram to obtain the current risk propagation status data.
[0046] S2.1: Read the basic data for data governance analysis, calculate the anomaly propagation correlation degree based on the actual data transmission relationship of historical anomaly handling instances, configure the anomaly propagation correlation degree with the governance object as the node and the directed relationship as the edge, and associate the normal monitoring benchmark and the current handling instance to obtain the risk propagation correlation graph.
[0047] Specifically, the process involves reading historical anomaly handling instances and corresponding input source records from the basic data for data governance analysis. Based on the actual reading relationship between input data batches and output data batches, the data transmission relationship between historical anomaly handling instances is determined. For upstream and downstream governance objects with direct directed relationships, the number of historical instances where an anomaly occurred in an upstream governance object and the corresponding output data was actually read by the downstream governance object is counted. Furthermore, the number of instances where the downstream governance object also experienced anomalies in the historical instances is counted to calculate the anomaly propagation correlation degree. Only when the downstream data processing instance actually reads the data generated by the upstream anomaly handling instance is it included in the anomaly propagation correlation degree statistics to avoid incorrectly identifying anomalies that are only temporally adjacent but have no actual data transmission relationship as propagation anomalies. Using governance objects as nodes and the directed relationships between governance objects as directed edges, the anomaly propagation correlation degree is configured to the corresponding directed edges. The normal monitoring benchmark, current processing instance, input source record, and data processing dependency relationship of each governance object are associated with the corresponding nodes to obtain a risk propagation correlation graph.
[0048] The formula for calculating the correlation degree of abnormal propagation is:
[0049]
[0050] In the formula, It is the object of governance To the governance target The correlation of abnormal propagation It is an upstream governance target An anomaly occurred and the corresponding output data was actually handled by the downstream governance object. Number of historical instances read Downstream governance targets in historical examples The same number of instances that experienced anomalies.
[0051] S2.2: Read the risk propagation correlation diagram, compare the current processing instance with the normal monitoring benchmark, obtain the numerical feature deviation, structural deviation, field distribution deviation and task status deviation, calculate the current local anomaly degree and determine the current deviation data range, and obtain the current risk propagation status data.
[0052] Specifically, the process involves reading the current processing instance and normal monitoring baseline corresponding to each governance object in the risk propagation correlation diagram, calculating the current numerical feature deviation for the numerical monitoring characteristics of the current processing instance of the governance object, reading the current structural fingerprint (recording the structural deviation as 0 if it belongs to the normal structural fingerprint set, otherwise recording it as 1), calculating the field distribution deviation for key fields requiring distribution monitoring based on the current field binning probability and the normal field binning probability (recording the task status deviation as 0 if the current processing instance completes normally, and recording it as 1 if the task fails, terminates abnormally, or the output data is unreadable), and then combining the numerical feature deviation, structural deviation, field distribution deviation, and task status deviation. The system constructs a current anomaly feature vector and calculates the current local anomaly degree. It determines the current deviation data range based on the output data batch, output data partition, and record location identifier corresponding to the anomaly features that deviated. If a specific record or data partition can be located, that record or data partition is determined as the current deviation data range. If further location is not possible but the anomaly involves the entire output data batch, the entire output data batch is determined as the current deviation data range. When the task execution fails and no valid output data is generated, only the task status anomaly is recorded, and the current deviation data range is not determined. The system correlates the current anomaly feature vector, the current local anomaly degree, the current deviation data range, and the current actual data transmission relationship to obtain the current risk propagation status data.
[0053] The current formula for calculating the deviation of numerical features is:
[0054]
[0055] In the formula, It is the object of governance In the current monitoring cycle The deviation of the current numerical feature corresponding to the m-th numerical monitoring feature. It is the object of governance The current processing instance's first Numerical monitoring characteristic value, when and When both are zero, the deviation of the corresponding current numerical feature is recorded as 0.
[0056] The formula for calculating field distribution deviation is:
[0057]
[0058] In the formula, It is the field distribution deviation. It is the first Number of bins for each key field It is the object of governance In the current monitoring cycle The Middle The key field corresponds to the first The current field binning probability of each bin. It is the object of governance No. The key field corresponds to the first Binning probability of normal fields in each bin.
[0059] The current anomaly feature vector is:
[0060]
[0061] In the formula, It is the current abnormal feature vector.
[0062] The formula for calculating the degree of local anomaly is as follows:
[0063]
[0064] In the formula, This indicates the current level of local anomaly. It is the th in the current abnormal feature vector Item abnormal feature value, It represents the actual number of abnormal features involved in anomaly detection.
[0065] S3. Assess data governance risks based on the current risk propagation status data, obtain data governance risk assessment results, determine risk objects to be verified based on data governance risk assessment results, and obtain a set of risk objects to be verified.
[0066] S3.1: Read the current risk propagation status data, calculate the current deviation data impact coverage, current abnormal propagation intensity and current propagation evidence, obtain the data governance risk value and abnormal source location value, determine candidate abnormal source data processing instances, determine the upper boundary of normal risk based on historical normal processing instances, and obtain the data governance risk assessment results.
[0067] Specifically, the system reads the current local anomaly level, current deviation from the data range, anomaly propagation correlation, and current actual data transmission relationship from the current risk propagation status data. For upstream and downstream governance objects with existing data transmission relationships, it counts the number of records actually read by the downstream governance object that fall within the current deviation from the data range of the upstream governance object, and also counts the total number of upstream input records actually read by the current processing instance of the downstream governance object, calculating the current deviation data impact coverage. When the current deviation from the data range of the upstream governance object is the entire output data batch, the number of records actually read by the current processing instance of the downstream governance object from that output data batch is determined as the number of records belonging to the current deviation from the data range of the upstream governance object. When there is no current actual data transmission relationship or the upstream governance object has not generated valid output data, the current deviation data impact coverage is recorded as 0. The system calculates the current anomaly propagation intensity based on the anomaly propagation correlation and the current deviation data impact coverage, determining the current propagation path only along directed edges where current actual data transmission exists and the current deviation data impact coverage is greater than 0, ensuring that the same propagation path does not repeatedly pass through the same governance object. For any current propagation path from an upstream governance object to the current governance object, the system calculates the current local anomaly level of the upstream governance object. The current anomaly propagation strength is multiplied by the current anomaly propagation strength of each directed edge in the path, and the maximum value is selected from all current propagation paths to obtain the current propagation evidence. Based on the current local anomaly level and the current propagation evidence, a data governance risk value is calculated, and the anomaly source location value of the data processing instance corresponding to each current anomaly governance object is calculated. The data processing instance corresponding to the governance object with the largest anomaly source location value is identified as a candidate anomaly source data processing instance, and the corresponding candidate anomaly source governance object and candidate anomaly source data processing instance identifier are recorded. When multiple data processing instances have the same anomaly source location value, all are identified as candidate anomaly source data processing instances. For historical normal processing instances... The historical normal local anomaly degree is calculated based on the deviation of monitoring characteristics from the normal monitoring benchmark, and the historical normal risk value is calculated in combination with the actual data transmission relationship of the corresponding historical processing instance to form a historical normal risk value sequence. According to the uniformly set normal risk upper quantile level, the normal risk upper boundary is determined from the historical normal risk value sequence. The governance object identifier, data processing instance identifier, current data governance risk value, normal risk upper boundary, candidate anomaly source governance object, main anomaly characteristics, current deviation data range, candidate anomaly source data processing instance identifier and current propagation path are associated to obtain the data governance risk assessment result.
[0068] The formula for calculating the impact of data deviation on coverage is:
[0069]
[0070] In the formula, It is the deviation from the data that affects the coverage. It is a downstream governance target The actual data read belongs to the governance object. The number of records currently deviating from the data range. It is a downstream governance target The total number of upstream input records actually read by the current processing instance.
[0071] The formula for calculating the intensity of abnormal propagation is:
[0072]
[0073] In the formula, It is the object of governance To the governance target The current abnormal propagation intensity, It is the correlation degree of abnormal propagation.
[0074] The formula for calculating evidence of dissemination is:
[0075]
[0076] In the formula, It is about disseminating evidence. From the upstream governance target Pointing to the governance object The current propagation path, It is the object of governance In the current monitoring cycle The current degree of local anomaly, It is the object of governance To the governance target The directed edges belong to the current propagation path. , This is the current transmission path. Each directed edge corresponds to the product of the current anomaly propagation intensity. When no upstream propagation path meets the conditions, then... Record it as 0.
[0077] The formula for calculating the data governance risk value is:
[0078]
[0079] In the formula, It is the object of governance In the current monitoring cycle The current data governance risk value, This indicates the current level of local anomaly.
[0080] The formula for calculating the anomaly source location value is:
[0081]
[0082] In the formula, It is the object of governance In the current monitoring cycle The anomaly source location value.
[0083] S3.2: Read the data governance risk assessment results, compare the current data governance risk value with the upper boundary of normal risk, determine the normal monitoring objects and the risk objects to be verified, and record the data processing instances of candidate abnormal sources, the current deviation from the data range and the current propagation path to obtain the set of risk objects to be verified.
[0084] Specifically, the current data governance risk value and the upper boundary of normal risk are read from the data governance risk assessment results. The current data governance risk value is compared with the corresponding upper boundary of normal risk. When the current data governance risk value is less than or equal to the upper boundary of normal risk, the corresponding data processing instance is identified as a normal monitoring object, and the current risk judgment result is recorded. When the current data governance risk value is greater than the upper boundary of normal risk, the corresponding data processing instance is identified as a risk object to be verified. For the risk object to be verified, the corresponding governance object identifier, data processing instance identifier, current data governance risk value, upper boundary of normal risk, candidate anomaly source governance object, main anomaly characteristics, current deviation from data range, and current propagation path are read. Based on the candidate anomaly source governance object and the data processing instance identifier, the corresponding input data version identifier is further associated. Alternatively, input data snapshot identifier, task version, task execution parameters, and data processing dependencies to determine the input data and execution conditions required for subsequent anomaly recalculation; when the current deviation from the data range can be located to a specific data partition or record range, record the corresponding deviation data range; when the current deviation data range cannot be further located, retain the corresponding output data batch identifier for subsequent verification based on complete input data; associate the governance object identifier, data processing instance identifier, current data governance risk value, normal risk upper boundary, candidate anomaly source governance object, main anomaly characteristics, current deviation data range, current propagation path, input data version identifier or input data snapshot identifier, task version, task execution parameters, and data processing dependencies of the risk object to be verified to obtain the set of risk objects to be verified.
[0085] S4. Perform anomaly recalculation verification based on the set of risk objects to be verified, obtain risk verification results, generate data governance risk warnings based on the risk verification results, and obtain the final data governance risk warning results.
[0086] S4.1: Read the set of risk objects to be verified and the basic dataset of data governance monitoring, construct abnormal dependency data slices for the current deviation data range that can be located and recalculate them. If partial recalculation is not possible, perform a complete recalculation or status verification. Determine the verification status based on the recalculated data governance risk value and the upper boundary of normal risk, and obtain the risk verification result.
[0087] Specifically, the process reads the set of risk objects to be verified and obtains the current deviation data range, data processing dependencies, input data version or input data snapshot, task version, and task execution parameters corresponding to the candidate anomaly source governance objects. When the current deviation data range can be determined and the corresponding data processing task supports partial re-execution, based on the mapping relationships of association fields, grouping fields, deduplication fields, window ranges, and input / output fields in the data processing dependencies, it traces all input records involved in the calculation of the corresponding output results from the current deviation data range back to the input end. When the input records are still generated by upstream data processing operations, it continues to trace upstream along the data processing dependencies until it reaches the source input data actually read by the current processing instance. Input records that have direct or recursive data dependencies with the current data deviation range are merged to form an abnormal dependency data slice. In an isolated execution area that does not cover the formal output data, the corresponding data processing operations are re-executed on the abnormal dependency data slice using the same task version and execution parameters as the original data processing instance, obtaining partially recalculated output data. Data outside the current data deviation range in the original output data is kept unchanged, and the corresponding deviation range is replaced with the partially recalculated output data, forming a recalculated and reconstructed output state used only for risk verification. Based on the recalculated and reconstructed output state, the number of output records, abnormal record rate, key field null value rate, structural fingerprint, field binning probability, and other parameters that can be determined from the recalculation results are re-obtained. The execution status of the task is monitored and the corresponding deviation is recalculated. For task processing time and the number of complete input records that cannot be directly reflected by partial recalculation, the monitoring values corresponding to the original data processing instance are kept unchanged, and then the degree of local anomaly is recalculated. The actual data transmission relationship and the coverage of the current deviation data impact of the original data processing instance are kept unchanged. The degree of local anomaly of the candidate anomaly source governance object is replaced with its original current degree of local anomaly, and the propagation evidence and the risk value of the corresponding risk object to be verified are recalculated. When the current deviation data range cannot be clearly located or the corresponding task cannot be partially recalculated, but the original input data version or input data snapshot can be read again, the original input data and the corresponding deviation data are used in the isolated execution area. The same task version and execution parameters are used to re-execute the complete data processing task, and the recalculated data governance risk value is calculated based on the complete recalculation output results. When the recalculation task is completed normally, the recalculated output data can be read normally, and the recalculated data governance risk value is less than or equal to the upper boundary of normal risk, the corresponding risk object to be verified is identified as a recoverable abnormal object. When the recalculation task still fails, terminates abnormally, or the output data is unreadable, or the recalculated data governance risk value is greater than the upper boundary of normal risk, the corresponding risk object to be verified is identified as a confirmed risk object. When the corresponding task cannot be re-executed or the original input data cannot be read again, the input data status, output data status, and task execution log of the original data processing instance are re-read.When the original monitored anomaly has disappeared, the task execution log does not contain a corresponding fault, and both input and output data can be read normally, the corresponding risk object to be verified is identified as a monitoring error object; otherwise, it is identified as a confirmed risk object. The risk object to be verified, the verification method, the risk value of the recalculated data governance or the duplicate status verification result, and the corresponding verification status are associated to obtain the risk verification result.
[0088] S4.2: Read the risk verification results, do not issue warnings for objects with monitoring errors, generate recovery prompts for recoverable abnormal objects, determine the data governance risk level for confirmed risk objects based on the actual operating status of downstream governance objects, update the normal monitoring benchmark and abnormal propagation correlation, and obtain the final data governance risk warning results.
[0089] Specifically, when reading risk verification results, for objects with monitoring errors, no data governance risk warning is generated, but the corresponding monitoring error information is recorded. For recoverable abnormal objects, data governance recovery prompts are generated, and the original data governance risk value, recalculated data governance risk value, and verification method are recorded. The recalculated data generated in the isolated execution area is only used for risk verification and does not cover the formal output data. For objects with confirmed risks, the running status of the actual downstream governance objects is read according to the corresponding current propagation path. When the confirmed risk has caused at least one downstream governance object that actually reads its output data to experience task failure, abnormal termination, or unreadable output data, it is determined to be a Level 3 data governance risk. When the conditions for Level 3 data governance risk are not met, but the data governance risk value of at least one actual downstream governance object exceeds the corresponding normal risk upper boundary, or the task is retried or the output status is abnormal, it is determined to be a Level 3 data governance risk. If the output data is still readable, it is identified as a Level 2 data governance risk; if neither the Level 3 nor Level 2 data governance risk conditions are met, it is identified as a Level 1 data governance risk. A final data governance risk warning result is generated based on the identified risk level. After the current data processing instance completes fault handling and data integrity verification, data processing instances that have completed their tasks normally, whose output data is readable, and which have not been identified as abnormal are added to the historical normal processing instances. Data processing instances corresponding to confirmed risks are added to the historical abnormal processing instances; confirmed risk data is not used as a normal baseline sample. The normal monitoring baseline is updated based on the newly added historical normal processing instances, and the abnormal propagation correlation is updated based on the newly added historical abnormal processing instances and the actual upstream and downstream data transmission relationships. The basic data for data governance analysis and the risk propagation correlation diagram are also updated.
[0090] It should be noted that the final data governance risk warning result should include at least the risk governance object identifier, data processing instance identifier, candidate anomaly source governance object, main anomaly characteristics, data governance risk value, current propagation path, affected downstream governance objects, data governance risk level, risk occurrence time, and risk verification result.
[0091] Reference Figure 2 This embodiment also provides a data governance risk early warning system based on big data mining, including:
[0092] The monitoring and acquisition module is used to collect data processing operation data and data status data, establish data processing dependencies, and form a basic dataset for data governance monitoring.
[0093] The benchmark analysis module is used to filter historical normal and abnormal handling instances, establish normal monitoring benchmarks, and form basic data for data governance analysis.
[0094] The propagation identification module is used to analyze the data transmission relationships of historical anomaly instances, calculate the anomaly propagation correlation degree, and construct a risk propagation correlation diagram.
[0095] The risk assessment module is used to identify the current abnormal state, calculate the intensity of abnormal propagation, evidence of propagation and data governance risk value, and determine the risk objects to be verified.
[0096] The verification and early warning module is used to perform partial or complete recalculation of the risk objects to be verified, determine the verification status, and generate the final data governance risk early warning result.
[0097] In summary, this invention establishes a data governance monitoring foundation and normal monitoring benchmark by collecting data processing operation status, data status, and actual upstream and downstream reading relationships. It then identifies abnormal propagation status by combining historical anomaly propagation patterns with current deviations from the actual impact range of the data, enabling continuous analysis of data processing anomalies, propagation risks, and candidate anomaly sources. Furthermore, it confirms and classifies risk objects through risk threshold screening, anomaly-dependent data slice recalculation, and complete recalculation verification. Simultaneously, it uses verification results to update the normal monitoring benchmark and anomaly propagation correlation, improving the accuracy of data governance risk identification, anomaly source location, and propagation judgment, reducing redundant recalculation of irrelevant data, and enhancing risk verification efficiency and the reliability of early warning results.
[0098] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.
Claims
1. A data governance risk early warning method based on big data mining, characterized in that, include: Collect data on data governance operations and establish data relationships to obtain a basic dataset for data governance monitoring. Based on the basic dataset for data governance monitoring, establish a data governance analysis benchmark and obtain benchmark data for data governance analysis. Based on the benchmark data of data governance analysis, anomaly propagation relationships are mined to obtain a risk propagation correlation graph. The current data processing anomalies are identified through the risk propagation correlation graph to obtain the current risk propagation status data. Based on the current risk propagation status data, assess data governance risks, obtain data governance risk assessment results, determine risk objects to be verified based on data governance risk assessment results, and obtain a set of risk objects to be verified. Anomaly recalculation verification is performed based on the set of risk objects to be verified, and the risk verification results are obtained. Data governance risk warnings are generated based on the risk verification results, and the final data governance risk warning results are obtained.
2. The data governance risk early warning method based on big data mining as described in claim 1, characterized in that, The process of collecting data for governance and establishing data relationships to obtain a basic dataset for data governance monitoring refers to collecting data processing operation data and data status data that includes input data version identifiers or input data snapshot identifiers, task versions and task execution parameters, establishing directed relationships between governance objects, data processing dependencies and input source records, and associating data by data processing instances to obtain a basic dataset for data governance monitoring.
3. The data governance risk early warning method based on big data mining as described in claim 2, characterized in that, The process of establishing a data governance analysis benchmark based on the basic dataset of data governance monitoring involves reading the basic dataset of data governance monitoring, filtering historical normal processing instances, historical abnormal processing instances, and current processing instances, establishing a normal monitoring benchmark based on historical normal processing instances, and associating input source records, data processing dependencies, and directed relationships to obtain basic data for data governance analysis.
4. The data governance risk early warning method based on big data mining as described in claim 3, characterized in that, The process of mining abnormal propagation relationships based on data governance analysis benchmark data to obtain a risk propagation correlation graph involves reading basic data for data governance analysis, calculating the abnormal propagation correlation degree based on the actual data transmission relationship of historical abnormal handling instances, configuring the abnormal propagation correlation degree with governance objects as nodes and directed relationships as edges, and associating normal monitoring benchmarks and current handling instances to obtain a risk propagation correlation graph.
5. The data governance risk early warning method based on big data mining as described in claim 4, characterized in that, The step of identifying current data processing anomalies through a risk propagation correlation diagram and obtaining current risk propagation status data refers to reading the risk propagation correlation diagram, comparing the current processing instance with the normal monitoring benchmark, obtaining the deviation of numerical features, structural deviation, field distribution deviation, and task status, calculating the current local anomaly degree, determining the current deviation data range, and obtaining the current risk propagation status data.
6. The data governance risk early warning method based on big data mining as described in claim 5, characterized in that, The process of assessing data governance risk based on current risk propagation status data and obtaining data governance risk assessment results involves reading current risk propagation status data, calculating the current deviation data impact coverage, current abnormal propagation intensity, and current propagation evidence, obtaining data governance risk values and abnormal source location values, identifying candidate abnormal source data processing instances, determining the upper boundary of normal risk based on historical normal processing instances, and obtaining data governance risk assessment results.
7. The data governance risk early warning method based on big data mining as described in claim 6, characterized in that, The step of determining the risk objects to be verified through the data governance risk assessment results and obtaining the set of risk objects to be verified refers to reading the data governance risk assessment results, comparing the current data governance risk value with the upper boundary of normal risk, determining the normal monitoring objects and the risk objects to be verified, and recording the data processing instances of candidate abnormal sources, the current deviation from the data range, and the current propagation path to obtain the set of risk objects to be verified.
8. The data governance risk early warning method based on big data mining as described in claim 7, characterized in that, The step of performing anomaly recalculation verification based on the set of risk objects to be verified and obtaining risk verification results refers to reading the set of risk objects to be verified and the basic dataset of data governance monitoring, constructing anomaly dependency data slices for the currently locatable deviation data range and recalculating them. If partial recalculation is not possible, a complete recalculation or status verification is performed. The verification status is determined based on the recalculated data governance risk value and the upper boundary of normal risk, and the risk verification results are obtained.
9. The data governance risk early warning method based on big data mining as described in claim 8, characterized in that, The process of generating a data governance risk warning based on the risk verification results and obtaining the final data governance risk warning result refers to reading the risk verification results, not issuing warnings for objects with monitoring errors, generating recovery prompts for recoverable abnormal objects, determining the data governance risk level for confirmed risk objects based on the actual operating status of downstream governance objects, and updating the normal monitoring benchmark and abnormal propagation correlation to obtain the final data governance risk warning result.
10. A data governance risk early warning system based on big data mining, comprising a data governance risk early warning method based on big data mining as described in any one of claims 1 to 9, characterized in that, include: The monitoring and acquisition module is used to collect data processing operation data and data status data, establish data processing dependencies, and form a basic dataset for data governance monitoring. The benchmark analysis module is used to filter historical normal and abnormal handling instances, establish normal monitoring benchmarks, and form basic data for data governance analysis. The propagation identification module is used to analyze the data transmission relationships of historical anomaly instances, calculate the anomaly propagation correlation degree, and construct a risk propagation correlation diagram. The risk assessment module is used to identify the current abnormal state, calculate the intensity of abnormal propagation, evidence of propagation and data governance risk value, and determine the risk objects to be verified; The verification and early warning module is used to perform partial or complete recalculation of the risk objects to be verified, determine the verification status, and generate the final data governance risk early warning result.
