Enterprise-level data asset quality evaluation and optimization method

Abstract

The application provides an enterprise-level data asset quality evaluation and optimization method, relates to the technical field of data asset management, and comprises the following steps: acquiring index configuration data, hierarchical object mapping data and scene state data corresponding to a target data asset; collecting multi-source quality evidence according to data source mapping and verifying to obtain original quality index values; associating the original quality index values to corresponding hierarchical objects and standardizing the original quality index values into quality standard scores; determining a dynamic weight vector based on the index configuration data and the scene state data; aggregating in a hierarchical order from a field, a table, a data set to an asset catalog to obtain an asset catalog quality score and a quality level; generating a management task according to a low-score index and a blood relationship influence range, and triggering retesting and rewriting after the management is completed, so that quantitative evaluation and closed-loop optimization of data asset quality are realized.

Description

Technical Field

[0001] This application belongs to the field of data asset management technology, specifically involving a method for enterprise-level data asset quality assessment and optimization. Background Technology

[0002] As enterprises continue to build their digital systems, data from business databases, data warehouses, lake warehouse platforms, log systems, data service interfaces, and third-party data sources are constantly converging into data platforms or digital asset management platforms. During the cross-system flow, processing, sharing, and reuse of data assets, their integrity, accuracy, consistency, timeliness, security, and compliance directly impact subsequent data analysis, business processing, risk identification, and data asset operation.

[0003] Existing data quality assessment methods typically involve metadata collection, data profiling, rule validation, master data comparison, lineage analysis, and access control auditing to check the quality of data tables, fields, or datasets, and output quality reports or problem lists based on pre-set quality rules. Some systems also use expert scoring, the analytic hierarchy process (AHP), or fixed-weight templates to assign weights to different quality indicators, thereby generating table-level or dataset-level data quality scores.

[0004] However, in enterprise-level data asset scenarios, data objects often have multiple hierarchical levels, such as fields, tables, datasets, and asset catalogs, and the same data quality issue may affect multiple downstream assets along the data lineage. Existing assessment methods typically focus more on the quality detection of a single table, single database, or single dataset. The lack of unified indicator coding, hierarchical object identification, and calculation methods among heterogeneous quality evidence makes it difficult for field-level quality issues to reliably propagate to asset catalog-level quality conclusions. Furthermore, fixed weights are difficult to adapt to changes in different business scenarios, asset importance, usage frequency, and historical issue status, making it difficult to accurately express the impact of the same quality issue in different asset scenarios. Summary of the Invention

[0005] This application provides a method for enterprise-level data asset quality assessment and optimization to address the aforementioned problems.

[0006] The technical solution adopted in this application is as follows: This application provides an enterprise-level data asset quality assessment and optimization method, applied to an enterprise-level data asset quality assessment and optimization system. The enterprise-level data asset quality assessment and optimization system is located on a digital asset management platform and is communicatively connected to metadata collection components, data profiling components, business rule components, lineage analysis components, security audit components, and governance task components. The method includes: Obtain the indicator configuration data, hierarchical object mapping data, and scene status data corresponding to the target data asset. The indicator configuration data includes indicator definitions, indicator thresholds, data source mappings, and quality rules. The hierarchical object mapping data is used to characterize the attribution and lineage relationships between fields, tables, datasets, and asset catalogs. According to the data source mapping, collect multi-source quality evidence corresponding to the target data asset, and verify the multi-source quality evidence based on the quality rules to obtain the original quality indicator value corresponding to the indicator definition. The multi-source quality evidence includes at least one of metadata, data analysis results, business rule verification results, lineage analysis results, and security audit results. Based on the indicator definition and the hierarchical object mapping data, the original quality indicator value is associated with the corresponding field, table, dataset or asset catalog hierarchical object, and the original quality indicator value is standardized and converted according to the indicator threshold to obtain the quality standard score; Based on the indicator configuration data and the scenario status data, a dynamic weight vector corresponding to each indicator in the current evaluation period is determined. The scenario status data is used to represent at least one of the following: business scenario, asset importance, usage frequency, and historical problem occurrence rate. Based on the quality standard score, the dynamic weight vector, and the hierarchical object mapping data, the data is aggregated level by level in the order of fields, tables, datasets to asset catalogs to obtain the asset catalog quality score and quality level corresponding to the target data asset. Based on the asset catalog quality score, the quality level, and the low-score indicators and lineage influence range determined during the hierarchical aggregation process, a governance task is generated. Upon completion of the governance task, a retest is triggered, and the retest results are written back as the dynamic weight calculation input for the next evaluation cycle.

[0007] This application also provides an enterprise-level data asset quality assessment and optimization system, including: The configuration acquisition unit is used to acquire indicator configuration data, hierarchical object mapping data and scene status data corresponding to the target data asset. The indicator configuration data includes indicator definition, indicator threshold, data source mapping and quality rules. The hierarchical object mapping data is used to characterize the affiliation relationship and lineage influence relationship between fields, tables, datasets and asset catalogs. An evidence processing unit is used to collect multi-source quality evidence corresponding to the target data asset according to the data source mapping, and to verify the multi-source quality evidence based on the quality rules to obtain the original quality indicator value corresponding to the indicator definition. The multi-source quality evidence includes at least one of metadata, data analysis results, business rule verification results, lineage analysis results, and security audit results. The standardization processing unit is used to associate the original quality indicator value with the corresponding field, table, dataset or asset catalog hierarchical object according to the indicator definition and the hierarchical object mapping data, and to standardize and convert the original quality indicator value according to the indicator threshold to obtain the quality standard score. The weight calculation unit is used to determine the dynamic weight vector corresponding to each indicator in the current evaluation period based on the indicator configuration data and the scenario status data. The scenario status data is used to represent at least one of business scenario, asset importance, usage frequency and historical problem occurrence rate. The hierarchical aggregation unit is used to aggregate data level by level based on the quality standard score, the dynamic weight vector, and the hierarchical object mapping data, in the hierarchical order of fields, tables, datasets to asset catalog, to obtain the asset catalog quality score and quality level corresponding to the target data asset. The governance feedback unit is used to generate governance tasks based on the asset catalog quality score, the quality level, and the low-score indicators and lineage influence range determined during the hierarchical aggregation process. After the governance task is completed, a retest is triggered, and the retest results are written back as the dynamic weight calculation input for the next evaluation cycle.

[0008] Due to the adoption of the above technical solution, the beneficial effects achieved by this application are as follows: This application acquires the indicator configuration data, hierarchical object mapping data, and scenario status data corresponding to the target data assets, and collects multi-source quality evidence based on the data source mapping. This enables metadata, data analysis results, business rule verification results, lineage analysis results, and security audit results to enter the subsequent calculation process according to a unified indicator definition, thereby avoiding the inability to uniformly evaluate quality evidence from different sources due to inconsistent standards.

[0009] This application, by associating the original quality indicator values ​​with field, table, dataset, or asset catalog level objects and standardizing them according to indicator thresholds, can convert quality evidence with different dimensions and value ranges into a unified quality standard score, thus providing a calculable and recalcible quality transmission basis between field-level, table-level, dataset-level, and asset catalog-level objects.

[0010] This application determines a dynamic weight vector based on indicator configuration data and scenario status data, enabling at least one of the following factors—business scenario, asset importance, usage frequency, and historical problem occurrence rate—to participate in the weight calculation. This allows the same quality indicator to have different degrees of computational influence in different business scenarios and asset states, avoiding the mismatch between quality scores and actual risks caused by using only fixed weights.

[0011] This application aggregates data level by level based on quality standard scores, dynamic weight vectors, and hierarchical object mapping data, following the hierarchical order of fields, tables, datasets, and asset catalogs. This allows issues such as missing fields, errors, delays, lineage gaps, or permission violations to be propagated to the upper-level asset catalog according to object ownership and lineage influence relationships, thereby obtaining an asset catalog quality score and quality level that can reflect the quality status of multi-level asset objects.

[0012] This application enables parallel data retrieval and rule matching during the collection of multi-source quality evidence according to indicator number, applicable level, data source and quality rules. This allows quality evidence generated by different components to be mapped to the same indicator scope and level object, reducing mismatch between multi-source quality evidence.

[0013] This application improves the stability and interpretability of the quality assessment process by setting up uncalculable indicators, governance blocking indicators, data supplementation tasks, or collection and repair tasks for situations where the data source is unavailable, the indicator denominator is zero, or the fields required for indicator calculation are missing.

[0014] This application aggregates field-level quality scores, table-level quality scores, dataset-level quality scores, and asset catalog quality scores at each level, and combines these with field importance, table importance, dataset importance, and asset value coefficients to match the contribution of objects at different levels to asset quality scoring with their actual scope of influence.

[0015] This application generates governance tasks based on low-scoring indicators, abnormal records, evidence samples, and the scope of kinship influence. After the governance tasks are completed, a retest is triggered, and the retest results are written back as the dynamic weight calculation input for the next evaluation cycle. This can form a closed loop of quality assessment, governance treatment, retest verification, and weight feedback, enabling subsequent evaluations to be dynamically corrected using historical governance effects and problem recurrence status. Attached Figure Description

[0016] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A schematic diagram of the system architecture of the enterprise-level data asset quality assessment and optimization system provided in the embodiments of this application; Figure 2 A flowchart illustrating the enterprise-level data asset quality assessment and optimization method provided in this application embodiment; Figure 3 A schematic diagram of the module structure of the enterprise-level data asset quality assessment and optimization system provided in this application embodiment. Detailed Implementation

[0017] The technical solution of this application will be further described below with reference to the accompanying drawings and embodiments. It should be understood that the following embodiments are only used to illustrate this application and are not intended to limit the scope of protection of this application.

[0018] In the description of the embodiments of this application, the terms "first," "second," etc., are only used to distinguish different objects and should not be construed as indicating or implying relative importance, nor should they be construed as limiting quantity or order. The term "multiple" refers to two or more. The term "and / or" is used to describe the relationship between related objects, indicating that three relationships can exist, such as A and / or B, which can mean A existing alone, A and B existing simultaneously, or B existing alone. The term "quality evidence" can be understood as raw data, statistical results, verification results, log records, lineage records, permission records, or governance records used to characterize the quality status of data assets. The term "hierarchical object" can be understood as any object in a field, table, dataset, or asset catalog.

[0019] In enterprise-level data asset quality assessment scenarios, target data assets can include at least one of the following: business database tables, data warehouse tables, lake warehouse tables, thematic datasets, indicator datasets, data service interfaces, report data, scheduled task data, and asset catalog data. Target data assets typically include the hierarchical relationships between fields, tables, datasets, and asset catalogs, and may also include lineage relationships such as field lineage, table lineage, task dependencies, interface dependencies, and report references.

[0020] Figure 1 This illustrates the system architecture for enterprise-level data asset quality assessment and optimization provided in an embodiment of this application. For example... Figure 1 As shown, the enterprise-level data asset quality assessment and optimization system can be located on the digital asset management platform and communicate with the metadata collection component, data analysis component, business rule component, lineage analysis component, security audit component, and governance task component.

[0021] The metadata collection component can be used to collect metadata such as table name, field name, field type, field length, primary key, unique key, partition information, asset owner, classification and grading tags, and lifecycle status of target data assets. The data profiling component can output profiling results such as the number of null values, duplicate values, format distribution, enumeration distribution, value range distribution, and field statistical distribution. The business rule component can perform rule verifications such as field non-nullability, format, value range, uniqueness, cross-table consistency, business closure, amount balance, and process status constraints. The lineage analysis component can parse SQL, ETL / ELT tasks, scheduling dependencies, API mappings, data service subscription relationships, and manually entered lineage information. The security audit component can obtain access logs, authorization records, export records, approval records, de-identification policies, encryption policies, and audit link records. The governance task component can generate, distribute, track, and retest data governance tasks.

[0022] In one possible implementation, an enterprise-level data asset quality assessment and optimization system may include a configuration acquisition module, an evidence processing module, a standardization processing module, a weight calculation module, a hierarchical aggregation module, and a governance feedback module. The configuration acquisition module acquires indicator configuration data, hierarchical object mapping data, and scenario status data; the evidence processing module collects multi-source quality evidence and performs quality rule verification; the standardization processing module converts raw quality indicator values ​​into quality standard scores; the weight calculation module determines the dynamic weight vector; the hierarchical aggregation module performs hierarchical aggregation according to the order of fields, tables, datasets, and asset catalogs; and the governance feedback module generates governance tasks, triggers retesting, and writes back the retest results.

[0023] Figure 2 This document illustrates a method and flowchart for enterprise-level data asset quality assessment and optimization provided in an embodiment of this application. For example... Figure 2 As shown, the method may include the following steps.

[0024] S100: Obtain the indicator configuration data, hierarchical object mapping data, and scene status data corresponding to the target data asset.

[0025] Specifically, indicator configuration data can include indicator definitions, indicator thresholds, data source mappings, and quality rules. Indicator definitions can include indicator number, primary indicator name, secondary indicator name, applicable level, calculation formula, positive or negative attributes, and whether it is mandatory. Indicator thresholds can include acceptable lower limits, optimal upper limits, blocking thresholds, and differentiated thresholds configured according to asset level, business domain, and data type. Data source mappings can be used to clarify the source of metadata collection, data profiling, business rule engine, lineage analysis, security audit, scheduling logs, or governance work orders corresponding to each indicator. Quality rules can include executable rules such as field non-nullability, format, value range, uniqueness, cross-table consistency, timeliness SLA, and permission compliance.

[0026] Hierarchical object mapping data can be used to represent the attribution and lineage relationships between fields, tables, datasets, and asset catalogs. For example, a field may belong to a business table, a business table may belong to a subject dataset, and a subject dataset may belong to an asset catalog. Furthermore, hierarchical object mapping data can also record lineage relationships such as upstream sources of fields, downstream table dependencies, API output relationships, and report reference relationships.

[0027] Scenario status data can be used to characterize at least one of the following: business scenario, asset importance, usage frequency, and historical problem occurrence rate. For example, business scenarios can include transaction-related, user-related, risk control-related, regulatory reporting-related, and operational analysis-related; asset importance can be determined based on asset catalog hierarchical tags, regulatory impact, production impact, business revenue impact, and the number of downstream dependencies; usage frequency can be determined based on the number of data service API calls, SQL queries, BI report references, downstream scheduling reads, and subscription pushes in the past 30 days; and historical problem occurrence rate can be determined based on the number of rule validation failures, governance work orders, audit findings, business complaints, and retest failure records in the past 90 days.

[0028] In one embodiment, the indicator configuration data can adopt the data structure shown in Table 1 below.

[0029] Table 1

[0030] S200: Collect multi-source quality evidence corresponding to the target data asset according to the data source mapping, and verify the multi-source quality evidence based on the quality rules to obtain the original quality indicator value corresponding to the indicator definition.

[0031] Specifically, the enterprise-level data asset quality assessment and optimization system can determine the data source, quality rules, and minimum calculation unit for each indicator based on the indicator number and applicable level in the indicator definition. For indicators that can be drilled down to the field level, fields are preferred as the minimum calculation unit; for indicators that cannot be directly drilled down to the field level, tables, tasks, APIs, reports, or business items can be used as the minimum calculation unit.

[0032] Multi-source quality evidence can include at least one of the following: metadata, data profiling results, business rule verification results, lineage analysis results, security audit results, scheduling logs, and governance work order records. The system can obtain corresponding quality evidence from different components in parallel based on data source mapping, and match the obtained quality evidence with the corresponding quality rules according to the indicator number to obtain the original quality indicator value.

[0033] In one embodiment, the secondary indicators, calculation rules, and data sources can be configured as shown in Table 2 below.

[0034] Table 2

[0035]

[0036]

[0037] In a specific example, the customer master data table `customer_info` has 10,000 records. The required fields are `customer_id`, `customer_name`, `mobile`, and `cert_no`, with corresponding importance values ​​of 0.30, 0.20, 0.20, and 0.30, respectively. Data analysis shows that the number of records with non-empty values ​​for each of the four fields are 10,000, 9,800, 9,500, and 9,700, respectively. This raises the question of completeness. The corresponding standard score S is: .

[0038] In another example, the order details table `order_detail` has a warehouse entry SLA of writing the order within 30 minutes of its creation. In a certain evaluation window, there are 50,000 orders to be warehoused. After comparing the scheduling logs and timestamp fields, 47,250 records are found to satisfy `ingest_time - order_time ≤ 30 minutes`. Therefore, the timeliness is... Corresponding standard score The 2,750 records that did not meet the standards were entered into the problem details.

[0039] In another example, the data asset `dws_customer_profile` has 80 fields. Lineage analysis revealed that 72 fields can be traced back to explicit upstream fields. The asset should have 120 SQL / ETL dependency edges to parse, and 108 were successfully parsed. The target traceability depth is 3 levels, and the current maximum traceability depth is also 3 levels. (Based on the lineage integrity formula...) The corresponding standard score S is: .

[0040] S300, based on the indicator definition and the hierarchical object mapping data, the original quality indicator value is associated with the corresponding field, table, dataset or asset catalog hierarchical object, and the original quality indicator value is standardized and converted according to the indicator threshold to obtain the quality standard score.

[0041] Specifically, the system associates the original quality indicator values ​​with the corresponding level objects in the fields, tables, datasets, or asset catalogs based on the applicable level in the indicator definition and the object identifier in the level object mapping data. For example, indicators such as completeness, accuracy, uniqueness, and validity can be preferentially associated with the field level; indicators such as business coverage and asset value level can be associated with the dataset or asset catalog level; and indicators such as permission compliance and audit traceability can be associated with the fields, tables, interfaces, or asset catalog levels based on the access object.

[0042] For indicators where a higher original value R indicates better quality, the system can convert R to the [0,100] standard interval according to the lower acceptable limit and the upper optimal limit. The conversion formula can be expressed as follows: ; in, This represents the quality standard score of the i-th indicator. This represents the original quality index value of the i-th indicator. This represents the lower limit of the acceptable level for the i-th indicator. This represents the optimal upper limit for the i-th indicator. This means that the calculation result will be truncated within the interval [0, 100].

[0043] For inverse indicators where lower original values ​​generally indicate better quality, an inverse attribute can be configured in the indicator definition, and the result can be converted according to the corresponding inverse threshold. The converted quality standard score, along with the positive indicator, enters the subsequent dynamic weight calculation and hierarchical aggregation process.

[0044] In one possible implementation, and The order of priority can be determined as follows: "mandatory standards first, business rules second, statistical distribution as a supplement, and manual approval as a fallback." Mandatory standards can be derived from regulatory requirements or corporate policies; business rules can be derived from the business domain rule base; statistical distribution can be derived from historical assessment data or the distribution of similar assets; and manual approval can be used to handle scenarios where historical samples are lacking or new assets are being launched.

[0045] In one abnormal branch, when the data source corresponding to the original quality indicator value is unavailable, the indicator denominator is zero, or the fields required for indicator calculation are missing, the system will mark the corresponding indicator as an uncalculated indicator instead of directly recording it as 0 points. When an uncalculated indicator is a mandatory item, the system will identify it as a governance blocking indicator and indicate the reason for the blocking in the quality report. When an uncalculated indicator is not a mandatory item, the system can remove it from the current level of aggregation calculation and generate a corresponding data supplementation task or collection and repair task.

[0046] The governance and blocking indicators may include mandatory, non-calcifiable indicators, as well as mandatory indicators whose quality standard scores or original quality indicator values ​​are lower than the corresponding blocking thresholds. For the first... If a quality indicator is mandatory and exists in a state where it cannot be calculated, it can be identified as a governance and prevention indicator; or, if the indicator is mandatory and meets one of the following conditions: ; or: ; This can also be used as a governance and prevention indicator. Among them, Indicates the first The quality standard score for each quality indicator Indicates the first The standard score blocking threshold corresponding to each quality indicator Indicates the first The original quality index values ​​of each quality indicator. Indicates the first The original indicator value blocking threshold corresponding to each quality indicator.

[0047] For governance blocking indicators, the system can restrict the highest quality level of the corresponding asset catalog, or suspend the data service release, sharing application, or rating improvement process for that asset until the retest results meet the corresponding blocking release conditions.

[0048] S400, based on the indicator configuration data and the scenario status data, determine the dynamic weight vector corresponding to each indicator in the current evaluation period.

[0049] Specifically, the dynamic weight vector is not an independent scoring result. Instead, it is constrained by the indicator number, indicator level, mandatory indicator attributes, and threshold configuration in the indicator configuration data. It also incorporates scenario status data such as business scenario tags, asset importance, usage frequency over the past 30 days, and historical problem occurrence rate over the past 90 days. The system matches the business scenario tags with the weight adjustment coefficients according to the indicator number, and combines the static baseline weight, data distribution weight, and dynamic adjustment coefficient to output the dynamic weight vector, weight version number, and calculation basis for each asset object within the current evaluation period.

[0050] In this embodiment, the quality standard score distribution refers to the set of quality standard scores and their statistical status calculated for the same quality indicator across multiple evaluated hierarchical objects within the current evaluation period. The evaluated hierarchical objects may include at least one of fields, tables, datasets, and asset catalogs. For example, for the first... Each quality indicator corresponds to a certain quality indicator within the current evaluation period. The quality standard score of an object at each level can be expressed as: ; in, Indicates the first The quality indicator in the first Quality standard scores on each level of object Indicates the first The number of hierarchical objects involved in the calculation of each quality indicator within the current evaluation period. The quality standard score distribution of this indicator is constituted by at least one of the following: the set formed by the aforementioned quality standard scores, the distribution interval, the degree of dispersion, and the information entropy.

[0051] The data distribution weight refers to the indicator-level weight correction term determined based on the quality standard distribution of each quality indicator within the current evaluation period. It corresponds to the entropy weight calculation result in the dynamic adaptive weighting and is used to characterize the dispersion and distinguishing ability of the same quality indicator among multiple evaluated levels of objects. The more dispersed the quality standard distribution, the stronger the indicator's ability to identify quality differences between different levels of objects, and the higher the corresponding data distribution weight. Conversely, the more concentrated the quality standard distribution, the weaker the indicator's distinguishing ability within the current evaluation period, and the lower the corresponding data distribution weight.

[0052] In one possible implementation, for the first... For each quality indicator, the system can determine its data distribution weights based on the entropy weight method. First, for the... The quality indicator in the first The weighting of quality standard scores on each level of object is calculated: ; in, Indicates the first The quality indicator in the first The standard score weighting of each level of object. This represents an extremely small positive number used to avoid denominators of zero or logarithms that cannot be calculated.

[0053] Then, calculate the first... Information entropy of a quality indicator : ; in, Indicates the first Information entropy of a quality indicator. If , or the If the proportion of missing quality standard scores for a quality indicator exceeds a preset missing proportion threshold, the system can set the data distribution weight of that indicator to a preset default value or use the data distribution weight from the previous evaluation period.

[0054] Next, calculate the first... Information difference coefficient of each quality indicator : ; in, Indicates the first The information difference coefficient is the coefficient of variation for a quality indicator. The larger the information difference coefficient, the more significant the difference in the standard scores of that indicator among objects at different levels, and the stronger its ability to distinguish quality differences.

[0055] Finally, the information difference coefficients of each quality indicator participating in the current level aggregation are normalized to obtain the data distribution weights. : ; in, Indicates the first The data distribution weights of each quality indicator This indicates the number of quality metrics participating in the current level of aggregation. This indicates the quality index number that participated in the current level aggregation.

[0056] Based on the above data distribution weights The system can combine static benchmark weights. and dynamic adjustment coefficient Calculate the dynamic normalized weights Specifically, it can be expressed as: ; in, Indicates the first Dynamic normalized weights of each quality indicator Indicates the first The static benchmark weights of each quality indicator, Indicates the first The data distribution weights of each quality indicator Indicates the first Dynamic adjustment coefficients for each quality indicator.

[0057] Finally, the static baseline weights With dynamic normalized weights The final weights are obtained by fusion. : ; in, and Let represent the fusion coefficient, and satisfy: ; After fusion, the system assigns final weights to each quality indicator. Perform normalization verification again to obtain the dynamic weight vector for the current evaluation period. The dynamic weight vector can be represented as: ; Through the above processing, the indicator weights can be subject to the joint constraints of the indicator system's benchmark importance, the entropy weight calculation results corresponding to the quality standard distribution, and the business scenario status, thus matching the dynamic weight calculation process in enterprise-level data asset quality assessment.

[0058] In one embodiment, static benchmark weights can be determined using AHP (Adaptive Hierarchical Method). AHP employs a 1-9 scale, where 1 represents equal importance, 2, 4, 6, and 8 represent the median values ​​between adjacent judgments, 3, 5, 7, and 9 represent the increasing importance of each subsequent indicator relative to the previous one, and the reciprocal represents the relative importance of the next indicator relative to the previous one. The system calculates the eigenvector using the geometric mean and normalizes it. The system is effective when the consistency ratio (CR) of the judgment matrix is ​​less than 0.10; if the CR does not meet the requirement, the system returns to the kernel for re-scoring.

[0059] The primary indicator judgment matrix and benchmark weights are shown in Table 3.

[0060] Table 3

[0061] In the technical quality dimension, the local weights within the dimensions of completeness, accuracy, consistency, timeliness, uniqueness, and effectiveness can be 0.2019, 0.2019, 0.2019, 0.1799, 0.1010, and 0.1133, respectively. In the business quality dimension, the local weights within the dimensions of business effectiveness, business relevance, and business coverage can be 0.5000, 0.2500, and 0.2500, respectively. In the asset quality dimension, the local weights within the dimensions of asset value level, reuse frequency, lineage integrity, and lifecycle health can be 0.3333, 0.1667, 0.3333, and 0.1667, respectively. In the security and compliance dimension, the local weights within the dimensions of compliance coverage, sensitive data identification rate, permission compliance, security level matching degree, and audit traceability can be 0.2250, 0.2250, 0.2250, 0.1959, and 0.1292, respectively.

[0062] The global static benchmark weights of each secondary indicator can be obtained by multiplying the primary benchmark weights by the secondary local weights within the dimension. As shown in Table 4.

[0063] Table 4

[0064] In one embodiment, the dynamic factor does not directly replace the static baseline weight, but instead generates an indicator-level adjustment coefficient. The coefficient for an indicator that does not match the mapping rule is set to 1.00; when the same asset matches multiple business scenario tags, the scenario coefficient for the same indicator is set to the maximum value to avoid abnormal weight inflation caused by the superposition of multiple tags.

[0065] Business scenario tags can be configured according to Table 5.

[0066] Table 5

[0067] Asset importance can be determined comprehensively based on asset catalog tier labels, regulatory impact, production impact, business revenue impact, and the number of downstream dependencies. The key indicator set K may include completeness, accuracy, timeliness, business effectiveness, lineage completeness, compliance coverage, and authority compliance; the related indicator set A may include consistency, effectiveness, business relevance, business coverage, asset value level, security level matching degree, and audit traceability. Asset importance coefficients can be configured according to Table 6.

[0068] Table 6

[0069] Usage frequency can be statistically analyzed using the number of asset calls (C_30) over the past 30 days. Calls include data service APIs, SQL queries, BI report references, downstream scheduling reads, subscription pushes, etc. The usage frequency coefficient can be configured according to Table 7.

[0070] Table 7

[0071] Historical issue occurrence rates can be calculated based on the past 90 days. Quality issues include rule validation failures, governance work orders, audit findings, business complaints, and failed retests, and are mapped to corresponding secondary indicators based on issue attribution. Historical issue coefficients can be configured according to Table 8.

[0072] Table 8

[0073] For each secondary indicator i, the dynamic adjustment coefficient C_i can be calculated first: ; in, Indicates the business scenario coefficient. This represents the asset importance coefficient. This indicates the use of frequency coefficients. Indicates the coefficient of historical issues. This represents the cutoff function, which takes the value 0.70 when the value is below 0.70 and 1.50 when the value is above 1.50.

[0074] Then, calculate the dynamic normalized weights based on the static baseline weights, data distribution weights, and dynamic adjustment coefficients. : ; in, This represents the dynamic normalized weight of the i-th quality indicator. This represents the static benchmark weight of the i-th quality indicator. This represents the data distribution weight of the i-th quality indicator. represents the dynamic adjustment coefficient of the i-th quality indicator, and m represents the number of quality indicators participating in the current level aggregation.

[0075] Finally, the static baseline weights With dynamic normalized weights The final weights are obtained by fusion. : ; Where α and β represent fusion coefficients and satisfy: ; After fusion, the system assigns final weights to each quality indicator. Perform normalization verification again to obtain the dynamic weight vector for the current evaluation period. The dynamic weight vector can be represented as: ; Where m represents the number of quality metrics participating in the current level aggregation.

[0076] In one embodiment, α=0.70 and β=0.30 are used by default, that is, the static benchmark weight accounts for 70% and the dynamic normalized weight accounts for 30%. The AHP static benchmark weight is used to ensure the long-term stability of the indicator system, and the dynamic factor is used to reflect the business scenario, asset status and problem trend.

[0077] In scenarios where dynamic factor data is incomplete or exhibits significant fluctuations, β can also be automatically adjusted according to rules. For example, when dynamic data is complete and there are medium- to high-frequency quality issues, the dynamic proportion can be increased; when dynamic data is incomplete, the dynamic proportion can be decreased to ensure the stability of the weighting model.

[0078] For example, an asset is marked as a transaction-related, L1 core asset, and has been called 350 times in the past 30 days. Its accuracy metric has a 6% problem occurrence rate in the past 90 days. The static baseline weight of the accuracy metric is 0.0897; the corresponding business scenario coefficient is 1.20, asset importance coefficient is 1.25, usage frequency coefficient is 1.05, and historical problem coefficient is 1.20. , If the data distribution weight of this indicator If the value is 1.10, then the weighting factor for this indicator before normalization is: After the system completes entropy weight calculation, dynamic adjustment coefficient calculation, and normalization processing for each indicator, it obtains the dynamically normalized weight. Press again and The final weight is formed by fusion. .

[0079] S500, based on the quality standard score, the dynamic weight vector, and the hierarchical object mapping data, aggregate step by step in the hierarchical order of fields, tables, datasets to asset catalog to obtain the asset catalog quality score and quality level corresponding to the target data asset.

[0080] Specifically, the system can first calculate the field-level quality score based on the quality standard score and dynamic weight corresponding to the field-level object; then, based on the attribution relationship between the field and the table, aggregate the field-level quality score into a table-level quality score according to the field importance; then, based on the attribution relationship between the table and the dataset, aggregate the table-level quality score into a dataset-level quality score according to the table importance; finally, based on the attribution relationship between the dataset and the asset catalog, aggregate the dataset-level quality score according to the dataset importance, and combine it with the asset value coefficient to obtain the asset catalog quality score.

[0081] The importance of a field can be determined based on its business role, constraint attributes, sensitivity attributes, and lineage influence within the data asset. For example, field importance can be configured as shown in Table 9 below: Table 9

[0082] When aggregating field-level quality scores into table-level quality scores, the system can perform weighted aggregation based on the aggregation weight coefficient corresponding to the field's importance. If a field meets the criteria for multiple field importance levels simultaneously, its aggregation weight coefficient can be determined according to the higher importance level.

[0083] A single-level quality score can be expressed as: ; Where G represents the level base score, This represents the quality standard score of the i-th indicator. This represents the dynamic weight of the i-th indicator.

[0084] After introducing the asset value coefficient, the final score of the tier can be expressed as: ; in, Indicates the final score for the level. This represents the asset value coefficient. The coefficient can range from 0.6 to 1.2, with higher values ​​resulting in larger coefficients. In actual calculations, the system can truncate the corrected score to the [0, 100] interval.

[0085] Aggregation from fields to tables can be performed based on field importance. Field importance can be determined based on whether a field is a primary key, required field, business-critical field, regulatory field, sensitive field, or downstream dependent field. Aggregation from tables to datasets can be performed based on table importance. Table importance can be determined by asset level, downstream dependencies, usage frequency, data size and update frequency, and sensitive regulatory attributes. The calculation formula can be: =0.30×Asset Grade Score +0.25×Downstream Dependence Score +0.20×Usage Frequency Score +0.15×Data Scale and Update Score +0.10×Sensitive Regulatory Score.

[0086] The importance level and aggregation weight coefficient can be configured according to Table 10.

[0087] Table 10

[0088] The importance of a dataset can be determined by business domain importance, core process coverage, cross-domain reuse, regulatory involvement, and contribution to asset valuation. The calculation formula can be: =0.35 × Business Domain Importance Score + 0.25 × Core Process Coverage Score + 0.20 × Cross-Domain Reuse Score + 0.10 × Regulatory Participation Score + 0.10 × Asset Valuation Contribution Score.

[0089] The importance level of the dataset and the aggregation weight coefficient can be configured according to Table 11.

[0090] Table 11

[0091] The asset catalog quality score can be calculated based on the score of the dataset to which it belongs, the importance of the dataset, and the asset value coefficient. After obtaining the asset catalog quality score, the system can determine the quality level according to preset grading thresholds. For example, 90 to 100 points corresponds to Grade A, 80 to 89 points corresponds to Grade B, 70 to 79 points corresponds to Grade C, 60 to 69 points corresponds to Grade D, and 0 to 59 points corresponds to Grade E.

[0092] In one possible implementation, the system can also adjust the quality level based on blocking rules. For example, if the core field integrity score is lower than the blocking threshold, the sensitive data identification rate is lower than the mandatory threshold, there are serious violations of permission compliance, or the lineage integrity cannot support regulatory traceability, the system can restrict the highest quality level of the asset catalog or mark the corresponding asset catalog as a governance blocking status.

[0093] S600: Based on the asset catalog quality score, the quality level, and the low-score indicators and lineage influence range determined during the hierarchical aggregation process, a governance task is generated. After the governance task is completed, a retest is triggered, and the retest results are written back as the dynamic weight calculation input for the next evaluation cycle.

[0094] Specifically, the hierarchical aggregation module can output four-level hierarchical scores, quality levels, a list of low-scoring indicators, anomaly records, evidence samples, a set of affected objects, and the scope of influence based on lineage. After receiving the above output, the governance feedback module can match the corresponding governance task template according to the indicator type of the low-scoring indicators, determine the upstream responsible objects, downstream affected objects, and related asset objects corresponding to the low-scoring indicators according to the scope of influence based on lineage, and determine the governance priority and governance time limit according to the quality level, asset importance, and the existence status of governance blocking indicators.

[0095] In this embodiment, the low-score indicator refers to a quality indicator whose quality standard score is lower than a preset low-score threshold or whose original quality indicator value does not reach the corresponding lower limit of qualification during the hierarchical aggregation process. The preset low-score threshold can be configured according to the indicator type, asset level, and business domain. For example, indicators with a quality standard score lower than 60 or lower than 70 can be identified as low-score indicators.

[0096] For the Each quality indicator, if its quality standard is... satisfy: ; Then the first Several quality indicators were identified as low-scoring indicators. Among them, Indicates the first The quality standard score for each quality indicator This indicates a preset low score threshold.

[0097] For mandatory indicators, even if their quality standard score is not lower than the preset low score threshold, if their original quality indicator value is lower than the blocking threshold, they can still be identified as low-score indicators or governance blocking indicators. Specifically, if the first... One quality indicator is mandatory, and its original quality indicator value satisfy: ; Then the first Several quality indicators were identified as governance and prevention indicators. Among them, Indicates the first The original quality index values ​​of each quality indicator. Indicates the first The blocking threshold corresponding to each quality indicator.

[0098] The term "lineage influence range" refers to the set of objects that extend upstream and downstream along the lineage influence relationships in the hierarchical object mapping data, starting from the field, table, dataset, or asset catalog corresponding to the low-scoring indicator. The lineage influence range may include at least one of the following: upstream source field, upstream source table, upstream processing task, downstream consumption table, downstream dataset, data service interface, report, indicator, scheduling task, and associated asset catalog.

[0099] In one embodiment, the governance task template can be configured according to Table 12.

[0100] Table 12

[0101]

[0102]

[0103] After the governance task is completed, the system can recalculate the retest quality score of the target data asset using the indicator definitions, indicator thresholds, data source mappings, and dynamic weight vectors corresponding to the current evaluation period. Using the same weight version and threshold configuration during the retest ensures the comparability of scores before and after governance.

[0104] The system can determine the repair rate and recurrence risk based on the quality score before treatment and the quality score after retesting. The repair rate can be determined based on factors such as the improvement of low-scoring indicators, the proportion of problematic samples eliminated, and whether the retest is passed; the recurrence risk can be determined based on factors such as the number of historical recurrences, the number of consecutive retest failures, and the recurrence of abnormalities in the same indicator.

[0105] In one embodiment, the location and update rules for governance feedback data entering the dynamic weight model can be shown in Table 13.

[0106] Table 13

[0107] Therefore, the retest results are not only used to determine whether a single governance task has been completed, but can also be written back to the input of historical problem occurrence rate, asset quality label, asset value coefficient, governance effect sample library and dynamic weight calculation for the next evaluation cycle, thus forming a closed loop of "evaluation - governance - retest - feedback - re-evaluation".

[0108] In one implementation of computational execution optimization, the enterprise-level data asset quality assessment and optimization system can perform multi-source quality evidence collection, quality standard score calculation, and hierarchical aggregation using distributed computing resources. Distributed computing resources may include a distributed task scheduling framework, a batch processing engine, a streaming engine, a metadata indexing service, a quality rule execution cluster, and a caching service.

[0109] When changes to metadata, data partitioning, quality rules, lineage, or permission configuration are detected, the system can determine the set of hierarchical objects to be recalculated based on hierarchical object mapping data and lineage influence relationships, and perform incremental evaluation calculations on the set of hierarchical objects to be recalculated.

[0110] Incremental calculation can include the following steps: First, record the event object, event type, version number, and water level; second, generate a set of objects to be recalculated based on the lineage influence diagram between fields, tables, tasks, datasets, and asset catalogs; third, update the basic statistics only for affected indicators; fourth, recalculate according to the order of indicator standardization, dynamic weighting, hierarchical aggregation, and correction; fifth, compare the incremental score with the previous baseline score, and if the deviation exceeds a preset threshold, trigger a sampling review or a full recalculation.

[0111] For metrics that cannot be accurately maintained through incremental statistics, such as complex repetition rates, cross-table consistency, and business closed-loop rules, the system can adopt a strategy of "incremental estimation + periodic full calibration". Incremental estimation is performed daily using affected partitions or affected tasks, and full baseline calibration is performed weekly or monthly on key assets to prevent long-term statistical drift.

[0112] In one sampling optimization implementation, for petabyte-scale or cross-domain massive data assets, the system can combine full scanning, incremental calculation, and intelligent sampling strategies to perform evaluation. For metadata, lineage, permissions, and audit-related indicators, full metadata or full logs can be used first; for indicators with high computational costs, such as data value distribution, duplication rate, and cross-table consistency, stratified sampling, adaptive expansion, and high-risk fallback full sampling strategies can be adopted.

[0113] Stratified sampling can be stratified according to business domain, asset level, partition time, data type, field importance, etc.; adaptive expansion can increase the sample size when the sampling results fluctuate greatly, low-scoring indicators are close to the blocking threshold, or the historical problem occurrence rate is high; high-risk fallback full verification can trigger full verification on core assets, regulatory-reported assets, or assets that have failed continuous retesting.

[0114] In a multi-tenant implementation, the system can maintain differentiated thresholds, quality rules, weight versions, and assessment result versions by tenant, business domain, asset type, and indicator type. Configuration changes can retain the effective date, approver, and historical versions to ensure that assessment results for different assessment periods are traceable and recalculated.

[0115] Figure 3 This application illustrates an enterprise-level data asset quality assessment and optimization system provided in an embodiment. For example... Figure 3As shown, the system may include a configuration acquisition unit, an evidence processing unit, a standardization processing unit, a weight calculation unit, a hierarchical aggregation unit, and a governance feedback unit.

[0116] The configuration acquisition unit acquires the indicator configuration data, hierarchical object mapping data, and scene status data corresponding to the target data asset. The evidence processing unit collects multi-source quality evidence corresponding to the target data asset based on data source mapping, and verifies the multi-source quality evidence based on quality rules to obtain the original quality indicator values ​​corresponding to the indicator definitions. The standardization processing unit associates the original quality indicator values ​​with the corresponding fields, tables, datasets, or asset catalog hierarchical objects based on the indicator definitions and hierarchical object mapping data, and standardizes the original quality indicator values ​​according to indicator thresholds to obtain the quality standard score. The weight calculation unit determines the dynamic weight vector corresponding to each indicator within the current evaluation period based on the indicator configuration data and scene status data. The hierarchical aggregation unit aggregates the quality standard score, dynamic weight vector, and hierarchical object mapping data level by level, according to the hierarchical order from fields, tables, datasets to the asset catalog, to obtain the asset catalog quality score and quality level corresponding to the target data asset. The governance feedback unit is used to generate governance tasks based on the asset catalog quality score, quality level, and low-scoring indicators and lineage influence range determined during the hierarchical aggregation process. After the governance task is completed, a retest is triggered, and the retest results are written back as the dynamic weight calculation input for the next evaluation cycle.

[0117] In one embodiment of an electronic device, the electronic device may include a processor, a memory, and a communication interface. The memory stores a computer program, and when the processor executes the computer program, it implements the enterprise-level data asset quality assessment and optimization method described in any of the above embodiments. The communication interface can be used to interact with metadata acquisition components, data profiling components, business rules components, lineage analysis components, security audit components, and governance task components.

[0118] In one embodiment of a computer-readable storage medium, a computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the enterprise-level data asset quality assessment and optimization method described in any of the above embodiments. The computer-readable storage medium may include a read-only memory, random access memory, a disk, an optical disk, a solid-state drive, flash memory, or other media capable of storing a computer program.

Claims

1. A method for assessing and optimizing the quality of enterprise-level data assets, characterized in that, An enterprise-level data asset quality assessment and optimization system is applied, wherein the system resides on a digital asset management platform and is communicatively connected to metadata acquisition components, data profiling components, business rule components, lineage analysis components, security audit components, and governance task components. The method includes: Obtain the indicator configuration data, hierarchical object mapping data, and scene status data corresponding to the target data asset. The indicator configuration data includes indicator definitions, indicator thresholds, data source mappings, and quality rules. The hierarchical object mapping data is used to characterize the attribution and lineage relationships between fields, tables, datasets, and asset catalogs. According to the data source mapping, collect multi-source quality evidence corresponding to the target data asset, and verify the multi-source quality evidence based on the quality rules to obtain the original quality indicator value corresponding to the indicator definition. The multi-source quality evidence includes at least one of metadata, data analysis results, business rule verification results, lineage analysis results, and security audit results. Based on the indicator definition and the hierarchical object mapping data, the original quality indicator value is associated with the corresponding field, table, dataset or asset catalog hierarchical object, and the original quality indicator value is standardized and converted according to the indicator threshold to obtain the quality standard score; Based on the indicator configuration data and the scenario status data, a dynamic weight vector corresponding to each indicator in the current evaluation period is determined. The scenario status data is used to represent at least one of the following: business scenario, asset importance, usage frequency, and historical problem occurrence rate. Based on the quality standard score, the dynamic weight vector, and the hierarchical object mapping data, the data is aggregated level by level in the order of fields, tables, datasets to asset catalogs to obtain the asset catalog quality score and quality level corresponding to the target data asset. Based on the asset catalog quality score, the quality level, and the low-score indicators and lineage influence range determined during the hierarchical aggregation process, a governance task is generated. Upon completion of the governance task, a retest is triggered, and the retest results are written back as the dynamic weight calculation input for the next evaluation cycle.

2. The method according to claim 1, characterized in that, Based on the data source mapping, multi-source quality evidence corresponding to the target data asset is collected, and the multi-source quality evidence is verified based on the quality rules to obtain the original quality indicator value corresponding to the indicator definition, including: Based on the indicator number and applicable level in the indicator definition, determine the data source, quality rules and minimum calculation unit corresponding to each indicator; Based on the data source, corresponding quality evidence is obtained in parallel from the metadata collection component, the data analysis component, the business rules component, the lineage analysis component, and the security audit component; The obtained quality evidence is matched with the corresponding quality rules according to the indicator number, and the rule verification result is associated with the corresponding field, table, dataset or asset catalog level object to obtain the original quality indicator value.

3. The method according to claim 1 or 2, characterized in that, Based on the indicator configuration data and the scenario status data, determine the dynamic weight vector corresponding to each indicator within the current evaluation period, including: Determine the static benchmark weights corresponding to each indicator based on the aforementioned indicator definitions; The data distribution weights are determined based on the quality standard distribution of each indicator within the current evaluation period. The dynamic adjustment coefficient is determined based on at least one of the following: business scenario, asset importance, usage frequency, and historical problem occurrence rate. The static baseline weights, the data distribution weights, and the dynamic adjustment coefficients are fused and normalized to obtain the dynamic weight vector.

4. The method according to claim 1, characterized in that, Based on the indicator definition and the hierarchical object mapping data, the original quality indicator value is associated with the corresponding field, table, dataset, or asset catalog hierarchical object, and the original quality indicator value is standardized and converted according to the indicator threshold to obtain a quality standard score, including: If the data source corresponding to the original quality indicator value is unavailable, the indicator denominator is zero, or the fields required for indicator calculation are missing, the corresponding indicator will be marked as an uncalcifiable indicator. If the incalculable indicator is a mandatory item, the incalculable indicator will be identified as a governance blocking indicator. If the incalculable indicator is not a mandatory item, the incalculable indicator will be removed from the current level of aggregation calculation, and a corresponding data supplementation task or collection and repair task will be generated.

5. The method according to claim 1, characterized in that, Based on the quality standard score, the dynamic weight vector, and the hierarchical object mapping data, aggregation is performed level by level according to the hierarchical order of fields, tables, datasets to asset catalogs to obtain the asset catalog quality score and quality level corresponding to the target data asset, including: Calculate the field-level quality score based on the quality standard score and dynamic weight corresponding to the field-level object; Based on the attribution relationship between fields and tables in the hierarchical object mapping data, the field-level quality scores are aggregated into table-level quality scores according to field importance. Based on the attribution relationship between tables and datasets in the hierarchical object mapping data, the table-level quality scores are aggregated into dataset-level quality scores according to table importance. Based on the attribution relationship between datasets and asset catalogs in the hierarchical object mapping data, the dataset-level quality scores are aggregated according to dataset importance and then corrected with asset value coefficients to obtain the asset catalog quality score. The quality level is determined based on the correspondence between the asset catalog quality score and the preset grading threshold.

6. The method according to claim 1, characterized in that, The enterprise-level data asset quality assessment and optimization system uses distributed computing resources to perform the collection of multi-source quality evidence, the calculation of quality standard scores, and the hierarchical aggregation. In the event of changes in metadata, data partitioning, quality rules, lineage, or permission configuration, the system determines the set of hierarchical objects to be recalculated based on the hierarchical object mapping data and the lineage influence relationship, and performs incremental assessment calculations on the set of hierarchical objects to be recalculated.

7. The method according to claim 1, characterized in that, Upon completion of the governance task, a retest is triggered, and the retest results are written back as input for the dynamic weight calculation of the next evaluation cycle, including: After the governance task is completed, the retest quality score of the target data asset is recalculated using the indicator definitions, indicator thresholds, data source mapping, and dynamic weight vector corresponding to the current evaluation cycle. The repair rate and recurrence risk are determined based on the pre-treatment quality score and the retest quality score. The repair rate and the recurrence risk are written back to the historical problem incidence rate, asset quality label and governance effectiveness sample library, and the dynamic adjustment coefficient for the next evaluation cycle is updated based on the write-back results.

8. The method according to claim 1, characterized in that, Based on the asset catalog quality score, the quality level, and the low-score indicators and lineage influence range determined during the hierarchical aggregation process, a governance task is generated, including: Match the corresponding governance task template according to the indicator type of the low-scoring indicator; Based on the scope of influence of blood ties, determine the upstream responsible parties, downstream affected parties, and related asset parties corresponding to the low-scoring indicators; The governance priority and timeframe are determined based on the quality level, the asset importance, and the presence status of the governance disruption indicators. The governance task is generated based on the governance task template, the upstream responsible object, the downstream affected object, the related asset object, the governance priority, and the governance time limit.

9. The method according to claim 1, characterized in that, The target data assets include at least one of the following: business database tables, data warehouse tables, lake warehouse tables, thematic datasets, indicator datasets, data service interfaces, report data, scheduling task data, and asset catalog data; the multi-source quality evidence includes at least one of the following: table field metadata, field statistical analysis results, business rule assertion results, cross-table consistency verification results, scheduling logs, field-level lineage resolution results, permission authorization records, access audit records, and governance work order records.

10. An enterprise-level data asset quality assessment and optimization system, characterized in that, include: The configuration acquisition unit is used to acquire indicator configuration data, hierarchical object mapping data and scene status data corresponding to the target data asset. The indicator configuration data includes indicator definition, indicator threshold, data source mapping and quality rules. The hierarchical object mapping data is used to characterize the affiliation relationship and lineage influence relationship between fields, tables, datasets and asset catalogs. An evidence processing unit is used to collect multi-source quality evidence corresponding to the target data asset according to the data source mapping, and to verify the multi-source quality evidence based on the quality rules to obtain the original quality indicator value corresponding to the indicator definition. The multi-source quality evidence includes at least one of metadata, data analysis results, business rule verification results, lineage analysis results, and security audit results. The standardization processing unit is used to associate the original quality indicator value with the corresponding field, table, dataset or asset catalog hierarchical object according to the indicator definition and the hierarchical object mapping data, and to standardize and convert the original quality indicator value according to the indicator threshold to obtain the quality standard score. The weight calculation unit is used to determine the dynamic weight vector corresponding to each indicator in the current evaluation period based on the indicator configuration data and the scenario status data. The scenario status data is used to represent at least one of business scenario, asset importance, usage frequency and historical problem occurrence rate. The hierarchical aggregation unit is used to aggregate data level by level based on the quality standard score, the dynamic weight vector, and the hierarchical object mapping data, in the hierarchical order of fields, tables, datasets to asset catalog, to obtain the asset catalog quality score and quality level corresponding to the target data asset. The governance feedback unit is used to generate governance tasks based on the asset catalog quality score, the quality level, and the low-score indicators and lineage influence range determined during the hierarchical aggregation process. After the governance task is completed, a retest is triggered, and the retest results are written back as the dynamic weight calculation input for the next evaluation cycle.

Images