Data Asset Quality Scoring Method and System Based on Multidimensional Features

Abstract

The present application relates to the technical field of data processing, and discloses a method and system for data asset quality scoring based on multi-dimensional features. The method includes: extracting dynamic and static features based on data asset lifecycle rules, and obtaining a feature mapping matrix through dimensionality reduction and verification; calculating the quantization of feature correlation strength through bidirectional coupling and reconstructing to obtain a fused feature chain; adopting a hybrid metric mechanism for feature quantization to obtain a quality evaluation value; standardizing and calibrating the score to obtain a benchmark quality score; analyzing the quality change law to predict the trend and obtain an evolution report; generating a hierarchical progressive improvement strategy through traceability analysis. The present application realizes the accurate evaluation and continuous improvement of data asset quality through the fusion of dynamic and static features, a multi-level metric system, and a progressive optimization strategy, and particularly solves the technical problem of insufficient feature interaction analysis in the process of data quality evaluation.

Description

Technical Field

[0001] This application relates to the field of data processing, and in particular, to a method and system for scoring the quality of data assets based on multi-dimensional features. Background Art

[0002] With the deepening of the digital transformation of enterprises, data assets have become one of the core competitiveness of enterprises. The existing data asset quality assessment methods mainly evaluate from static dimensions such as the integrity, accuracy, and consistency of the data itself, and comprehensively measure by combining dynamic features such as the timeliness and availability of the data. These methods set scoring criteria, use calculation methods such as weighted average to obtain data quality scores, and perform data governance and optimization based on the quality scores. In practical applications, enterprises deploy data quality monitoring tools to regularly scan and evaluate data assets, discover data quality problems and rectify them, and establish a basic data quality management system.

[0003] However, the existing data asset quality assessment methods have obvious deficiencies: First, the quality assessment index system is too static and isolated, and fails to fully reflect the dynamic and correlation features of the data during the business process; Second, the scoring method is too simple and mechanical, usually using a linear weighting method to calculate the quality score, and unable to accurately describe the complex interaction relationships between different dimensional features; Third, the diagnosis of quality problems lacks systematicness and foresight, and often can only find surface problems and it is difficult to trace the root cause, resulting in governance measures being difficult to solve problems fundamentally; Finally, the quality improvement suggestions lack pertinence and operability, and do not consider the actual constraints and resource limitations of the enterprise, making it difficult for improvement measures to be effectively implemented. Summary of the Invention

[0004] This application provides a method and system for scoring the quality of data assets based on multi-dimensional features, which realizes the accurate assessment and continuous improvement of data asset quality through the fusion of dynamic and static features, a multi-level measurement system, and a progressive optimization strategy, and particularly solves the technical problem of insufficient feature interaction analysis in the data quality assessment process.

[0005] First aspect, the present application provides a method for scoring the quality of data assets based on multi-dimensional features. The method for scoring the quality of data assets based on multi-dimensional features includes: extracting static and dynamic features of data assets according to the collection rules of the life cycle characteristics of data assets, where the static features include integrity features, consistency features, and accuracy features, and the dynamic features include timeliness features, utilization features, and coverage features. After feature dimensionality reduction and effectiveness verification, a dual-state asset feature mapping matrix is obtained; according to the dual-state asset feature mapping matrix, the positive and negative correlation strengths between features are quantified through a two-way coupling degree calculation method, and strongly correlated features are adaptively grouped and reconstructed to obtain a multi-dimensional fusion feature chain; based on the multi-dimensional fusion feature chain, feature quantification is performed through a hierarchical hybrid quality measurement mechanism, where a deterministic quantification method is used for static features, a probabilistic quantification method is used for dynamic features, and adaptive weight assignment is performed on the quantification results to obtain a hierarchical quality evaluation value; according to the hierarchical quality evaluation value, the score is standardized through a multi-level quality benchmark mapping method, and the score deviation is automatically calibrated in combination with historical score data to obtain a dynamic benchmark quality score; according to the dynamic benchmark quality score, the quality change law is decomposed through a multi-scale time series analysis method, and the future quality trend is evaluated in combination with a multi-scenario quality prediction model to obtain a hierarchical quality evolution report; based on the hierarchical quality evolution report, the problem is located and analyzed through a quality problem traceability system, and an optimization plan is generated in combination with multi-dimensional constraint conditions to obtain a hierarchical progressive quality improvement strategy.

[0006] Second aspect, the present application provides a system for scoring the quality of data assets based on multi-dimensional features. The system for scoring the quality of data assets based on multi-dimensional features includes:

[0007] An extraction module, configured to extract static and dynamic features of data assets according to the collection rules of the life cycle characteristics of data assets, where the static features include integrity features, consistency features, and accuracy features, and the dynamic features include timeliness features, utilization features, and coverage features. After feature dimensionality reduction and effectiveness verification, a dual-state asset feature mapping matrix is obtained;

[0008] A quantification module, configured to quantify the positive and negative correlation strengths between features through a two-way coupling degree calculation method according to the dual-state asset feature mapping matrix, and adaptively group and reconstruct strongly correlated features to obtain a multi-dimensional fusion feature chain;

[0009] An assignment module, configured to perform feature quantification through a hierarchical hybrid quality measurement mechanism based on the multi-dimensional fusion feature chain, where a deterministic quantification method is used for static features, a probabilistic quantification method is used for dynamic features, and adaptive weight assignment is performed on the quantification results to obtain a hierarchical quality evaluation value;

[0010] A calibration module, which is used to standardize the score through a multi-level quality benchmark mapping method according to the hierarchical quality evaluation value, and automatically calibrate the score deviation by combining historical score data to obtain a dynamic benchmark quality score;

[0011] A decomposition module, which is used to decompose the quality change law through a multi-scale time series analysis method according to the dynamic benchmark quality score, and evaluate the future quality trend by combining a multi-scenario quality prediction model to obtain a hierarchical quality evolution report;

[0012] A positioning module, which is used to locate and analyze the problem through a quality problem traceability system according to the hierarchical quality evolution report, and generate an optimization plan by combining multi-dimensional constraint conditions to obtain a hierarchical progressive quality improvement strategy.

[0013] In the technical solution provided by this application, the multi-dimensional feature-based data asset quality scoring method comprehensively depicts the quality of data assets through dynamic and static feature extraction. Among them, static features reflect the inherent attributes of data assets, including integrity, consistency, and accuracy features, and dynamic features reflect the operating state of data assets, including timeliness, utilization, and coverage features. The combination of the two-state features provides a more complete feature basis for data quality evaluation. The positive and negative correlation strengths between features are quantified through bidirectional coupling degree calculation, revealing the promotion and restriction relationships between features. The multi-dimensional fusion feature chain formed by the adaptive grouping and reconstruction of strongly correlated features accurately reflects the hierarchical structure and transmission law between features. The hierarchical hybrid quality measurement mechanism adopts different quantization methods for static and dynamic features. The deterministic quantization method ensures the stability of static feature evaluation, the probabilistic quantization method reflects the time-varying characteristics of dynamic features, and the adaptive weight allocation balances the importance of different features. The multi-level quality benchmark mapping realizes the standardization processing of scores, and the automatic calibration combined with historical score data eliminates the subjectivity and volatility of scores. The multi-scale time series analysis method decomposes the quality change law, and combines the multi-scenario quality prediction model to achieve an accurate evaluation of the future quality trend. The quality problem traceability system finds the root cause of the problem through positioning analysis, and the hierarchical progressive quality improvement strategy generated by combining multi-dimensional constraint conditions ensures the feasibility and effectiveness of improvement measures. This solution makes full use of the advantages of machine learning algorithms in feature extraction, correlation analysis, trend prediction, etc., and realizes the intelligence and precision of data asset quality evaluation. Description of the Drawings

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0015] Figure 1 Schematic diagram of an embodiment of the data asset quality scoring method based on multi-dimensional features in the embodiments of the present application;

[0016] Figure 2 Schematic diagram of an embodiment of the data asset quality scoring system based on multi-dimensional features in the embodiments of the present application. Detailed implementation manners

[0017] The embodiments of the present application provide a data asset quality scoring method and system based on multi-dimensional features. Terms such as "first", "second", "third", "fourth", etc. (if any) in the specification, claims and above-mentioned drawings of the present application are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments described here can be implemented in an order other than those illustrated or described here. In addition, the terms "comprising" or "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0018] For ease of understanding, the specific processes of the embodiments of the present application are described below. Please refer to Figure 1 An embodiment of the data asset quality scoring method based on multi-dimensional features in the embodiments of the present application includes:

[0019] Step S101: Extract static and dynamic features of the data asset according to the collection rules of the life cycle characteristics of the data asset, where the static features include integrity features, consistency features, and accuracy features, and the dynamic features include timeliness features, utilization features, and coverage features. After feature dimensionality reduction and effectiveness verification, a dual-state asset feature mapping matrix is obtained;

[0020] Step S102: According to the dual-state asset feature mapping matrix, quantify the positive and negative association strengths between features through a two-way coupling degree calculation method, and perform adaptive grouping and reconstruction on strongly associated features to obtain a multi-dimensional fusion feature chain;

[0021] Step S103: Based on the multi-dimensional fusion feature chain, perform feature quantization through a hierarchical hybrid quality measurement mechanism, where a deterministic quantization method is used for static features, a probabilistic quantization method is used for dynamic features, and an adaptive weight assignment is performed on the quantization results to obtain a hierarchical quality evaluation value;

[0022] Step S104: Standardize the score through a multi-level quality benchmark mapping method according to the hierarchical quality evaluation value, and automatically calibrate the score deviation in combination with historical score data to obtain the dynamic benchmark quality score;

[0023] Step S105: Decompose the quality change law through a multi-scale time series analysis method according to the dynamic benchmark quality score, and evaluate the future quality trend in combination with a multi-scenario quality prediction model to obtain a hierarchical quality evolution report;

[0024] Step S106: Locate and analyze the problem through a quality problem traceability system based on the hierarchical quality evolution report, and generate an optimization plan in combination with multi-dimensional constraint conditions to obtain a hierarchical progressive quality improvement strategy.

[0025] It can be understood that the execution subject of this application can be a data asset quality scoring system based on multi-dimensional features, or a terminal or a server. Specifically, it is not limited here. This application embodiment is described by taking the server as the execution subject as an example.

[0026] Specifically, dynamic and static features of the data asset are extracted. The dynamic features reflect the change characteristics of the data asset during use. The timeliness feature reflects the data update frequency and time interval. The utilization feature reflects the frequency of data being called and used. The coverage feature represents the coverage range of the data for business scenarios. The static features, on the other hand, reflect the quality attributes of the data itself. The integrity feature measures the non-null rate of data fields. The consistency feature checks the consistency degree of cross-table data. The accuracy feature verifies the correctness of data values. For example, when extracting features from enterprise customer data, the static features can check the integrity of customer basic information, the consistency of customer numbers, and the accuracy of contact information; the dynamic features focus on the update timeliness of customer information, the frequency of data being called by business departments, and the coverage of data in scenarios such as customer acquisition, marketing, and service.

[0027] Construct a dual-state asset feature mapping matrix through feature correlation analysis, which records the association relationships between features. Calculate the two-way coupling degree of the matrix to analyze the positive promotion and negative restriction relationships between features. Positive associations indicate the existence of synergy effects between features. For example, the higher the data update frequency, the better the accuracy usually is; negative associations indicate the existence of restriction effects between features. For example, excessive pursuit of data integrity may affect data timeliness. Group and reconstruct strongly associated features to form a multi-dimensional fusion feature chain, which reflects the hierarchical structure and transmission relationships of data quality features. In the feature quantification stage, adopt a hierarchical hybrid quality measurement mechanism. For static features, use a deterministic quantification method to directly calculate the feature values. For example, the completeness can be calculated by the proportion of non-empty values; for dynamic features, use a probabilistic quantification method, considering the time decay factor. For example, the timeliness can be calculated by combining the data update time interval and the business timeliness requirements. Through adaptive weight assignment, adjust the feature weights according to different business scenarios to generate hierarchical quality assessment values.

[0028] Standardize the hierarchical quality assessment values and establish a multi-level quality benchmark mapping relationship. Combine historical scoring data for deviation calibration to eliminate the subjectivity and inconsistency of scoring and obtain objective dynamic benchmark quality scores. This score reflects the quality level of data assets at different times and in different scenarios. Through multi-scale time series analysis, decompose the trend of the dynamic benchmark quality scores to identify seasonal fluctuations, periodic changes, and long-term trends. Combine multi-scenario prediction to analyze the quality change trends in different business scenarios and form a hierarchical quality evolution report. This report describes the historical performance, current state, and future trends of data quality.

[0029] Based on the hierarchical quality evolution report, trace the problems back to their sources, and locate the key factors affecting data quality through the quality problem traceability system. Combine business constraint conditions to generate a hierarchical progressive improvement strategy to handle problems at different levels. Taking enterprise customer data as an example, when it is found that the data quality shows periodic decline, through traceability analysis, it may be found that it is due to the temporary quality fluctuations caused by monthly batch data updates, and accordingly, improvement measures for source governance and process control are formulated. Through the dynamic and static combined feature system, multi-level measurement mechanism, and progressive optimization strategy, a systematic solution is provided for data asset quality management. In practical applications, by extracting data features, quantifying quality scores, analyzing evolution trends, and formulating improvement strategies, it helps enterprises continuously enhance the value of data assets. The processing process and logical relationships between data are clear, and each step is interconnected, forming a closed-loop for data asset quality assessment and optimization.

[0030] In the embodiments of the present application, the data asset quality scoring method based on multi-dimensional features comprehensively characterizes the quality of data assets through the extraction of dynamic and static features. Among them, static features reflect the inherent attributes of data assets, including integrity, consistency, and accuracy features, while dynamic features reflect the operating status of data assets, including timeliness, utilization, and coverage features. The combination of the two types of features provides a more complete feature basis for data quality assessment. By calculating the two-way coupling degree, the positive and negative correlation strengths between features are quantified, revealing the promotion and restriction relationships between features. The multi-dimensional fusion feature chain formed by the adaptive grouping and reconstruction of strongly correlated features accurately reflects the hierarchical structure and transmission law between features. The hierarchical hybrid quality measurement mechanism adopts different quantification methods for static and dynamic features. The deterministic quantification method ensures the stability of static feature evaluation, the probabilistic quantification method reflects the time-varying characteristics of dynamic features, and the adaptive weight allocation balances the importance of different features. The multi-level quality benchmark mapping realizes the standardized processing of scoring, and the automatic calibration combined with historical scoring data eliminates the subjectivity and volatility of scoring. The multi-scale time series analysis method decomposes the quality change law, and combines with the multi-scenario quality prediction model to achieve an accurate assessment of future quality trends. The quality problem tracing system locates and analyzes the root cause of the problem, and the hierarchical progressive quality improvement strategy generated in combination with multi-dimensional constraint conditions ensures the feasibility and effectiveness of improvement measures. This solution makes full use of the advantages of machine learning algorithms in feature extraction, correlation analysis, trend prediction, etc., and realizes the intelligence and precision of data asset quality assessment. In a specific embodiment, the process of executing step S101 may specifically include the following steps:

[0031] (1) Scan the operation logs of the data asset through a set time window to obtain the original dynamic feature data stream;

[0032] (2) Classify the original dynamic feature data stream by time segments to obtain the time-segmented feature set;

[0033] (3) Extract the timeliness feature, utilization feature, and coverage feature from the time-segmented feature set to obtain the dynamic feature vector;

[0034] (4) Extract the integrity feature, consistency feature, and accuracy feature from the data items of the data asset to obtain the static feature vector;

[0035] (5) Combine and map the dynamic feature vector and the static feature vector to obtain the feature combination set;

[0036] (6) Screen the features in the feature combination set through feature correlation tests, eliminate redundant features, and obtain the feature candidate set;

[0037] (7) Perform dimensionality reduction on the candidate feature set and verify its effectiveness through cross-validation to obtain the dual-state asset feature mapping matrix.

[0038] Specifically, process the operation logs of data assets through a time window scanning mechanism. The size of the time window is set according to the business characteristics of the data assets. For example, it is set to 24 hours for transaction data assets and 7 days for customer information data assets. The operation logs record behaviors such as data creation, update, and deletion. The scanning process converts these discrete operation records into a continuous feature data stream, forming the original dynamic feature data stream. Segment and classify the original dynamic feature data stream according to the time granularity, which can be units such as hours, days, weeks, etc. Through time segmentation, aggregate the feature data within different time windows, and calculate the feature statistical values for each time period, such as indicators like data update frequency and access volume, so as to obtain the time-segmented feature set. This set contains the dynamic behavior characteristics of the data assets in different time periods. Extract three types of core dynamic features from the time-segmented feature set: The timeliness feature reflects the update cycle and real-time nature of the data, and is obtained by calculating the mean and variance of the data update time intervals; The utilization feature represents the usage situation of the data, and is obtained by counting the data access frequency and the number of accessing users; The coverage feature describes the application scope of the data in business scenarios, and is obtained by calculating the usage ratio of the data in different business scenarios. These three types of features form a dynamic feature vector, which is expressed as: ; where is the timeliness feature value, is the utilization feature value, is the coverage feature value. For static feature extraction, directly calculate three indicators of integrity, consistency, and accuracy from the data items of the data assets: The integrity is calculated by the proportion of non-empty values, the consistency is obtained through cross-table data consistency checks, and the accuracy is obtained through data value validity verification. The static feature vector is expressed as: ; where is the integrity feature value, is the consistency feature value, is the accuracy feature value.

[0039] Combine and map the dynamic feature vector and the static feature vector to construct a feature combination set. The mapping process considers the mutual influence between features, such as the influence of timeliness on accuracy, the influence of integrity on coverage, etc. The feature combination set is expressed as: ; Screen the features in the feature combination set through feature correlation tests, and calculate the correlation coefficient matrix between features: ; where represents the correlation coefficient between the i-th feature and the j-th feature. Set the correlation threshold , when When it indicates that there is a strong correlation between features, feature screening is required. After screening, a feature candidate set is obtained, and further dimensionality reduction is performed through principal component analysis to obtain the dimensionality-reduced feature matrix. Finally, the effectiveness of the features is verified through the cross-validation method to form a dual-state asset feature mapping matrix. Taking the customer data assets of an enterprise as an example, the time window is set to 7 days, and operations such as customer information updates and queries are recorded. Through time-period feature extraction, it is found that during weekdays, the data update frequency is relatively high, and data access is concentrated from 9 am to 5 pm, mainly involving two business scenarios: sales and customer service. The calculated timeliness feature value is 0.85 (good update timeliness), the utilization feature value is 0.72 (moderate usage frequency), and the coverage feature value is 0.68 (wide business coverage). In terms of static features, the integrity is 0.93 (basic information is complete), the consistency is 0.88 (cross-departmental data is basically consistent), and the accuracy is 0.91 (high data quality). After feature correlation analysis and dimensionality reduction processing, a mapping matrix reflecting the quality characteristics of data assets is finally obtained.

[0040] In a specific embodiment, the process of executing step S102 may specifically include the following steps:

[0041] (1) Divide the business feature layer of the dual-state asset feature mapping matrix according to the data asset business scenario to obtain a feature scenario matrix;

[0042] (2) Perform forward transfer association calculation on the features in the feature scenario matrix to obtain a forward business impact matrix;

[0043] (3) Perform reverse traceability association calculation on the features in the feature scenario matrix to obtain a reverse business dependency matrix;

[0044] (4) Perform bidirectional mapping on the forward business impact matrix and the reverse business dependency matrix to obtain a business coupling association matrix;

[0045] (5) Group the features according to the business association threshold based on the business coupling association matrix to obtain a strongly associated business feature group;

[0046] (6) Reconstruct and merge the features within the strongly associated business feature group to obtain a fused business feature unit, and chain the fused business feature units to obtain a multi-dimensional fused feature chain.

[0047] Specifically, in the dual-state asset feature mapping matrix, first perform hierarchical processing on the data assets according to the business scenario to form a feature scenario matrix. For the data asset management system of an enterprise, the business scenarios include three main levels: data collection, data processing, and data application. The feature scenario matrix is expressed as: ; where It represents the weight value of the i-th feature in the j-th business scenario, where n is the number of features and k is the number of business scenarios. Each weight value is calculated based on the importance of the feature in the corresponding business scenario. Forward transmission correlation analysis is performed on the feature scenario matrix to calculate the influence strength between features:

[0048] ; where represents the positive influence degree of the i-th feature on the j-th feature. At the same time, reverse traceability correlation calculation is performed:

[0049] ; where represents the dependence degree of the j-th feature on the i-th feature.

[0050] The forward influence matrix and the reverse dependence matrix are bidirectionally mapped to obtain the business coupling correlation matrix: ; where and are weight coefficients, satisfying , which is used to balance the importance of forward influence and reverse dependence.

[0051] Based on the business coupling correlation matrix, a threshold is set to group the features: when > , feature i and feature j are classified into the same strong correlation group. The features within the strong correlation feature group are reconstructed to obtain the fused feature unit: ; where is the k-th fused feature unit, is the number of features within this group, is the weight of the i-th feature, is the original feature vector.

[0052] Taking the quality assessment of enterprise customer data as an example, at the data collection level, the integrity and accuracy features are focused on. At the data processing level, the consistency and timeliness features are concerned. At the data application level, the utilization and coverage features are emphasized. By calculating the positive influence and reverse dependence relationships between features, it is found that the timeliness feature has a strong positive influence on the accuracy feature, while the coverage feature depends on the integrity feature. Through feature fusion and reconstruction, a multi-dimensional fused feature chain reflecting the quality characteristics of data assets is formed, providing a basis for subsequent quality assessment.

[0053] In a specific embodiment, the process of executing step S103 may specifically include the following steps:

[0054] (1) Extract the business transfer path from the multi-dimensional fused feature chain to obtain the feature metric sequence;

[0055] (2) Establish a quality standard baseline for the static features in the feature metric sequence to obtain the deterministic metric value;

[0056] (3) Construct a time decay function for the dynamic features in the feature metric sequence to obtain a probabilistic metric value;

[0057] (4) Combine the deterministic metric value and the probabilistic metric value in a hierarchical manner to obtain a hybrid metric matrix;

[0058] (5) Assign weights to the hybrid metric matrix according to the depth of business impact to obtain a weighted metric result;

[0059] (6) Perform hierarchical normalization on the weighted metric result to obtain a hierarchical quality evaluation value.

[0060] Specifically, extract the business flow path from the multi-dimensional fusion feature chain. For example, during the process where a company's customer data is collected from the CRM system, processed through the data warehouse, and finally flows to the marketing and service systems, the business flow sequence can be expressed as: ; where represents the data collection node, represents the data cleaning node, represents the data storage node, represents the data usage node. The flow relationship between these nodes is described by the flow matrix: ; for example, t_{12} represents that the flow probability from the collection node to the cleaning node is 0.95, indicating that 95% of the collected data will enter the cleaning link; t_{23} is 0.9, indicating that 90% of the cleaned data will be stored; t_{34} is 0.85, indicating that 85% of the stored data will be used by the business.

[0061] For the quality standard baseline of static features, taking customer basic information as an example: ; where represents the integrity feature (such as the filling rate of name, phone number, and address), represents the consistency feature (such as the consistency of customer numbers across systems), represents the accuracy feature (such as the correctness of phone number format). The weight coefficients , , are set according to business importance, such as 0.4, 0.3, and 0.3 respectively.

[0062] For dynamic features, consider the time decay of data. For example, the update frequency of customer contact information will affect data quality:

[0063] ;

[0064] where is the initial contact information feature value, The decay rate is 0.1 per month, and t is the number of months. This indicates that the credibility of unupdated contact information will be lost by 10% per month.

[0065] The hybrid metric matrix combines static and dynamic features: ; At the business level, the marketing department attaches importance to the timeliness of customer contact information, while the finance department is more concerned about the accuracy of customer information. Therefore, the weight matrix reflects the needs of different departments:

[0066] ;

[0067] The final quality assessment results are normalized to obtain the scores for each dimension. For example, after evaluating a certain customer's data: the timeliness of contact information is 0.85 (full score 1.0), indicating that the contact information is relatively new; the accuracy of information is 0.92, indicating high-quality basic data; the cross-departmental consistency is 0.78, indicating that data integration needs to be optimized.

[0068] In a specific embodiment, the process of executing step S104 may specifically include the following steps:

[0069] (1) Hierarchically decompose the hierarchical quality assessment values according to the business quality level to obtain a quality level matrix;

[0070] (2) Establish a quality benchmark interval for the quality level matrix to obtain a benchmark mapping table;

[0071] (3) Standardize the scores of the quality level matrix through the benchmark mapping table to obtain a set of standard scores;

[0072] (4) Extract historical scoring data from the data asset quality records to obtain a historical distribution sequence;

[0073] (5) Perform deviation correction on the set of standard scores and the historical distribution sequence to obtain a calibrated score group; perform dynamic benchmark update on the calibrated score group to obtain dynamic benchmark quality scores.

[0074] Specifically, hierarchically decompose the hierarchical quality assessment values according to the business quality level. The decomposition process is classified according to the application characteristics of data assets in different business scenarios, such as the application characteristics of enterprise customer data assets in business scenarios such as sales, service, and marketing, to form a quality level matrix. This matrix reflects the quality performance of data assets at each business level. For example, in the sales scenario, the customer basic information focuses on information accuracy; in the service scenario, it focuses on information integrity; in the marketing scenario, it focuses on information timeliness.

[0075] When establishing a quality benchmark interval for the quality level matrix, it is necessary to set grading criteria. Taking enterprise customer data as an example, the integrity of basic information is divided into three levels: unqualified, up to standard, and excellent. Thresholds are set based on industry standards and business requirements. For customer contact information, the unqualified standard is that the filling rate is lower than 60%, the up-to-standard standard is that the filling rate is between 60% and 90%, and the excellent standard is that the filling rate is higher than 90%. These benchmark standards are recorded in the benchmark mapping table.

[0076] The quality level matrix is processed for score standardization through the benchmark mapping table. The standardization process maps the original scores to a unified interval, making the quality scores in different dimensions and different business scenarios comparable. For example, if the filling rate of customer contact information is 85%, it is mapped to a standard score of 0.8; if the completeness rate of customer transaction records is 95%, it is mapped to a standard score of 0.9. This standardization process eliminates the dimensional differences between different quality indicators. Historical score data is extracted from the data asset quality records to form a historical distribution sequence. This sequence contains the situation of the data asset quality changing over time, reflecting the stability and change trend of the data quality. For example, the quality score records of a certain enterprise's customer data in the past six months show the fluctuation range and overall level of the data quality.

[0077] The standard score set and the historical distribution sequence are subjected to deviation correction to obtain a calibrated score group. The correction process takes into account the relationship between the current score and the historical performance to avoid abnormal fluctuations in the score. Taking the quality of customer data as an example, if the current score is significantly higher than the historical average level, it needs to be corrected through historical data to ensure the stability and credibility of the score. Finally, dynamic benchmark updates are performed on the calibrated score group to generate dynamic benchmark quality scores. The update process adopts a sliding window mechanism, balancing the influence of historical performance and the latest score. For example, the quality score of enterprise customer data is updated monthly, and the new score is weighted and averaged with the historical benchmark to obtain the updated dynamic benchmark quality score. This dynamic update mechanism not only maintains the stability of the score but also reflects the real-time changes in data quality.

[0078] Taking the quality assessment of member data of a certain retail enterprise as an example, the quality assessment of member data is divided into the basic information layer, transaction record layer, and behavior data layer through hierarchical decomposition. At the basic information level, the filling rates of information such as contact information, age, and occupation are processed through standardization to obtain basic information quality scores; at the transaction record level, the accuracy and completeness of information such as consumption amount, time, and commodities are standardized to obtain transaction data quality scores; at the behavior data level, the timeliness and relevance of information such as member points, coupon usage, and shopping preferences are standardized to obtain behavior data quality scores. These scores are compared and calibrated with the historical score data of the past six months, and finally, dynamic benchmark quality scores reflecting the overall quality level of member data are formed.

[0079] In a specific embodiment, the process of executing step S105 may specifically include the following steps:

[0080] (1) Segment the dynamic reference quality score according to a time window to obtain a time series score sequence;

[0081] (2) Decompose the time series score sequence according to the business cycle to obtain a group of periodic change characteristics;

[0082] (3) Extract the quality trend from the group of periodic change characteristics to obtain a quality fluctuation matrix;

[0083] (4) Classify the quality fluctuation matrix according to the business scenario to obtain a scenario-quality mapping table;

[0084] (5) Construct a quality trend sequence based on the scenario-quality mapping table to obtain a trend prediction result;

[0085] (6) Classify the trend prediction result according to the business level to obtain a hierarchical quality evolution report.

[0086] Specifically, perform time window segmentation processing on the dynamic reference quality score. The selection of the time window is based on the business characteristics of the data asset. For enterprise customer data assets, time windows with three granularities of daily, weekly, and monthly are used for segmentation. The segmentation process cuts the continuous quality scores into discrete time series, and each series corresponds to the quality change situation within a time window. Decomposing the time series score sequence according to the business cycle is a key step in understanding the law of data quality change. The business cycle includes a daily cycle, a weekly cycle, and a monthly cycle, reflecting the periodic change characteristics of data quality with business activities. For the membership data of a retail enterprise, the data quality differences between weekdays and holidays, the data update rules at the beginning and end of the month, and the data quality fluctuation characteristics during holiday promotions are identified through business cycle decomposition.

[0087] The process of extracting the quality trend from the group of periodic change characteristics focuses on the change direction and amplitude of data quality. Trend extraction eliminates the influence of short-term fluctuations through a moving average method and highlights the long-term change trend. For enterprise data assets, short-term trends, medium-term trends, and long-term trends are respectively extracted to form a multi-dimensional quality fluctuation matrix. This matrix records the quality change situations at different time scales. The quality fluctuation matrix is classified according to the business scenario to form a scenario-quality mapping table. Business scenario classification is based on the usage scenarios of data assets, such as customer acquisition scenarios, service delivery scenarios, risk control scenarios, etc. The quality fluctuation characteristics under each scenario are mapped to the corresponding quality indicators to establish the correspondence between the scenario and the quality change.

[0088] When constructing the quality trend sequence based on the scenario quality mapping table, the relevance between different scenarios needs to be considered. The construction process of the trend sequence includes in-scenario trend analysis and inter-scenario trend correlation analysis to identify the conduction path and influence scope of data quality changes. Through time series analysis methods and combined with business knowledge, the quality change trend in the future period is predicted. The trend prediction results are classified according to the business levels to generate a hierarchical quality evolution report. The business level division is based on the enterprise's organizational structure and business processes, including the basic data layer, business processing layer, and decision support layer. The quality evolution characteristics of each layer are clearly recorded and displayed to provide guidance for data governance work at different levels.

[0089] Taking the enterprise customer master data management as an example, the process of data quality evolution analysis is specifically described as follows: First, the customer data quality scores in the recent year are segmented by monthly time windows to obtain 12 months of time series data. These data reflect the monthly changes in the quality of customer information. Through business cycle decomposition, it is found that at the beginning of each month, the quality scores fluctuate due to batch data updates, remain relatively stable in the middle of the month, and data is centrally updated at the end of the month due to business indicator assessments. Quality trend extraction shows that in the short term, there are small fluctuations in data quality, showing a trend of steady improvement in the medium term, and the quality level steadily increases in the long term. In different business scenarios, the marketing scenario has high requirements for data timeliness, the customer service scenario attaches importance to data accuracy, and the risk management scenario focuses on data integrity. These scenario characteristics are mapped to the corresponding quality indicators. By analyzing historical trends and scenario characteristics, the future data quality change trend is predicted, and a quality evolution report is generated according to the three levels of data collection, processing, and application to guide data governance work at each level.

[0090] In a specific embodiment, the process of executing step S106 may specifically include the following steps:

[0091] (1) Extract the quality change points in the hierarchical quality evolution report to obtain the problem trigger sequence;

[0092] (2) Trace back the problem trigger sequence according to the business impact path to obtain the problem traceability map;

[0093] (3) Extract the characteristics of the key nodes in the problem traceability map to obtain the problem feature set;

[0094] (4) Match the problem feature set with the business constraint rules to obtain the constraint condition matrix;

[0095] (5) Perform hierarchical optimization decomposition on the constraint condition matrix to obtain a group of hierarchical optimization schemes; Organize the group of hierarchical optimization schemes according to the progressive relationship to obtain a hierarchical progressive quality improvement strategy.

[0096] Specifically, quality change points are extracted from the hierarchical quality evolution report. A quality change point refers to the time node when the data quality indicator changes significantly and is identified by setting a change threshold. The specific operation is to compare the change range of the quality score with the preset threshold, and when the change range exceeds the threshold, it is marked as a quality change point. These change points are arranged in chronological order to form a problem trigger sequence. For example, in the quality management of enterprise customer data, when the integrity indicator of customer information suddenly drops on a certain day, or the timeliness indicator of data update is lower than the baseline for three consecutive days, they will be marked as quality change points. The backtracking of the business impact path of the problem trigger sequence is a process of analyzing from the surface to the essence. By tracing the data flow process, identifying the propagation chain of quality problems, and establishing a problem traceability map. In the links of data collection, cleaning, storage, and application, relevant information on data processing is recorded in each link. By analyzing this information, the occurrence, development, and propagation process of data quality problems can be restored. For enterprise customer data, through log records, it can be traced which link the data problem occurred in, whether it is a manual input error or a system synchronization exception, and how this problem affects the downstream business system.

[0097] The extraction of key nodes in the problem traceability map is an important link in discovering the essence of the problem. Key nodes include the problem occurrence point, diffusion point, and impact point. By analyzing the characteristics of these nodes, characteristics such as the type, severity, and impact scope of the problem are extracted to form a problem feature set. In customer data quality problems, key node analysis can discover whether the problem stems from inconsistent data collection specifications, data synchronization delays across systems, or data inconsistencies caused by business rule changes. The problem feature set is matched and analyzed with business constraint rules to generate a constraint condition matrix. Business constraint rules come from enterprise business specifications, industry standards, and legal requirements, and define the various conditions that data quality should meet. The matching process corresponds problem features to corresponding constraint rules, identifies which rules are violated, and what improvement measures need to be taken. In customer data management, constraint rules include requirements for mandatory fields of customer information, requirements for data update timeliness, cross-system consistency requirements, etc.

[0098] The hierarchical optimization decomposition of the constraint condition matrix is the basis for formulating improvement plans. According to the urgency, influence scope, and difficulty of solving the problem, the constraint conditions are divided into different levels. For each level of constraint conditions, corresponding optimization plans are designed to form a group of hierarchical optimization plans. For example, for customer data quality problems, high-priority issues include the absence of key fields and inaccurate core business data, which need to be solved immediately; medium-priority issues such as untimely data updates and non-standard formats can be improved step by step; low-priority issues such as the improvement of non-key fields can be gradually solved through daily maintenance. Finally, the group of hierarchical optimization plans is organized according to the progressive relationship to form a hierarchical progressive quality improvement strategy. The progressive relationship takes into account the dependencies and implementation order between the plans to ensure that the improvement measures can be promoted in an orderly manner. For example, when dealing with the enterprise customer data quality problem, first solve the data collection standardization problem, then optimize the data synchronization mechanism, and finally improve the data monitoring system to form a complete strategy chain from source governance to process control and then to continuous improvement.

[0099] Taking the supplier master data governance of a manufacturing enterprise as an example, the quality evolution report shows that the integrity and accuracy of supplier information fluctuate. Through the extraction of change points, it is found that during the switching period between the old and new systems, the supplier data quality has decreased significantly. The business path backtracking shows that the problem starts from the data migration link during the procurement system upgrade. Due to the imperfect field mapping rules, some supplier information is lost or misaligned. The key node analysis finds that there is a lack of a quality verification mechanism in the data migration process, and the inconsistent data standards are the root cause. After matching with the business constraint rules, it is determined that the integrity requirements and field standardization requirements of data migration are violated. Accordingly, a hierarchical improvement plan is formulated: in the first stage, improve the data migration rules and supplement the lost data; in the second stage, unify the data standards and standardize the field definitions; in the third stage, establish a data quality monitoring mechanism to prevent similar problems from occurring again.

[0100] The above describes the data asset quality scoring method based on multi-dimensional features in the embodiments of the present application. Next, the data asset quality scoring system based on multi-dimensional features in the embodiments of the present application will be described. Please refer to Figure 2 , an embodiment of the data asset quality scoring system based on multi-dimensional features in the embodiments of the present application includes:

[0101] An extraction module, configured to extract dynamic and static features of a data asset according to the life cycle feature collection rules of the data asset, where the static features include integrity features, consistency features, and accuracy features, and the dynamic features include timeliness features, utilization features, and coverage features. After feature dimensionality reduction and validity verification, a dual-state asset feature mapping matrix is obtained;

[0102] A quantization module, which is used to quantify the positive and negative correlation strengths between features according to the binary asset feature mapping matrix through a two-way coupling degree calculation method, and perform adaptive grouping and reconstruction on strongly correlated features to obtain a multi-dimensional fusion feature chain; An allocation module, which is used to perform feature quantization according to the multi-dimensional fusion feature chain through a hierarchical hybrid quality measurement mechanism, where a deterministic quantization method is used for static features, a probabilistic quantization method is used for dynamic features, and an adaptive weight allocation is performed on the quantization results to obtain a hierarchical quality evaluation value;

[0103] A calibration module, which is used to standardize the score according to the hierarchical quality evaluation value through a multi-level quality benchmark mapping method, and automatically calibrate the score deviation in combination with historical score data to obtain a dynamic benchmark quality score;

[0104] A decomposition module, which is used to decompose the quality change law according to the dynamic benchmark quality score through a multi-scale time series analysis method, and evaluate the future quality trend in combination with a multi-scenario quality prediction model to obtain a hierarchical quality evolution report;

[0105] A positioning module, which is used to perform positioning analysis on problems according to the hierarchical quality evolution report through a quality problem traceability system, and generate an optimization plan in combination with multi-dimensional constraint conditions to obtain a hierarchical progressive quality improvement strategy.

[0106] Through the collaborative cooperation of the above-mentioned various components, the multi-dimensional feature-based data asset quality scoring method comprehensively characterizes the quality of data assets through the extraction of static and dynamic features. Among them, static features reflect the inherent attributes of data assets, including integrity, consistency, and accuracy features, while dynamic features reflect the operating status of data assets, including timeliness, utilization, and coverage features. The combination of these two types of features provides a more complete feature basis for data quality assessment. By calculating the two-way coupling degree, the positive and negative correlation strengths between features are quantified, revealing the promotion and restriction relationships between features. The multi-dimensional fusion feature chain formed by the adaptive grouping and reconstruction of strongly correlated features accurately reflects the hierarchical structure and transmission law between features. The hierarchical hybrid quality measurement mechanism adopts different quantification methods for static and dynamic features. The deterministic quantification method ensures the stability of static feature evaluation, the probabilistic quantification method reflects the time-varying characteristics of dynamic features, and the adaptive weight allocation balances the importance of different features. The multi-level quality benchmark mapping realizes the standardized processing of scoring, and the automatic calibration combined with historical scoring data eliminates the subjectivity and volatility of scoring. The multi-scale time series analysis method decomposes the quality change law, and combined with the multi-scenario quality prediction model, it realizes the accurate assessment of future quality trends. The quality problem tracing system finds the root cause of the problem through location analysis, and the hierarchical progressive quality improvement strategy generated in combination with multi-dimensional constraint conditions ensures the feasibility and effectiveness of improvement measures. This solution makes full use of the advantages of machine learning algorithms in feature extraction, correlation analysis, trend prediction, etc., and realizes the intelligence and accuracy of data asset quality assessment.

[0107] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the above-described systems, systems, and units can refer to the corresponding processes in the foregoing method embodiments, and will not be described herein again.

[0108] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data asset quality scoring method based on multi-dimensional features, characterized in that The data asset quality scoring method based on multi-dimensional features includes: According to the collection rules of the life cycle characteristics of data assets, extract static and dynamic characteristics of the data assets. The static characteristics include integrity characteristics, consistency characteristics, and accuracy characteristics, and the dynamic characteristics include timeliness characteristics, utilization characteristics, and coverage characteristics. After feature dimensionality reduction and validity verification, a dual-state asset feature mapping matrix is obtained; Divide the business feature layer of the dual-state asset feature mapping matrix according to the data asset business scenario to obtain a feature scenario matrix; perform forward transfer association calculation on the features in the feature scenario matrix to obtain a forward business impact matrix; perform reverse traceability association calculation on the features in the feature scenario matrix to obtain a reverse business dependence matrix; perform two-way mapping on the forward business impact matrix and the reverse business dependence matrix to obtain a business coupling association matrix; group the features according to the business association threshold based on the business coupling association matrix to obtain a strongly associated business feature group; perform business reconstruction and merging on the features within the strongly associated business feature group to obtain a fused business feature unit, and chain the fused business feature units to obtain a multi-dimensional fused feature chain; Extract the business transfer path from the multi-dimensional fused feature chain to obtain a feature metric sequence; Establish a quality standard baseline for the static characteristics in the feature metric sequence to obtain a deterministic metric value; construct a time decay function for the dynamic characteristics in the feature metric sequence to obtain a probabilistic metric value; perform hierarchical combination of the deterministic metric value and the probabilistic metric value to obtain a mixed metric matrix; perform weight assignment on the mixed metric matrix according to the business impact depth to obtain a weighted metric result; perform hierarchical normalization processing on the weighted metric result to obtain a hierarchical quality evaluation value; According to the hierarchical quality evaluation value, standardize the score through a multi-level quality benchmark mapping method, and automatically calibrate the score deviation in combination with historical score data to obtain a dynamic benchmark quality score; According to the dynamic benchmark quality score, decompose the quality change law through a multi-scale time series analysis method, and evaluate the future quality trend in combination with a multi-scenario quality prediction model to obtain a hierarchical quality evolution report; Based on the hierarchical quality evolution report, locate and analyze the problems through a quality problem traceability system, and generate an optimization plan in combination with multi-dimensional constraint conditions to obtain a hierarchical progressive quality improvement strategy.

2. The method for scoring the quality of data assets based on multi-dimensional features according to claim 1, wherein, The extraction of static and dynamic characteristics of data assets according to the collection rules of the life cycle characteristics of data assets, where the static characteristics include integrity characteristics, consistency characteristics, and accuracy characteristics, and the dynamic characteristics include timeliness characteristics, utilization characteristics, and coverage characteristics. After feature dimensionality reduction and validity verification, a dual-state asset feature mapping matrix is obtained, including: Scan the operation logs of data assets through a set time window to obtain an original dynamic feature data stream; Classify the original dynamic feature data stream by time segments to obtain a time-segmented feature set; Extract timeliness characteristics, utilization characteristics, and coverage characteristics from the time-segmented feature set to obtain a dynamic feature vector; Extract the integrity feature, consistency feature, and accuracy feature from the data items of the data asset to obtain a static feature vector; Combine and map the dynamic feature vector and the static feature vector to obtain a feature combination set; Screen the features in the feature combination set through feature correlation tests, eliminate redundant features, and obtain a feature candidate set; Perform dimensionality reduction on the feature candidate set and verify its effectiveness through the cross-validation method to obtain a dual-state asset feature mapping matrix.

3. The method for scoring the quality of data assets based on multi-dimensional features according to claim 1, wherein, The method for standardizing the score through a multi-level quality benchmark mapping method according to the hierarchical quality evaluation value and automatically calibrating the score deviation in combination with historical score data to obtain a dynamic benchmark quality score includes: Decompose the hierarchical quality evaluation value according to the business quality level to obtain a quality level matrix; Establish a quality benchmark interval for the quality level matrix to obtain a benchmark mapping table; Standardize the scores of the quality level matrix through the benchmark mapping table to obtain a set of standard scores; Extract historical score data from the data asset quality records to obtain a historical distribution sequence; Perform deviation correction on the set of standard scores and the historical distribution sequence to obtain a calibrated score group; perform dynamic benchmark update on the calibrated score group to obtain a dynamic benchmark quality score.

4. The method for scoring the quality of data assets based on multi-dimensional features according to claim 1, wherein The method for decomposing the quality change law through a multi-scale time series analysis method according to the dynamic benchmark quality score and evaluating the future quality trend in combination with a multi-scenario quality prediction model to obtain a hierarchical quality evolution report includes: Segment the dynamic benchmark quality score according to a time window to obtain a time series score sequence; Decompose the time series score sequence according to the business cycle to obtain a set of periodic change features; Extract the quality trend from the set of periodic change features to obtain a quality fluctuation matrix; Classify the business scenarios of the quality fluctuation matrix to obtain a scenario quality mapping table; Construct a quality trend sequence based on the scenario quality mapping table to obtain a trend prediction result; Classify the trend prediction result according to the business level to obtain a hierarchical quality evolution report.

5. The method for scoring the quality of data assets based on multi-dimensional features according to claim 1, wherein The method for positioning and analyzing problems through a quality problem traceability system according to the hierarchical quality evolution report and generating an optimization plan in combination with multi-dimensional constraint conditions to obtain a hierarchical progressive quality improvement strategy includes: Extract the quality change points in the hierarchical quality evolution report to obtain a problem trigger sequence; Trace back the problem trigger sequence according to the business impact path to obtain a problem traceability map; Extract the features of the key nodes in the problem traceability map to obtain a problem feature set; Match the problem feature set with the business constraint rules to obtain a constraint condition matrix; Perform hierarchical optimization decomposition on the constraint condition matrix to obtain a set of hierarchical optimization plans; organize the set of hierarchical optimization plans according to the progressive relationship to obtain a hierarchical progressive quality improvement strategy.

6. A data asset quality scoring system based on multi-dimensional features, which is used to implement the data asset quality scoring method based on multi-dimensional features as described in any one of claims 1-5, characterized in that, The data asset quality scoring system based on multi-dimensional features includes: An extraction module, which is used to extract static and dynamic features of data assets according to the collection rules of the life cycle characteristics of data assets. The static features include integrity features, consistency features, and accuracy features, and the dynamic features include timeliness features, utilization features, and coverage features. After feature dimension reduction and validity verification, a dual-state asset feature mapping matrix is obtained; A quantization module, which is used to divide the business feature layer of the dual-state asset feature mapping matrix according to the data asset business scenario to obtain a feature scenario matrix; perform forward transfer correlation calculation on the features in the feature scenario matrix to obtain a forward business impact matrix; perform reverse traceability correlation calculation on the features in the feature scenario matrix to obtain a reverse business dependency matrix; perform two-way mapping on the forward business impact matrix and the reverse business dependency matrix to obtain a business coupling correlation matrix; group the features according to the business correlation threshold based on the business coupling correlation matrix to obtain a strongly correlated business feature group; perform business reconstruction and merging on the features within the strongly correlated business feature group to obtain a fused business feature unit, and connect the fused business feature units in a business chain to obtain a multi-dimensional fused feature chain; An allocation module, which is used to extract the business transfer path from the multi-dimensional fused feature chain to obtain a feature metric sequence; establish a quality standard baseline for the static features in the feature metric sequence to obtain a deterministic metric value; construct a time decay function for the dynamic features in the feature metric sequence to obtain a probabilistic metric value; perform hierarchical combination on the deterministic metric value and the probabilistic metric value to obtain a mixed metric matrix; allocate weights to the mixed metric matrix according to the business impact depth to obtain a weighted metric result; perform hierarchical normalization processing on the weighted metric result to obtain a hierarchical quality evaluation value; A calibration module, which is used to standardize the score through a multi-level quality benchmark mapping method according to the hierarchical quality evaluation value, and automatically calibrate the score deviation in combination with historical score data to obtain a dynamic benchmark quality score; A decomposition module, which is used to decompose the quality change law through a multi-scale time series analysis method according to the dynamic benchmark quality score, and evaluate the future quality trend in combination with a multi-scenario quality prediction model to obtain a hierarchical quality evolution report; A positioning module, which is used to perform positioning analysis on the problem through a quality problem traceability system based on the hierarchical quality evolution report, and generate an optimization plan in combination with multi-dimensional constraint conditions to obtain a hierarchical progressive quality improvement strategy.

Images