Multi-dimensional evaluation method and system for data quality

Abstract

The application is suitable for the technical field of data quality evaluation, and provides a multi-dimension evaluation method and system for data quality. The application accesses multiple data sources of a data center, carries out stage data collection, standardization processing and correlation analysis, generates multiple dynamic quality benchmarks, carries out multi-dimension synchronous evaluation on the data to be evaluated, carries out correlation verification, after the verification is passed, carries out fusion analysis on the multi-dimension quality results, when the comprehensive quality level is lower than the standard threshold, carries out abnormal positioning analysis and targeted optimization processing. The application can carry out stage data collection, standardization processing and correlation analysis, construct a correlation data network, construct multiple data quality benchmarks, carry out dynamic correction on the benchmarks, and then carry out multi-dimension synchronous evaluation, correlation verification and abnormal optimization processing, so that the one-sided problem of data quality evaluation is avoided, the data quality evaluation result can truly reflect the overall quality level of the data, and the risk of misjudgment and missed judgment is reduced.

Description

Technical Field

[0001] This invention belongs to the field of data quality assessment technology, and in particular relates to a multi-dimensional assessment method and system for data quality. Background Technology

[0002] Data quality assessment is a systematic, quantitative analysis and comprehensive evaluation of the quality status of data throughout its entire lifecycle, including collection, transmission, storage, processing, sharing, and application. Its purpose is to objectively reflect whether the data meets the actual needs of business applications, management decisions, and intelligent analysis scenarios, provide a basis for data governance, realize real-time perception and early warning of data quality status, improve the credibility, reliability, and application value of data resources, and thus provide a high-quality data foundation for data centers, enterprise digital operations, and artificial intelligence applications.

[0003] In existing technologies, data quality assessment of data centers typically only targets one or a few indicators of data integrity, consistency, and accuracy for independent testing. These indicators are isolated from each other and lack correlation analysis, resulting in a one-sided data quality assessment. The data quality assessment results cannot truly reflect the overall quality level of the data and are prone to misjudgment and omission. Summary of the Invention

[0004] The purpose of this invention is to provide a multi-dimensional evaluation method and system for data quality, aiming to solve the technical problems existing in the prior art mentioned in the background.

[0005] The embodiments of the present invention are implemented as follows: A multi-dimensional data quality assessment method, specifically including the following steps: Access multiple data sources in the data center, determine the types of multiple source data, perform phased data aggregation, standardization processing and correlation analysis, and build a correlated data network; Based on the multiple source data types, the operation pattern of the associated data network is analyzed, multiple data quality benchmarks are constructed, and the benchmarks are dynamically corrected to generate multiple dynamic quality benchmarks. The data to be evaluated is acquired, and a multi-dimensional synchronous evaluation is performed on the data to be evaluated based on multiple dynamic quality benchmarks to obtain multi-dimensional quality results. The multidimensional quality results are correlated and verified. After the verification is passed, the multidimensional quality results are fused and analyzed to determine the comprehensive quality level and compare it with the preset standard threshold. When the overall quality level is lower than the standard threshold, anomaly location analysis is performed to determine the anomaly location results, and targeted optimization processing is carried out.

[0006] As a further limitation of the technical solution of this invention embodiment, the process of accessing multiple data sources in the data center, determining multiple source data types, performing phased data aggregation, standardization processing, and correlation analysis, and constructing a correlated data network specifically includes the following steps: Access to multiple data sources in the data center; Determine the multiple source data types corresponding to the multiple data sources; Planning data collection phase; According to the data aggregation stages, the data from multiple data sources is aggregated in stages to obtain a stage data set; The aforementioned stage data set is standardized to obtain a standard data set; The standard data set is correlated by time, business, user, and device to construct a correlated data network.

[0007] As a further limitation of the technical solution of this embodiment of the invention, the step of analyzing the operational rules of the associated data network according to multiple source data types, constructing multiple data quality benchmarks, and dynamically correcting the benchmarks to generate multiple dynamic quality benchmarks specifically includes the following steps: Based on the multiple source data types, the operation pattern of the associated data network is analyzed, and multiple types of pattern data are statistically analyzed; Based on the data of the aforementioned types, multiple quality reference intervals are generated; Multiple data quality benchmarks are constructed based on the aforementioned quality reference intervals; Get the current running status; Based on the current operating status, multiple data quality benchmarks are dynamically corrected to generate multiple dynamic quality benchmarks.

[0008] As a further limitation of the technical solution of this embodiment of the invention, the step of obtaining the data to be evaluated, and performing multi-dimensional synchronous evaluation on the data to be evaluated based on multiple dynamic quality benchmarks to obtain multi-dimensional quality results specifically includes the following steps: Obtain the data to be evaluated; Based on multiple dynamic quality benchmarks, the data integrity assessment is performed on the data to be evaluated to obtain the integrity assessment result. Based on multiple dynamic quality benchmarks, a data consistency assessment is performed on the data to be evaluated to obtain a consistency assessment result. Based on multiple dynamic quality benchmarks, the data to be evaluated is assessed for accuracy, and an accuracy assessment result is obtained. Based on multiple dynamic quality benchmarks, the timeliness of the data to be evaluated is assessed to obtain the timeliness assessment result. Based on multiple dynamic quality benchmarks, the data stability assessment of the data to be evaluated is performed to obtain the stability assessment results. By combining the completeness assessment results, the consistency assessment results, the accuracy assessment results, the timeliness assessment results, and the stability assessment results, a multidimensional quality result is generated.

[0009] As a further limitation of the technical solution of this embodiment of the invention, the step of performing correlation verification on the multidimensional quality results, and after the verification is passed, performing fusion analysis on the multidimensional quality results to determine the comprehensive quality level, and comparing it with a preset standard threshold specifically includes the following steps: The multidimensional quality results are subjected to correlation verification to determine whether the verification passes. After successful verification, the evaluation fusion weights are determined; According to the evaluation fusion weights, the multidimensional quality results are fused and analyzed to calculate the comprehensive quality credibility index. Based on the comprehensive quality reliability index, the corresponding comprehensive quality level is matched; The overall quality level is compared with a preset standard threshold.

[0010] As a further limitation of the technical solution of this invention, the step of performing anomaly localization analysis, determining the anomaly localization result, and performing targeted optimization processing when the overall quality level is lower than the standard threshold specifically includes the following steps: When the overall quality level is lower than the standard threshold, it is determined that there is a data anomaly; Anomaly localization analysis was performed on the multidimensional quality results to determine the anomaly localization results; Based on the anomaly localization results, formulate anomaly optimization strategies; Targeted optimization processes are performed according to the aforementioned anomaly optimization strategy.

[0011] A multi-dimensional data quality assessment system, comprising a data aggregation and processing module, a benchmark construction and correction module, a multi-dimensional data assessment module, a quality level analysis module, and an anomaly location and processing module, wherein: The data aggregation and processing module is used to access multiple data sources in the data center, determine the types of multiple source data, perform phased data aggregation, standardization processing and correlation analysis, and build a related data network. The benchmark construction and correction module is used to analyze the operating rules of the associated data network according to multiple source data types, construct multiple data quality benchmarks, and perform dynamic benchmark correction to generate multiple dynamic quality benchmarks. The data multidimensional evaluation module is used to acquire the data to be evaluated, and to perform multidimensional synchronous evaluation of the data to be evaluated based on multiple dynamic quality benchmarks to obtain multidimensional quality results. The quality level analysis module is used to perform correlation verification on the multidimensional quality results, and after the verification is passed, to perform fusion analysis on the multidimensional quality results to determine the comprehensive quality level and compare it with the preset standard threshold. The anomaly location processing module is used to perform anomaly location analysis, determine the anomaly location result, and perform targeted optimization processing when the overall quality level is lower than the standard threshold.

[0012] As a further limitation of the technical solution of this embodiment of the invention, the data collection and processing module specifically includes: Data source access unit, used to access multiple data sources in the data center; A type determination unit is used to determine multiple source data types corresponding to the multiple data sources; Phase planning unit, used for planning the data collection phase; The data aggregation unit is used to aggregate data from multiple data sources according to the data aggregation stage, and obtain a stage data set. A standardization processing unit is used to standardize the stage data set to obtain a standard data set. The data association unit is used to perform time association, business association, user association and device association on the standard data set to build an associated data network.

[0013] As a further limitation of the technical solution of this embodiment of the invention, the benchmark construction correction module specifically includes: The pattern analysis unit is used to perform operational pattern analysis on the associated data network according to multiple source data types, and to statistically analyze multiple types of pattern data; An interval generation unit is used to generate multiple quality reference intervals based on multiple types of regular data. A benchmark construction unit is used to construct multiple data quality benchmarks according to multiple quality reference intervals; The status acquisition unit is used to acquire the current running status; The dynamic correction unit is used to dynamically correct multiple data quality benchmarks based on the current operating state, thereby generating multiple dynamic quality benchmarks.

[0014] As a further limitation of the technical solution of this embodiment of the invention, the quality grade analysis module specifically includes: The correlation verification unit is used to perform correlation verification on the multidimensional quality results and determine whether the verification passes. The weight determination unit is used to determine the evaluation fusion weights after the verification is passed; The fusion analysis unit is used to perform fusion analysis on the multidimensional quality results according to the evaluation fusion weights and calculate the comprehensive quality credibility index. A grade matching unit is used to match the corresponding comprehensive quality grade based on the comprehensive quality reliability index. The threshold comparison unit is used to compare the overall quality level with a preset standard threshold.

[0015] Compared with the prior art, the beneficial effects of the present invention are: This invention, through accessing multiple data sources in a data center, performs phased data aggregation, standardization processing, and correlation analysis; generates multiple dynamic quality benchmarks; conducts multi-dimensional synchronous evaluation of the data to be evaluated; performs correlation verification; and after successful verification, integrates and analyzes the multi-dimensional quality results; and when the overall quality level is below the standard threshold, performs anomaly localization analysis and targeted optimization. It can perform phased data aggregation, standardization processing, and correlation analysis, construct a correlated data network, build multiple data quality benchmarks, dynamically correct the benchmarks, and then perform multi-dimensional synchronous evaluation, correlation verification, and anomaly optimization, thereby avoiding the one-sidedness of data quality evaluation, ensuring that the data quality evaluation results can truly reflect the overall quality level of the data, and reducing the risk of misjudgment and omission. Attached Figure Description

[0016] Figure 1 A flowchart of the multi-dimensional data quality assessment method provided in an embodiment of the present invention is shown; Figure 2 The application architecture diagram of the multi-dimensional data quality assessment system provided in the embodiment of the present invention is shown. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0018] Understandably, in existing technologies, data quality assessments for data centers typically only target one or a few indicators among data integrity, consistency, and accuracy for independent testing. These indicators are isolated from each other and lack correlation analysis, leading to a one-sided data quality assessment. The data quality assessment results cannot truly reflect the overall quality level of the data and are prone to misjudgment and omission.

[0019] To address the aforementioned issues, the multi-dimensional data quality assessment method and system disclosed in this invention involves accessing multiple data sources from a data center, determining multiple source data types, performing phased data aggregation, standardization processing, and correlation analysis to construct a correlated data network. Based on the multiple source data types, the system analyzes the operational patterns of the correlated data network, constructs multiple data quality benchmarks, and dynamically adjusts these benchmarks to generate multiple dynamic quality benchmarks. It then acquires the data to be assessed, performs multi-dimensional synchronous assessment based on these dynamic quality benchmarks, and obtains multi-dimensional quality results. The multi-dimensional quality results undergo correlation verification, and after successful verification, a fusion analysis is performed to determine the comprehensive quality level, which is then compared with a preset standard threshold. When the comprehensive quality level falls below the standard threshold, anomaly localization analysis is conducted to determine the anomaly location results, and targeted optimization processing is implemented. This approach enables phased data aggregation, standardization processing, and correlation analysis; the construction of a correlated data network; the establishment of multiple data quality benchmarks; dynamic benchmark adjustment; and multi-dimensional synchronous assessment, correlation verification, and anomaly optimization processing. This avoids the one-sidedness of data quality assessment, ensuring that the data quality assessment results accurately reflect the overall data quality level and reducing the risk of misjudgment or omission.

[0020] Specifically, Figure 1 A flowchart of a multi-dimensional data quality assessment method provided by an embodiment of the present invention is shown.

[0021] In a preferred embodiment of the present invention, the multi-dimensional data quality assessment method specifically includes the following steps: Step S101: Connect to multiple data sources in the data center, determine the types of multiple source data, perform phased data aggregation, standardization processing and correlation analysis, and build a related data network.

[0022] In this embodiment of the invention, multiple data sources such as business databases, log databases, monitoring databases, and operation and maintenance databases in the data center are accessed, and multiple source data types such as JSON, XML, CSV, time-series data, semi-structured data, and unstructured data are determined. Data aggregation stages are planned, and then stage data aggregation is performed on multiple data sources according to the data aggregation stages to obtain stage data sets. After that, the stage data sets are standardized by unifying fields, units, time, and encoding to obtain standard data sets. Then, time association, business association, user association, and device association are performed on the standard data sets to build an associated data network.

[0023] Specifically, in another preferred embodiment provided by the present invention, the process of accessing multiple data sources in the data center, determining multiple source data types, performing phased data aggregation, standardization processing, and correlation analysis, and constructing a correlated data network specifically includes the following steps: Access to multiple data sources in the data center; Determine the multiple source data types corresponding to the multiple data sources; Planning data collection phase; According to the data aggregation stages, the data from multiple data sources is aggregated in stages to obtain a stage data set; The aforementioned stage data set is standardized to obtain a standard data set; The standard data set is correlated by time, business, user, and device to construct a correlated data network.

[0024] Furthermore, the multi-dimensional data quality assessment method also includes the following steps: Step S102: According to the multiple source data types, perform operational pattern analysis on the associated data network, construct multiple data quality benchmarks, and dynamically correct the benchmarks to generate multiple dynamic quality benchmarks.

[0025] In this embodiment of the invention, the operation pattern analysis of the associated data network is performed according to multiple source data types, including update cycle, fluctuation range, data distribution, missing patterns and / or growth patterns. Multiple types of pattern data are statistically analyzed, and multiple quality reference intervals are generated based on these multiple types of pattern data. Then, data quality benchmarks corresponding to multiple source data types are constructed according to the multiple quality reference intervals, and the current operating status is obtained. Subsequently, the multiple data quality benchmarks are dynamically corrected according to the current operating status to generate multiple dynamic quality benchmarks.

[0026] It is understandable that both data quality benchmarks and dynamic quality benchmarks are composed of corresponding integrity benchmarks, consistency benchmarks, timeliness benchmarks, accuracy benchmarks, and stability benchmarks.

[0027] Understandably, dynamically adjusting multiple data quality benchmarks to match their current state is to ensure that the generated dynamic quality benchmarks are consistent with the current operating status. For example, if business grows by 20% for a consecutive month, the corresponding data scale standard needs to be increased.

[0028] Specifically, in another preferred embodiment provided by the present invention, the step of analyzing the operational patterns of the associated data network according to multiple source data types, constructing multiple data quality benchmarks, and dynamically correcting the benchmarks to generate multiple dynamic quality benchmarks specifically includes the following steps: Based on the multiple source data types, the operation pattern of the associated data network is analyzed, and multiple types of pattern data are statistically analyzed; Based on the data of the aforementioned types, multiple quality reference intervals are generated; Multiple data quality benchmarks are constructed based on the aforementioned quality reference intervals; Get the current running status; Based on the current operating status, multiple data quality benchmarks are dynamically corrected to generate multiple dynamic quality benchmarks.

[0029] Furthermore, the multi-dimensional data quality assessment method also includes the following steps: Step S103: Obtain the data to be evaluated, and perform multi-dimensional synchronous evaluation on the data to be evaluated based on multiple dynamic quality benchmarks to obtain multi-dimensional quality results.

[0030] In this embodiment of the invention, after acquiring the data to be evaluated, based on multiple dynamic quality benchmarks, data integrity assessment, data consistency assessment, data accuracy assessment, data timeliness assessment, and data stability assessment are performed on the data to be evaluated, respectively obtaining integrity assessment results, consistency assessment results, accuracy assessment results, timeliness assessment results, and stability assessment results. Then, the integrity assessment results, consistency assessment results, accuracy assessment results, timeliness assessment results, and stability assessment results are integrated to generate a multidimensional quality result. Specifically, data integrity assessment involves analyzing data missing rate, field null value rate, record continuity rate, etc.; data consistency assessment involves analyzing differences between databases, logs, caches, and backups; data accuracy assessment involves analyzing whether constraints are violated, whether outliers occur, and whether abnormal combinations occur; data timeliness assessment involves analyzing update time delay, synchronization delay, acquisition delay, and write delay; and data stability assessment involves analyzing volatility, abnormal growth rate, abnormal decline rate, and jitter frequency.

[0031] Specifically, in another preferred embodiment provided by the present invention, the step of acquiring the data to be evaluated and performing a multi-dimensional synchronous evaluation of the data to be evaluated based on multiple dynamic quality benchmarks to obtain a multi-dimensional quality result specifically includes the following steps: Obtain the data to be evaluated; Based on multiple dynamic quality benchmarks, the data integrity assessment is performed on the data to be evaluated to obtain the integrity assessment result. Based on multiple dynamic quality benchmarks, a data consistency assessment is performed on the data to be evaluated to obtain a consistency assessment result. Based on multiple dynamic quality benchmarks, the data to be evaluated is assessed for accuracy, and an accuracy assessment result is obtained. Based on multiple dynamic quality benchmarks, the timeliness of the data to be evaluated is assessed to obtain the timeliness assessment result. Based on multiple dynamic quality benchmarks, the data stability assessment of the data to be evaluated is performed to obtain the stability assessment results. By combining the completeness assessment results, the consistency assessment results, the accuracy assessment results, the timeliness assessment results, and the stability assessment results, a multidimensional quality result is generated.

[0032] Furthermore, the multi-dimensional data quality assessment method also includes the following steps: Step S104: Perform correlation verification on the multidimensional quality results, and after the verification is passed, perform fusion analysis on the multidimensional quality results to determine the comprehensive quality level and compare it with the preset standard threshold.

[0033] In this embodiment of the invention, the multidimensional quality results are correlated and verified to determine whether the verification is successful. If the verification is successful, the evaluation fusion weight is determined. Then, the multidimensional quality results are fused and analyzed according to the evaluation fusion weight to calculate the comprehensive quality credibility index. According to the comprehensive quality credibility index, the corresponding comprehensive quality level is matched from the preset quality grading table, and the comprehensive quality level is compared with the preset standard threshold.

[0034] Understandably, correlation verification is a process of analyzing the internal correlation of multidimensional quality results to determine whether they meet the corresponding conditions. For example, when the probability of synchronization failure is high, integrity and consistency will decrease simultaneously; when there is network latency, timeliness decreases, but accuracy remains normal.

[0035] Specifically, in another preferred embodiment provided by the present invention, the step of performing correlation verification on the multidimensional quality results, and after the verification is passed, performing fusion analysis on the multidimensional quality results to determine the comprehensive quality level, and comparing it with a preset standard threshold, specifically includes the following steps: The multidimensional quality results are subjected to correlation verification to determine whether the verification passes. After successful verification, the evaluation fusion weights are determined; According to the evaluation fusion weights, the multidimensional quality results are fused and analyzed to calculate the comprehensive quality credibility index. Based on the comprehensive quality reliability index, the corresponding comprehensive quality level is matched; The overall quality level is compared with a preset standard threshold.

[0036] Furthermore, the multi-dimensional data quality assessment method also includes the following steps: Step S105: When the overall quality level is lower than the standard threshold, perform anomaly location analysis, determine the anomaly location result, and perform targeted optimization processing.

[0037] In this embodiment of the invention, when the overall quality level is lower than the standard threshold, data anomalies are identified. At this point, anomaly localization analysis is performed on the multi-dimensional quality results, including tracing anomalies in the data source, synchronization link, database, acquisition interface, cache node, and computing node. The anomaly localization results are then determined, and an anomaly optimization strategy is formulated based on these results. Subsequently, targeted optimization processes are performed according to the anomaly optimization strategy. For example, if there is a synchronization delay, the synchronization frequency is increased; if data is missing, supplementary acquisition is initiated; if a field is abnormal, it is re-verified; if the cache is abnormal, the cache is refreshed.

[0038] Specifically, in another preferred embodiment provided by the present invention, the step of performing anomaly localization analysis, determining the anomaly localization result, and performing targeted optimization processing when the overall quality level is lower than the standard threshold specifically includes the following steps: When the overall quality level is lower than the standard threshold, it is determined that there is a data anomaly; Anomaly localization analysis was performed on the multidimensional quality results to determine the anomaly localization results; Based on the anomaly localization results, formulate anomaly optimization strategies; Targeted optimization processes are performed according to the aforementioned anomaly optimization strategy.

[0039] Furthermore, Figure 2 The application architecture diagram of the multi-dimensional data quality assessment system provided in the embodiment of the present invention is shown.

[0040] Specifically, in another preferred embodiment provided by the present invention, the multi-dimensional data quality assessment system includes: The data aggregation and processing module 101 is used to access multiple data sources in the data center, determine the types of multiple source data, perform phased data aggregation, standardization processing and correlation analysis, and build a related data network.

[0041] In this embodiment of the invention, the data aggregation and processing module 101 accesses multiple data sources such as the business database, log database, monitoring database, and operation and maintenance database of the data center, and determines multiple source data types such as JSON, XML, CSV, time-series data, semi-structured data, and unstructured data. It also plans data aggregation stages and then aggregates data from multiple data sources according to the data aggregation stages to obtain stage data sets. After that, it performs standardization processing on the stage data sets, such as field unification, unit unification, time unification, and encoding unification, to obtain standard data sets. Finally, it performs time association, business association, user association, and device association on the standard data sets to build an associated data network.

[0042] Specifically, in another preferred embodiment provided by the present invention, the data collection and processing module 101 specifically includes: Data source access unit, used to access multiple data sources in the data center; A type determination unit is used to determine multiple source data types corresponding to the multiple data sources; Phase planning unit, used for planning the data collection phase; The data aggregation unit is used to aggregate data from multiple data sources according to the data aggregation stage, and obtain a stage data set. A standardization processing unit is used to standardize the stage data set to obtain a standard data set. The data association unit is used to perform time association, business association, user association and device association on the standard data set to build an associated data network.

[0043] Furthermore, the multi-dimensional data quality assessment system also includes: The benchmark construction and correction module 102 is used to analyze the operating rules of the associated data network according to the multiple source data types, construct multiple data quality benchmarks, and perform dynamic benchmark correction to generate multiple dynamic quality benchmarks.

[0044] In this embodiment of the invention, the benchmark construction and correction module 102 performs operational pattern analysis on the associated data network according to multiple source data types, including update cycle, fluctuation range, data distribution, missing patterns and / or growth patterns, statistically analyzes multiple types of pattern data, and generates multiple quality reference intervals based on the multiple types of pattern data. Then, according to the multiple quality reference intervals, it constructs data quality benchmarks corresponding to multiple source data types, obtains the current operating status, and then performs dynamic benchmark correction on the multiple data quality benchmarks according to the current operating status to generate multiple dynamic quality benchmarks.

[0045] Specifically, in another preferred embodiment provided by the present invention, the benchmark construction correction module 102 specifically includes: The pattern analysis unit is used to perform operational pattern analysis on the associated data network according to multiple source data types, and to statistically analyze multiple types of pattern data; An interval generation unit is used to generate multiple quality reference intervals based on multiple types of regular data. A benchmark construction unit is used to construct multiple data quality benchmarks according to multiple quality reference intervals; The status acquisition unit is used to acquire the current running status; The dynamic correction unit is used to dynamically correct multiple data quality benchmarks based on the current operating state, thereby generating multiple dynamic quality benchmarks.

[0046] Furthermore, the multi-dimensional data quality assessment system also includes: The data multidimensional evaluation module 103 is used to acquire the data to be evaluated, and to perform multidimensional synchronous evaluation of the data to be evaluated based on multiple dynamic quality benchmarks to obtain multidimensional quality results.

[0047] In this embodiment of the invention, after the data multidimensional evaluation module 103 acquires the data to be evaluated, it performs data integrity evaluation, data consistency evaluation, data accuracy evaluation, data timeliness evaluation, and data stability evaluation on the data to be evaluated based on multiple dynamic quality benchmarks, obtaining integrity evaluation results, consistency evaluation results, accuracy evaluation results, timeliness evaluation results, and stability evaluation results, respectively. Then, the integrity evaluation results, consistency evaluation results, accuracy evaluation results, timeliness evaluation results, and stability evaluation results are integrated to generate a multidimensional quality result. Specifically, data integrity evaluation is a process of analyzing data missing rate, field null value rate, record continuity rate, etc.; data consistency evaluation is a process of analyzing differences between databases, logs, caches, and backups; data accuracy evaluation is a process of analyzing whether constraints are violated, whether outliers occur, and whether abnormal combinations occur; data timeliness evaluation is a process of analyzing update time delay, synchronization delay, acquisition delay, and write delay; and data stability evaluation is a process of analyzing volatility, abnormal growth rate, abnormal decline rate, and jitter frequency.

[0048] The quality level analysis module 104 is used to perform correlation verification on the multidimensional quality results, and after the verification is passed, to perform fusion analysis on the multidimensional quality results to determine the comprehensive quality level and compare it with the preset standard threshold.

[0049] In this embodiment of the invention, the quality level analysis module 104 performs correlation verification on the multidimensional quality results, determines whether the verification is successful, and determines the evaluation fusion weight if the verification is successful. Then, according to the evaluation fusion weight, it performs fusion analysis on the multidimensional quality results, calculates the comprehensive quality credibility index, and matches the corresponding comprehensive quality level from the preset quality grading table according to the comprehensive quality credibility index, and compares the comprehensive quality level with the preset standard threshold.

[0050] Specifically, in another preferred embodiment provided by the present invention, the quality grade analysis module 104 specifically includes: The correlation verification unit is used to perform correlation verification on the multidimensional quality results and determine whether the verification passes. The weight determination unit is used to determine the evaluation fusion weights after the verification is passed; The fusion analysis unit is used to perform fusion analysis on the multidimensional quality results according to the evaluation fusion weights and calculate the comprehensive quality credibility index. A grade matching unit is used to match the corresponding comprehensive quality grade based on the comprehensive quality reliability index. The threshold comparison unit is used to compare the overall quality level with a preset standard threshold.

[0051] Furthermore, the multi-dimensional data quality assessment system also includes: The anomaly location processing module 105 is used to perform anomaly location analysis, determine the anomaly location result, and perform targeted optimization processing when the overall quality level is lower than the standard threshold.

[0052] In this embodiment of the invention, when the overall quality level is lower than the standard threshold, the anomaly location processing module 105 determines that there is a data anomaly. At this time, by performing anomaly location analysis on the multi-dimensional quality results, including anomaly location tracing of data sources, synchronization links, databases, acquisition interfaces, cache nodes and computing nodes, the anomaly location result is determined. Then, according to the anomaly location result, an anomaly optimization strategy is formulated, and then targeted optimization processing is performed according to the anomaly optimization strategy.

[0053] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A multi-dimensional data quality assessment method, characterized in that, The method specifically includes the following steps: Access multiple data sources in the data center, determine the types of multiple source data, perform phased data aggregation, standardization processing and correlation analysis, and build a correlated data network; Based on the multiple source data types, the operation pattern of the associated data network is analyzed, multiple data quality benchmarks are constructed, and the benchmarks are dynamically corrected to generate multiple dynamic quality benchmarks. The data to be evaluated is acquired, and a multi-dimensional synchronous evaluation is performed on the data to be evaluated based on multiple dynamic quality benchmarks to obtain multi-dimensional quality results. The multidimensional quality results are correlated and verified. After the verification is passed, the multidimensional quality results are fused and analyzed to determine the comprehensive quality level and compare it with the preset standard threshold. When the overall quality level is lower than the standard threshold, anomaly location analysis is performed to determine the anomaly location results, and targeted optimization processing is carried out.

2. The multi-dimensional data quality assessment method according to claim 1, characterized in that, The process of accessing multiple data sources in the data center, determining the data types of multiple sources, performing phased data aggregation, standardization processing, and correlation analysis, and constructing a correlated data network specifically includes the following steps: Access to multiple data sources in the data center; Determine the multiple source data types corresponding to the multiple data sources; Planning data collection phase; According to the data aggregation stages, the data from multiple data sources is aggregated in stages to obtain a stage data set; The aforementioned stage data set is standardized to obtain a standard data set; The standard data set is correlated by time, business, user, and device to construct a correlated data network.

3. The multi-dimensional data quality assessment method according to claim 1, characterized in that, The step of analyzing the operational patterns of the associated data network according to multiple source data types, constructing multiple data quality benchmarks, and dynamically correcting these benchmarks to generate multiple dynamic quality benchmarks specifically includes the following steps: Based on the multiple source data types, the operation pattern of the associated data network is analyzed, and multiple types of pattern data are statistically analyzed; Based on the data of the aforementioned types, multiple quality reference intervals are generated; Multiple data quality benchmarks are constructed based on the aforementioned quality reference intervals; Get the current running status; Based on the current operating status, multiple data quality benchmarks are dynamically corrected to generate multiple dynamic quality benchmarks.

4. The multi-dimensional data quality assessment method according to claim 1, characterized in that, The process of acquiring the data to be evaluated, and performing a multi-dimensional synchronous evaluation of the data based on multiple dynamic quality benchmarks to obtain multi-dimensional quality results, specifically includes the following steps: Obtain the data to be evaluated; Based on multiple dynamic quality benchmarks, the data integrity assessment is performed on the data to be evaluated to obtain the integrity assessment result. Based on multiple dynamic quality benchmarks, a data consistency assessment is performed on the data to be evaluated to obtain a consistency assessment result. Based on multiple dynamic quality benchmarks, the data to be evaluated is assessed for accuracy, and an accuracy assessment result is obtained. Based on multiple dynamic quality benchmarks, the timeliness of the data to be evaluated is assessed to obtain the timeliness assessment result. Based on multiple dynamic quality benchmarks, the data stability assessment of the data to be evaluated is performed to obtain the stability assessment results. By combining the completeness assessment results, the consistency assessment results, the accuracy assessment results, the timeliness assessment results, and the stability assessment results, a multidimensional quality result is generated.

5. The multi-dimensional data quality assessment method according to claim 1, characterized in that, The process of performing correlation verification on the multidimensional quality results, and after successful verification, performing fusion analysis on the multidimensional quality results to determine the comprehensive quality level and comparing it with a preset standard threshold specifically includes the following steps: The multidimensional quality results are subjected to correlation verification to determine whether the verification passes. After successful verification, the evaluation fusion weights are determined; According to the evaluation fusion weights, the multidimensional quality results are fused and analyzed to calculate the comprehensive quality credibility index. Based on the comprehensive quality reliability index, the corresponding comprehensive quality level is matched; The overall quality level is compared with a preset standard threshold.

6. The multi-dimensional data quality assessment method according to claim 1, characterized in that, When the overall quality level is lower than the standard threshold, anomaly localization analysis is performed to determine the anomaly localization results, and targeted optimization processing is carried out. Specifically, this includes the following steps: When the overall quality level is lower than the standard threshold, it is determined that there is a data anomaly; Anomaly localization analysis was performed on the multidimensional quality results to determine the anomaly localization results; Based on the anomaly localization results, formulate anomaly optimization strategies; Targeted optimization processes are performed according to the aforementioned anomaly optimization strategy.

7. A multi-dimensional data quality assessment system, characterized in that, The system includes a data aggregation and processing module, a benchmark construction and correction module, a multidimensional data evaluation module, a quality level analysis module, and an anomaly location and processing module, wherein: The data aggregation and processing module is used to access multiple data sources in the data center, determine the types of multiple source data, perform phased data aggregation, standardization processing and correlation analysis, and build a related data network. The benchmark construction and correction module is used to analyze the operating rules of the associated data network according to multiple source data types, construct multiple data quality benchmarks, and perform dynamic benchmark correction to generate multiple dynamic quality benchmarks. The data multidimensional evaluation module is used to acquire the data to be evaluated, and to perform multidimensional synchronous evaluation of the data to be evaluated based on multiple dynamic quality benchmarks to obtain multidimensional quality results. The quality level analysis module is used to perform correlation verification on the multidimensional quality results, and after the verification is passed, to perform fusion analysis on the multidimensional quality results to determine the comprehensive quality level and compare it with the preset standard threshold. The anomaly location processing module is used to perform anomaly location analysis, determine the anomaly location result, and perform targeted optimization processing when the overall quality level is lower than the standard threshold.

8. The multi-dimensional data quality assessment system according to claim 7, characterized in that, The data aggregation and processing module specifically includes: Data source access unit, used to access multiple data sources in the data center; A type determination unit is used to determine multiple source data types corresponding to the multiple data sources; Phase planning unit, used for planning the data collection phase; The data aggregation unit is used to aggregate data from multiple data sources according to the data aggregation stage, and obtain a stage data set. A standardization processing unit is used to standardize the stage data set to obtain a standard data set. The data association unit is used to perform time association, business association, user association and device association on the standard data set to build an associated data network.

9. The multi-dimensional data quality assessment system according to claim 7, characterized in that, The benchmark construction and correction module specifically includes: The pattern analysis unit is used to perform operational pattern analysis on the associated data network according to multiple source data types, and to statistically analyze multiple types of pattern data; An interval generation unit is used to generate multiple quality reference intervals based on multiple types of regular data. A benchmark construction unit is used to construct multiple data quality benchmarks according to multiple quality reference intervals; The status acquisition unit is used to acquire the current running status; The dynamic correction unit is used to dynamically correct multiple data quality benchmarks based on the current operating state, thereby generating multiple dynamic quality benchmarks.

10. The multi-dimensional data quality assessment system according to claim 7, characterized in that, The quality level analysis module specifically includes: The correlation verification unit is used to perform correlation verification on the multidimensional quality results and determine whether the verification passes. The weight determination unit is used to determine the evaluation fusion weights after the verification is passed; The fusion analysis unit is used to perform fusion analysis on the multidimensional quality results according to the evaluation fusion weights and calculate the comprehensive quality credibility index. A grade matching unit is used to match the corresponding comprehensive quality grade based on the comprehensive quality reliability index. The threshold comparison unit is used to compare the overall quality level with a preset standard threshold.

Images