Method and system for validating data

Abstract

A method and system for validating data. Warehouse data is generated by transforming source data via an ETL transformation model. A data cube is generated by transforming the warehouse data via an OLAP transformation model. A report dataset (MDS1) is generated from the data cube. A reference dataset (S) is generated from the source data. Whether MDS1 matches S is determined. If MDS1 doesn't match S, then an OLAP inverse transformation is performed on MDS1 to generate an OLAP dataset (MDS2) and whether MDS2 matches S is determined. If MDS1 doesn't match S and MDS2 does not match S, then an ETF inverse transformation is performed on MDS2 to generate an ETL dataset (MDS3) and whether MDS2 matches MDS1 and whether MDS3 matches S is determined. If MDS1 doesn't match S and MDS2 does not match S and MDS3 does not match S, then whether MDS3 matches MDS2 is determined.

Description

FIELD OF THE INVENTION[0001]The present invention relates to data processing technology, and in particular, to a method and a system for validating data.BACKGROUND OF THE INVENTION[0002]With the development of information technology, more and more people begin to use relevant technology on business intelligence to analyze and process business data to provide powerful support for decision-makers. Also, with the development and application of database technology, the data amount stored in a database rocketed high from mega (M) bytes and gigabytes (G) in the 1980s to current trillion (T) bytes and peta (P) bytes. Meanwhile, query requirements from users also become increasingly complex, which involves not only querying or manipulating one or more pieces of records in a relational table but also performing data analysis and information syntheses on tens of millions of pieces of recorded data in a plurality of tables. However, a transaction processing type relational database system cann...

AI Technical Summary

Benefits of technology

This approach enables easy validation of data problems and reduces the workload required to determine their causes, ensuring data accuracy and improving the efficiency of data processing in BI systems.

Problems solved by technology

Meanwhile, query requirements from users also become increasingly complex, which involves not only querying or manipulating one or more pieces of records in a relational table but also performing data analysis and information syntheses on tens of millions of pieces of recorded data in a plurality of tables.

However, a transaction processing type relational database system cannot meet all such requirements.

For operation and analytical type applications, they cannot meet performance requirements; thus, people always release the restriction on redundancy in a relational database and introduces statistical and integrated data.

However, the application logics of such statistical and integrated data are dispersed, random, and unsystematic; thus, the analytical function is limited, inflexible, and difficult to maintain.

However, since there are varieties of data sources and processing of ETL model and OLAP model involves a great mount of data, error likely occurs during the BI data processing process.

However, since the data amount in the report is too high, a comprehensible comparison is usually impossible.

Besides, even if it is found that the data in the report is inconsistent with the original data in the application system, it is impossible to determine the cause of the problem.

The workload for comprehensively checking data in the models and data warehouse is overwhelmingly large, which always needs considerable time to determine the cause of the problem.

Images