Data management method and system based on financial cloud data warehouse and data lake

Abstract

The invention provides a data management method and system based on a financial cloud data warehouse and a data lake, and the method comprises the steps: 1, distinguishing backbone data and edge data according to data properties and data values when a new data source is introduced; 2, for the backbone data, performing table-level analysis and field-level analysis on the original data according to a data model construction thought, and completing introduction of the backbone data according to an analysis result; and 3, introducing the edge data according to a real-time wide table, establishing a target wide table according to a source data format, compiling an ETL script, and completing script unit testing, thereby completing the introduction of the edge data. According to the invention, a set of solution that the data warehouse is combined with the data lake is realized, and the data warehouse has the characteristics of high data relevance and uniform standard according to the principle of sorting and storing; the data lake is more beneficial to efficient data analysis and data exploration according to the principle of first storage and then use.

Description

technical field [0001] The present invention relates to the technical field of data management, in particular to a data management method and system based on a financial cloud data warehouse and a data lake. Background technique [0002] The data warehouse in the financial cloud currently serves more than 150 cooperative banks, and basically deals with homogeneous data problems. The traditional warehouse model can solve such problems well; however, there are inevitably individual needs. At this time, the traditional Warehouse model processing ideas often require a lot of research and development costs; [0003] The data lake solution is somewhat similar to a large-scale ODS. Compared with the data warehouse, the R&D cost is lower and the R&D cycle is faster, but the data integration degree is low, and the data quality and reliability are relatively low. [0004] After implementing the financial cloud data warehouse for a period of time, I found some problems: (1) If the dat...

AI Technical Summary

Problems solved by technology

[0002] The data warehouse in the financial cloud currently serves more than 150 cooperative banks, and basically deals with homogeneous data problems. The traditional warehouse model can solve such problems well; however, there are inevitably individual needs. At this time, the traditional Warehouse model processing ideas often require a lot of research and development costs;

[0003] The data lake solution is somewhat similar to a large ODS. Compared with the data warehouse, the R&D cost is lower and the R&D cycle is faster, but the data integration degree is low, and the data quality and credibility are relatively low.

[0004] After implementing the financial cloud data warehouse for a period of time, I found some problems: (1) If the data ecology of the enterprise is in a stable state, it will be very efficient to use the data warehouse as a data solution; but if the data ecology changes frequently, using the data warehouse will require a lot of Manpower to operate the model; (2) The main function of using data warehouse is to solve the problem of data islands, but the whole process is not very intelligent and requires enough manpower; (3) Use the traditional structure of data warehouse to realize some deep data mining and real-time data analysis, very awkward

However, this patent still requires a large amount of data processing and cannot perfectly solve the problem of data islands

Images