Data management method and system based on financial cloud data warehouse and data lake

Abstract

The present invention provides a data management method and system based on a financial cloud data warehouse and a data lake, including: Step 1: When introducing a new data source, distinguish between backbone data and edge data according to the nature and value of the data; Step 2: For backbone data, perform table-level analysis and field-level analysis on the original data according to the data model construction idea, and complete the introduction of backbone data according to the analysis results; Step 3: For edge data, introduce according to the real-time wide table, establish the target wide table according to the source data format, write the ETL script, complete the script unit test, and thus complete the introduction of edge data. The present invention implements a solution of combining a data warehouse with a data lake. The data warehouse has the characteristics of strong data correlation and unified standards based on the principle of sorting first and then storing; the data lake is more conducive to efficient data analysis and data exploration based on the principle of storing first and then using.

Description

Technical Field

[0001] The present invention relates to the field of data management technology, and in particular to a data management method and system based on a financial cloud data warehouse and a data lake. Background Art

[0002] The data warehouse in the financial cloud currently serves more than 150 partner banks, and basically handles homogeneous data problems. Traditional warehouse models can solve such problems very well. However, there are inevitably personalized needs. At this time, the use of traditional warehouse model processing ideas often requires a lot of R&D costs.

[0003] The data lake solution is somewhat similar to a large ODS. Compared with a data warehouse, it has a lower R&D cost and a faster R&D cycle, but it has a low degree of data integration, and the data quality and credibility are relatively low.

[0004] After implementing the financial cloud data warehouse for a period of time, we found some problems: (1) If the enterprise's data ecology is in a stable state, using the data warehouse to provide data solutions will be very efficient; but if the data ecology changes frequently, using the data warehouse will require a lot of manpower to operate the model; (2) One of the main functions of using the data warehouse is to solve the problem of data islands, but the whole process is not very smart and requires sufficient manpower; (3) Using the traditional data warehouse architecture to implement some data deep mining and real-time data analysis is very awkward. Therefore, it is necessary to introduce the concept of data lake, supplemented by strict data life cycle management, so as to achieve "short, flat, and fast" data research and development.

[0005] Patent document CN113434497A (application number: CN202110984703.2) discloses a data element vault composed of a data warehouse and a data element warehouse, which includes: a data warehouse for storing data, which cannot be output from the data element vault; a data element warehouse for storing data elements, which include element models and element results, and the element results are data sets for outputting the data element vault after the data is processed by the element model. However, this patent still requires a lot of data processing and cannot perfectly solve the problem of data islands. Summary of the invention

[0006] In view of the defects in the prior art, the purpose of the present invention is to provide a data management method and system based on a financial cloud data warehouse and a data lake.

[0007] The data management method based on the financial cloud data warehouse and data lake provided by the present invention includes:

[0008] Step 1: When introducing new data sources, distinguish between backbone data and edge data based on the nature and value of the data;

[0009] Step 2: For the backbone data, according to the idea of ​​data model construction, perform table-level analysis and field-level analysis on the original data, and complete the introduction of backbone data based on the analysis results;

[0010] Step 3: For edge data, import it according to the real-time wide table, create the target wide table according to the source data format, write the ETL script, and complete the script unit test to complete the edge data introduction.

[0011] Preferably, the step 2 comprises:

[0012] Step 2.1: Write Mapping based on the analysis results of table-level analysis and field-level analysis, clarify the model positioning of data tables or fields, generate ETL scripts, and complete script unit testing;

[0013] Step 2.2: Verify the logic correctness and accuracy of the script in the test environment, and complete data quality checks, including data uniqueness, data relevance, and data standardization;

[0014] Step 2.3: Launch the script, configure scheduling, and complete the introduction of backbone data.

[0015] Preferably, step 3 comprises:

[0016] Step 3.1: Verify the logic correctness and accuracy of the script in the test environment;

[0017] Step 3.2: Launch the script, configure the schedule, determine the data retention policy, and complete the edge data introduction;

[0018] Step 3.3: Identify data consumers when ingesting edge data and manage the data lifecycle.

[0019] Preferably, the backbone data includes account, customer, organization, account and channel data; the backbone data is integrated into the data warehouse according to the model construction idea to ensure the collection degree of the introduced data;

[0020] The edge data includes transaction details, flow information and statistical analysis wide tables; the edge data is stored in the form of detailed fact wide tables, while maintaining the correlation between the edge data and the backbone data.

[0021] Preferably, the data properties are distinguished by manual identification, or the data properties are distinguished automatically by machine learning;

[0022] The data value is identified through manual recognition. The higher the data reuse rate, the higher the data value.

[0023] The data management system based on the financial cloud data warehouse and data lake provided by the present invention includes:

[0024] Module M1: When introducing new data sources, distinguish between backbone data and edge data based on the nature and value of the data;

[0025] Module M2: For backbone data, according to the idea of ​​data model construction, table-level analysis and field-level analysis are performed on the original data, and the backbone data is introduced according to the analysis results;

[0026] Module M3: For edge data, according to the real-time wide table introduction, the target wide table is established according to the source data format, the ETL script is written, and the script unit test is completed to complete the edge data introduction.

[0027] Preferably, the module M2 comprises:

[0028] Module M2.1: Write Mapping based on the analysis results of table-level analysis and field-level analysis, clarify the model positioning of data tables or fields, generate ETL scripts, and complete script unit testing;

[0029] Module M2.2: Verify the logical correctness and accuracy of the script in the test environment, and complete data quality checks, including data uniqueness, data relevance, and data standardization;

[0030] Module M2.3: Launch scripts, configure scheduling, and complete the introduction of backbone data.

[0031] Preferably, the module M3 includes:

[0032] Module M3.1: Verify the logical correctness and accuracy of the script in the test environment;

[0033] Module M3.2: Launch scripts, configure scheduling, determine data retention policies, and complete edge data introduction;

[0034] Module M3.3: Identify data consumers when introducing edge data and perform data lifecycle management.

[0035] Preferably, the backbone data includes account, customer, organization, account and channel data; the backbone data is integrated into the data warehouse according to the model construction idea to ensure the collection degree of the introduced data;

[0036] The edge data includes transaction details, flow information and statistical analysis wide tables; the edge data is stored in the form of detailed fact wide tables, while maintaining the correlation between the edge data and the backbone data.

[0037] Preferably, the data properties are distinguished by manual identification, or the data properties are distinguished automatically by machine learning;

[0038] The data value is identified through manual recognition. The higher the data reuse rate, the higher the data value.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] The present invention implements a solution of combining a data warehouse with a data lake. The data warehouse is based on the principle of sorting first and then storing, and has the characteristics of strong data correlation and unified standards; the data lake is based on the principle of storing first and then using, which is more conducive to efficient data analysis and data exploration. The solution uses the advantages of the above two respectively: using the data warehouse model as the skeleton, supplemented by the data detail layer of the data lake, the backbone data is sorted and analyzed and then entered into the model, and the edge data is stored in the data detail layer close to the source; it can reduce data research and development costs, while ensuring data aggregation to avoid the generation of data islands. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0042] Figure 1 This is the architectural diagram of the lake-warehouse integrated solution. DETAILED DESCRIPTION

[0043] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0044] Example:

[0045] Conventional introduction of loan systems often requires in-depth understanding of the system's internal accounting system, risk assessment system, and audit process, which means that the development and testing cycle will basically not be less than three months. The financial cloud data warehouse serves more than 150 partner banks, and the online loan products used by each bank are diverse. Introducing them in a conventional way will require a lot of manpower, and this access solution is basically not feasible.

[0046] The lake-warehouse integrated model is adopted to split the contracts, promissory notes, and customer information in the online loan system and incorporate them into the subject model to ensure the OneData data standards of accounts and customers; the remaining guarantee information, credit information, repayment flow and other information are stored in the detail layer according to the source system structure; this can greatly reduce R&D costs and control the development cycle within one person per month; for data consumers, because the model guarantees the data standards of accounts and customers, it will not significantly increase R&D testing costs and is within an acceptable range.

[0047] The data management method based on the financial cloud data warehouse and data lake provided by the present invention includes:

[0048] Step 1: When introducing a new data source, data analysts need to clarify the importance of the data; distinguish the data based on its nature and value, and classify it as backbone data or edge data;

[0049] Step 2: Core data generally includes data on accounts, customers, institutions, accounts, and channels. This type of data is integrated into the data warehouse according to the model construction idea, so as to ensure that the introduced data has a certain degree of aggregation and does not form data islands;

[0050] Step 3: Edge data generally includes transaction details, flow information, and wide tables of statistical analysis in a specific field. This type of data has clear application scenarios, low reuse, and low integration value, and is stored in the form of wide tables of detailed facts. However, it is necessary to ensure that edge data and backbone data have a certain degree of correlation.

[0051] Specifically, the step 1 includes:

[0052] Step 1.1: When introducing a new data source, roughly distinguish between backbone data and edge data based on the nature and value of the data;

[0053] The nature of data can be explained by an example. When describing an event, the three Ws (when, who, where) are the main focus. These information constitute the backbone of the event, and all other information is descriptive information. For the financial industry, accounts, customers, institutions, and channels are its backbone information. These information together constitute the data framework of the industry. In terms of data nature, this part of data is backbone data; data nature can be identified manually or automatically distinguished through machine learning.

[0054] The value of data is reflected by the data reuse rate. Generally, the higher the reuse rate of data, the higher the requirements for data accuracy and standardization. Such data needs to be integrated into the model to ensure data quality. For the financial industry, accounting data is data with higher data value; because the value of data is gradually reflected after data analysis, it is not distinguished when the data is first introduced, and is only refined into the data model afterwards, usually by manual identification.

[0055] The step 2 comprises:

[0056] Step 2.1: The backbone data performs table-level and field-level analysis on the original data according to the data model construction ideas, and clarifies the model positioning of the data table or some fields;

[0057] Step 2.2: Based on the analysis results, write the Mapping, generate the ETL script, and complete the script unit test;

[0058] Step 2.3: Verify the logic correctness and accuracy of the script in the test environment; at the same time, complete data quality checks, such as data uniqueness, data relevance, and data standardization;

[0059] Step 2.4: Launch the script, configure scheduling, and complete the introduction of backbone data.

[0060] The step 3 comprises:

[0061] Step 3.1: Import edge data into a real-time wide table, create a target wide table based on the source data format, write an ETL script, and complete the script unit test;

[0062] Step 3.2: Verify the logic correctness and accuracy of the script in the test environment;

[0063] Step 3.3: Launch the script, configure the schedule, determine the data retention policy, and complete the edge data introduction;

[0064] Step 3.4: Edge data import must have clear data consumers and establish strict data lifecycle management to avoid data being stored but not used, thus forming a data swamp.

[0065] like Figure 1 Compared with the traditional data warehouse solution, the present invention mainly has an additional data detail layer (DWD), which stores data in the form of a source wide table and stores history in the form of snapshots, specifically:

[0066] 1. The DWD layer and the DW layer are in parallel relationship, not front-to-back dependency: Because after the data is introduced, the backbone data will be extracted into the model, so the DWD and DW layers can be associated through the backbone data to avoid data islands; secondly, the DWD layer is close to the source storage, without the need for complex ETL processing, reducing R&D costs.

[0067] 2. Support both structured and unstructured data; data lake can support the storage and use of unstructured data.

[0068] 3. Support both batch and real-time modes; the traditional data warehouse architecture can also support the real-time mode, but it is not very user-friendly. Taking the real-time Lambda architecture as an example, it requires the establishment of a batch data processing flow and a streaming data processing flow. Since the data warehouse mode requires strict data standards and a high degree of data integration, first of all, the execution efficiency of streaming data processing will be difficult to guarantee, and secondly, it will be very difficult to ensure the consistency of data caliber of the two processing flows at the same time; the lake warehouse integration is very suitable for the real-time mode. The DW layer completes the batch data processing flow, and the DWD layer is based on the batch processing results, and the streaming processing results are incrementally superimposed, which can guarantee data accuracy to a certain extent and execution efficiency.

[0069] 4. The DM layer is used to support traditional statistical analysis data applications that comply with strict data standards and have high data accuracy. The data basically comes from the DW layer; SandBox is used for data exploration, which requires speed and flexibility. The data comes from both the DWD layer and the DW layer.

[0070] 5. The DW layer and DM layer comply with OneData standards and can be defined as data trusted areas; the DWD layer and SandBox are defined as data native areas.

[0071] Those skilled in the art know that, in addition to implementing the system, device and its various modules provided by the present invention in a purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers and embedded microcontrollers by logically programming the method steps. Therefore, the system, device and its various modules provided by the present invention can be considered as a hardware component, and the modules included therein for implementing various programs can also be considered as structures within the hardware component; the modules for implementing various functions can also be considered as both software programs for implementing the method and structures within the hardware component.

[0072] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A data management method based on financial cloud data warehouse and data lake, characterized in that: include: Step 1: When introducing new data sources, distinguish between backbone data and edge data based on the nature and value of the data; Step 2: For the backbone data, according to the idea of ​​data model construction, perform table-level analysis and field-level analysis on the original data, and complete the introduction of backbone data based on the analysis results; Step 3: For edge data, import it according to the real-time wide table, create the target wide table according to the source data format, write the ETL script, and complete the script unit test to complete the edge data import; The step 2 comprises: Step 2.1: Write Mapping based on the analysis results of table-level analysis and field-level analysis, clarify the model positioning of data tables or fields, generate ETL scripts, and complete script unit testing; Step 2.2: Verify the logic correctness and accuracy of the script in the test environment, and complete data quality checks, including data uniqueness, data relevance, and data standardization; Step 2.3: Launch the script, configure the schedule, and complete the introduction of backbone data; The step 3 comprises: Step 3.1: Verify the logic correctness and accuracy of the script in the test environment; Step 3.2: Launch the script, configure the schedule, determine the data retention policy, and complete the edge data introduction; Step 3.3: Identify data consumers when introducing edge data and manage the data lifecycle; The backbone data includes account, customer, organization, financial and channel data; the backbone data is integrated into the data warehouse according to the model construction idea to ensure the collection degree of the introduced data; The edge data includes transaction details, flow information and statistical analysis wide tables; the edge data is stored in the form of detailed fact wide tables, while maintaining the correlation between the edge data and the backbone data; Distinguish the nature of data through manual identification, or realize automatic differentiation of data nature through machine learning; The data value is identified through manual recognition. The higher the data reuse rate, the higher the data value.

2. A data management system based on financial cloud data warehouse and data lake, characterized in that: include: Module M1: When introducing new data sources, distinguish between backbone data and edge data based on the nature and value of the data; Module M2: For backbone data, according to the idea of ​​data model construction, table-level analysis and field-level analysis are performed on the original data, and the backbone data is introduced according to the analysis results; Module M3: For edge data, according to the real-time wide table import, the target wide table is established according to the source data format, the ETL script is written, and the script unit test is completed to complete the edge data import; The module M2 comprises: Module M2.1: Write Mapping based on the analysis results of table-level analysis and field-level analysis, clarify the model positioning of data tables or fields, generate ETL scripts, and complete script unit testing; Module M2.2: Verify the logical correctness and accuracy of the script in the test environment, and complete data quality checks, including data uniqueness, data relevance, and data standardization; Module M2.3: Launch scripts, configure scheduling, and complete the introduction of backbone data; The module M3 comprises: Module M3.1: Verify the logical correctness and accuracy of the script in the test environment; Module M3.2: Launch scripts, configure scheduling, determine data retention policies, and complete edge data introduction; Module M3.3: Identify data consumers when introducing edge data and manage the data lifecycle; The backbone data includes account, customer, organization, financial and channel data; the backbone data is integrated into the data warehouse according to the model construction idea to ensure the collection degree of the introduced data; The edge data includes transaction details, flow information and statistical analysis wide tables; the edge data is stored in the form of detailed fact wide tables, while maintaining the correlation between the edge data and the backbone data; Distinguish the nature of data through manual identification, or realize automatic differentiation of data nature through machine learning; The data value is identified through manual recognition. The higher the data reuse rate, the higher the data value.

Images