Visual management method based on financial data assets

Abstract

The invention discloses a visual management method based on financial data assets, and relates to the technical field of visual management, and the method comprises the following steps: accessing a data source configuration module to a data platform; setting a number warehouse layering and theme division rule; the logic model configures metadata input by the module according to the configured rule and the data source; transmitting the complete financial data warehouse physical model to a table management module; the table management module automatically updates and synchronizes the result of the physical model; the data consanguinity map module integrates physical model update information; the data asset management module takes data of the blood relationship map module as a basis; according to the method, the data asset management system can perform visual display and data management analysis on the updated model file, so that the data warehouse management efficiency is improved, a data warehouse modeling process can be standardized, modeling redundancy is reduced, an automatic, agile and comprehensive modeling system is realized, and the modeling efficiency is improved. And making and management of decisions related to warehouse counting by a user are facilitated.

Description

Technical Field

[0001] The present invention relates to the technical field of visual management, and particularly relates to a visual management method based on financial data assets. Background Art

[0002] A data warehouse is an important way for enterprises to process and store data, and is one of the important bases for enterprise decision-making analysis. Users can perform operations such as addition, deletion, modification, and query in the data warehouse, and build business intelligence based on the data warehouse. With the rapid development of the information technology era, the requirements for the storage, management, and analysis of financial big data in the financial industry are gradually increasing. Different types of data warehouse models need to be established according to different types of financial data. Therefore, building a financial big data warehouse is the only way for financial enterprises to achieve the development of financial technology.

[0003] Currently, the data in the financial industry covers a wide range of aspects, with the characteristics of multiple types and large volumes. Data application is an important direction for the transformation of financial enterprises. The dependence of financial enterprises on big data is gradually increasing. In addition to requiring enterprises to have high data analysis capabilities, it also requires high data management capabilities. Having the correct data flow and clear lineage analysis is an important guarantee factor for making rapid, correct, and timely enterprise business decisions. Therefore, a complete and efficient financial big data asset management system needs to be established. The visualization of data asset management can not only more intuitively display the data logical association relationship for users, but also help data managers more conveniently and quickly master the changes and inventory of data assets and identify data risks.

[0004] Among them, in a visual data warehouse modeling method and system with Chinese Patent Application No. CN202011542971.0, a visual data warehouse modeling method and system are disclosed. The main method is to combine the web side and the hive data warehouse, and build data quality warning rules according to the directory configured on the web side. This method mainly focuses on the retrieval and monitoring of the data warehouse, and does not involve how to improve the efficiency of data warehouse modeling and the process of visualizing data warehouse data assets. Summary of the Invention

[0005] The purpose of the present invention is to provide a visual management method based on financial data assets to overcome the above-mentioned defects in the prior art.

[0006] A visual management method based on financial data assets includes the following steps:

[0007] S1. A data warehouse engineer accesses the data source configuration module on the data platform, sorts out the metadata according to business requirements, and transmits it to the logical model processing module;

[0008] S2. In the logical model processing module, the data warehouse layering and theme division rules need to be set for processing metadata.

[0009] S3. The logical model makes rule judgments based on the configured rules and the metadata input by the data source configuration module, and inputs the logical model results into the financial theme domain module.

[0010] S4. The financial theme domain module integrates the input results of the logical model and transmits the complete physical model of the financial data warehouse to the table management module.

[0011] S5. The table management module automatically updates and synchronizes the results of the physical model and transmits the model information to the data asset management system.

[0012] S6. The data lineage map module integrates the physical model update information, integrates the relationships of each physical model in the map, and updates the information in the data asset management system.

[0013] S7. The data asset management module updates the indicators and descriptions related to the data asset dimension based on the data of the data lineage map module.

[0014] S8. According to the physical model update data obtained in steps S6 to S7, the data asset management system performs corresponding visualization processing and display operations.

[0015] Preferably, the specific process of step S1 is that the data warehouse engineer configures the data source according to the common data structures and output forms in the financial industry, and plans the data warehouse configuration scheme within the financial sub-industry.

[0016] Preferably, the specific process of step S2 is that the data warehouse engineer conducts the overall planning of the data warehouse according to the common layering and theme domain division rules in the financial industry, and sets the flowchart for the logical model to process metadata.

[0017] Preferably, the specific process of step S3 includes the following steps:

[0018] S3.1. The data warehouse engineer uses a visual modeling tool in the logical processing module and designs it using the drag-and-drop method.

[0019] S3.2. Extract the metadata required for the database tables from the data platform.

[0020] S3.3. The design area can automatically identify the mapping relationships between metadata, automatically update the relationships between metadata through version control, and establish logical relationships between database tables based on the primary key relationships.

[0021] S3.4. Allow design through custom logic methods, create model entities and attributes, and establish connections with other entities in the data warehouse.

[0022] Preferably, the specific process of step S4 includes the following steps:

[0023] S4.1. The data warehouse engineer classifies the logical model into the corresponding subject area in the financial subject area module, and obtains the physical model after executing the code script in their respective subject area modules;

[0024] S4.2. Form the mapping relationship between the metadata and the database table, generate the corresponding version number, and provide a basis for automatically updating the entity logical relationship of the model;

[0025] S4.3. The data platform automatically updates the metadata according to the version number, records the update history and the corresponding version; at the same time, generates a backup table of the previous version of the metadata, which does not affect the normal use of database queries, so as to realize the automatic inspection and update of the system metadata without affecting downstream use. This type of update supports the corresponding synchronization and recovery function.

[0026] Preferably, the content of the record updated by the table management module in step S5 includes: the basic attributes of the table fields in the physical model, the content of the table fields, the rules of the table fields, and the update rules and save rules of the table. The table management module interacts the metadata update information with the data lineage map module and the asset management module. The data lineage map module sorts out the metadata update information and updates the visual lineage analysis view displayed on the front end, and the data asset management sorts out the metadata update information and updates the data report displayed on the front end.

[0027] Preferably, step S6 should specifically include a data asset intelligent search module, a directory access module, and a table structure query module. Users can obtain the upstream and downstream table information corresponding to the target physical model through the intelligent search module, obtain the hierarchical or subject area table structure relationship of the data warehouse through the directory access module, and obtain the specific information of the searched target physical model through the table structure query module.

[0028] Preferably, the data lineage map module in step S6 supports real-time update and second-level query functions, so that designers can make more appropriate data solutions according to the latest information during front-end design. The specific lineage units in the data lineage map module should include: ETL lineage, table-table lineage, and field lineage.

[0029] Preferably, step S7 should specifically include a data panoramic view module, a data overview report module, a data redundancy evaluation module, and a data change evaluation module. Users can obtain data panoramic information through the data panoramic view module, obtain the overall information of the data assets through the data overview report module, evaluate the redundancy degree of the data assets through the data redundancy evaluation module, and evaluate the scientificity and increase or decrease of the data asset changes through the data change evaluation module.

[0030] The beneficial effects achieved by the present invention are as follows:

[0031] By establishing a data warehouse visualization system and a data asset management system, the data asset management system can obtain data information through the data warehouse visualization system, and then perform updates, processing, and visualization, helping users to more conveniently and efficiently master the overall picture of data assets and improving data management efficiency and data modeling efficiency. Among them, the data warehouse visualization system realizes the automatic update and preservation of table relationships through the logical model function, which is beneficial to the unified management of the data warehouse model; through the table management module, it realizes the interaction with the data asset management system, enabling the data asset management system to perform visual display and data management analysis on the updated model files, improving the efficiency of data warehouse management, facilitating the standardization of the data warehouse modeling process, reducing modeling redundancy, realizing an automatic, agile, and comprehensive modeling system, and facilitating users to make decisions and manage related to the data warehouse. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a functional relationship diagram of the data visualization system of the present invention;

[0033] Figure 2 It is a flowchart of the visualization method of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0034] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Apparently, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0035] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs; the terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" and any variations thereof in the specification and claims of this application and the drawings are intended to cover non-exclusive inclusion.

[0036] References to "embodiments" in this specification mean that the specific features, structures, or characteristics described in connection with the embodiments can be included in at least one embodiment of the present application. The phrase "embodiment" appearing at various positions in the specification does not necessarily refer to the same embodiment, nor are they independent or alternative embodiments mutually exclusive of other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0037] Embodiment 1

[0038] As Figure 1 shown, a visualization system based on financial data assets includes a data warehouse visualization system and a data asset management system. The data warehouse visualization system includes a data source configuration module, a logical model processing module, a financial subject domain management module, and a table management module; the data asset management system includes a data lineage map module and a data asset management module.

[0039] Among them, the data source configuration module is responsible for connecting to the logical model processing module, transmitting metadata configuration information, and implementing the configuration of the specified data source; the logical model processing module is responsible for connecting to the financial subject domain management module, using the input metadata information and the data warehouse hierarchy relationship already configured in the data warehouse to establish a logical model, realizing the visualization model design. Data warehouse engineers can perform drag-and-drop design in the custom panel, define the field attributes within the model, support the import or export of models from other enterprise modeling tools (such as Huawei Cloud, etc.), and finally transmit information related to the generation and update of the logical model.

[0040] The financial subject domain module interacts with the data asset management system. This module is responsible for generating a physical model and the corresponding data warehouse execution script, and matching the generated physical model with the corresponding data warehouse subject domain; the table management module performs version management on the content of the physical model that needs to be updated, maintains the content of the data warehouse physical model table, realizes seamless update, transmits information related to the update of the physical model to the data asset management system, and interacts with the data lineage map module and the data asset management module; the data lineage map module is provided with a data asset intelligent search module, a directory viewing module, and a table structure query module.

[0041] The upstream and downstream table information corresponding to the target physical model can be obtained through the intelligent search module, the hierarchical or subject domain table structure relationship of the data warehouse can be obtained through the directory viewing module, and the specific information of the searched target physical model can be obtained through the table structure query module; the data asset management module is provided with a data panoramic view module, a data overview report module, a data redundancy evaluation module, and a data change evaluation module.

[0042] Data panoramic information can be obtained through the data panoramic view module, the overall information of data assets can be obtained through the data overview report module, the redundancy degree of data assets can be evaluated through the data redundancy evaluation module, and the scientific nature and increase or decrease of data asset changes can be evaluated through the data change evaluation module, so as to update and maintain the indicators in the data asset management system.

[0043] Example 2

[0044] As Figure 2 shown, a processing step of a visualization system based on financial data assets is as follows:

[0045] Step 1: The data warehouse engineer combines the requirements characteristics of the financial industry and the characteristics of financial data to conduct overall planning and configuration item setting for the data warehouse, formulates general rules for data warehouse model design, and configures the set rules in the data warehouse. The general rules need to cover themes such as parties, products, agreements, events, assets, finance, institutions, marketing, channels, etc.;

[0046] Step 2: The data warehouse engineer sorts out the metadata according to different business requirements, transfers the metadata to the logical model processing module, and constructs a logical model;

[0047] Step 3: The data warehouse engineer uses a visualization modeling tool for custom development and design, sets the corresponding field content, and creates the relationship between the logical model and other data models in the database table according to the internal and external primary keys of the database table;

[0048] Step 4: According to the data warehouse theme domain rules formulated by the data warehouse engineer, the generated logical model is automatically assigned to the corresponding theme domain in a matching manner. This process should support custom modification, and for special cases, manual assignment to the corresponding data warehouse theme domain is supported;

[0049] Step 5: In the physical model processing interface, generate and execute the script of the executable physical model, and finally construct the physical model. After the physical model is constructed, generate the mapping relationship between the metadata and the existing physical model of the database, generate the corresponding version number, provide a basis for database change management, and facilitate version tracing in the future. For the database tables that need to be updated, perform a seamless update, and the system automatically backs up the database tables before the update;

[0050] Step 6: Update the records in the table management module according to the database tables that need to be updated in Step 5, and at the same time update and record the current latest inter-table dependency relationship. Specifically, the content of the update record by the table management module should include: the basic attributes of the table fields in the physical model, the content of the table fields, the rules of the table fields, and the update rules and save rules of the table.

[0051] Step 7: The data lineage map module receives the table structure update information in Step 6, updates the lineage map according to the database table logical relationships recorded therein, so as to display the data lineage of the data warehouse in the whole link. The data lineage map module shall support real-time update and second-level query functions, so that designers can make more appropriate data solutions based on the latest information during front-end design. The query function shall include a data asset intelligent search module, a directory access module, and a table structure query module. Users can obtain the upstream and downstream table information corresponding to the target physical model through the intelligent search module, obtain the hierarchical or subject domain table structure relationship of the data warehouse through the directory access module, and obtain the specific information of the searched target physical model through the table structure query module. The specific lineage units in the data lineage map module shall include: ETL lineage, table-to-table lineage, and field lineage.

[0052] Step 8: The data asset management module receives the table structure update information in Step 6 and updates the corresponding data panoramic view module, data overview report module, data redundancy assessment module, and data change assessment module. The data panoramic view module and the data overview report module visually display the updated information, and form a monitoring system for data quality in the data redundancy assessment module and the data change assessment module.

[0053] The embodiments of the present invention described above do not constitute a limitation on the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the claims of the present invention.

Claims

1. A visualization management method based on financial data assets, characterized in that: It includes the following steps: S1. The data warehouse engineer accesses the data source configuration module in the data platform, sorts out the metadata according to business requirements, and transmits it to the logical model processing module; S2. In the logical model processing module, the data warehouse layering and theme division rules need to be set for processing the metadata; S3. The logical model makes rule judgments based on the configured rules and the metadata input by the data source configuration module, and inputs the logical model results into the financial theme domain module; S4. The financial theme domain module integrates the input results of the logical model and transmits the complete physical model of the financial data warehouse to the table management module; S5. The table management module automatically updates and synchronizes the results of the physical model, and transmits the model information to the data asset management system; S6. The data lineage map module integrates the physical model update information, integrates the relationships of each physical model in the map, and updates the information in the data asset management system; S7. The data asset management module updates the indicators and descriptions related to the data asset dimension based on the data of the lineage map module; S8. According to the physical model update data obtained in steps S6 to S7, the data asset management system performs corresponding visualization processing and display operations.

2. The visualization management method based on financial data assets according to claim 1, characterized in that: The specific process of step S1 is that the data warehouse engineer configures the data source according to the common data structures and output forms in the financial industry, and plans the data warehouse configuration plan in the financial sub-industry.

3. A visualization management method based on financial data assets according to claim 1, characterized in that: The specific process of step S2 is that the data warehouse engineer makes an overall plan for the data warehouse according to the common layering and theme domain division rules in the financial industry, and sets the flowchart for the logical model to process the metadata.

4. A visualization management method based on financial data assets according to claim 1, characterized in that: The specific process of step S3 includes the following steps: S3.

1. The data warehouse engineer uses a visual modeling tool in the logical processing module and designs it using a drag-and-drop method; S3.

2. Extract the metadata required for the database tables from the data platform; S3.

3. The design area can automatically identify the mapping relationships between the metadata, automatically update the relationships between the metadata through version control, and establish the logical relationships between the database tables according to the primary key relationships. S3.

4. Allow design through custom logic, create model entities and attributes, and establish connections with other entities in the data warehouse.

5. A visualization management method based on financial data assets according to claim 1, characterized in that: The specific process of step S4 includes the following steps: S4.

1. The data warehouse engineer classifies the logical model into the corresponding theme domain in the financial theme domain module, and obtains the physical model after executing the code script in each theme domain module; S4.

2. Form the mapping relationship between the metadata and the database tables, generate the corresponding version number, and provide a basis for automatically updating the entity logical relationships of the model; S4.

3. The data platform automatically updates the metadata according to the version number, records the update history and the corresponding version; at the same time, generates a backup table of the previous version of the metadata, which does not affect the normal use of database queries, so as to realize the automatic inspection and update of the system metadata without affecting downstream use, and this type of update supports the corresponding synchronization and recovery functions.

6. A visualization management method based on financial data assets according to claim 1, characterized in that: The specific process of step S5 is that the table management module updates the record content, including: the basic attributes of the table fields in the physical model, the content of the table fields, the rules of the table fields, as well as the update rules and save rules of the table. The table management module interacts the metadata update information with the data lineage map module and the asset management module. The data lineage map module sorts out the metadata update information and updates the visualized lineage analysis view displayed on the front end. The data asset management sorts out the metadata update information and updates the data report displayed on the front end.

7. A visualization management method based on financial data assets according to claim 1, characterized in that: Step S6 should specifically include a data asset intelligent search module, a directory access module, and a table structure query module. Users can obtain the upstream and downstream table information corresponding to the target physical model through the intelligent search module, obtain the hierarchical or subject domain table structure relationship of the data warehouse through the directory access module, and obtain the specific information of the searched target physical model through the table structure query module.

8. A visualization management method based on financial data assets according to claim 7, characterized in that: The data lineage map module in step S6 supports real-time update and second-level query functions, so that designers can make more appropriate data solutions based on the latest information during front-end design. The specific lineage units in the data lineage map module should include: ETL lineage, table-table lineage, and field lineage.

9. A visualization management method based on financial data assets as described in claim 1, characterized in that: Step S7 should specifically include a data panoramic view module, a data overview report module, a data redundancy assessment module, and a data change assessment module. Users can obtain data panoramic information through the data panoramic view module, obtain the overall information of the data assets through the data overview report module, evaluate the redundancy degree of the data assets through the data redundancy assessment module, and evaluate the scientificity and increase or decrease of the data asset changes through the data change assessment module.

Images