Metadata-based Data Asset Management Method and Its Application System

Abstract

The present invention discloses a data asset management method and its application system based on metadata. The data asset management method includes accessing a preset database to obtain data asset information and corresponding metadata, and storing the metadata in a local database in a preset meta-model; in response to an operation request, obtaining the metadata corresponding to the operation request from the local database, and processing the metadata according to the content of the operation request, so as to output a processing result in a form recognizable by various users. The present invention can implement data analysis according to business requirements and output it in a form recognizable by various users, enhancing the readability of data, helping to lower the usage threshold for business personnel, facilitating business personnel to find the data they want at any time, greatly improving work efficiency, and moreover, when the required data cannot be obtained, it can be further implemented by combining an operation addition function, which is more user-friendly.

Description

Technical Field

[0001] The present invention relates to the field of data management, and particularly to a method for managing data assets based on metadata and its application system. Background Art

[0002] Traditional database management tools have a certain degree of professionalism and usage thresholds. Many business personnel in small and medium-sized banks do not have strong professional technical capabilities. Traditional database management tools can only display the SQL statements and execution results of stored procedures. Since business personnel in small and medium-sized banks do not have excellent database professional capabilities, their ability to write SQL statements for querying required data is limited. In the data traceability process, it is inconvenient to locate the data source through the results. Therefore, there is a certain degree of difficulty in using these tools for business personnel. After many small and medium-sized banks have established data platforms, due to the weak professional technical capabilities of many business personnel, and the display effects of many third-party database management tools have a certain degree of generality, without the function of classifying and displaying various types of data according to the in-house business requirements, and all have certain usage thresholds. Facing a large number of tables, views, stored procedures, etc. in the data warehouse, it is very difficult for business personnel to conveniently and quickly find the data they want.

[0003] Currently, although existing third-party database management tools are all very powerful, they have a certain degree of professionalism and usage thresholds, which are difficult for non-professional personnel to get started with and inconvenient to use. It is not visible what data there is and where the data is. It is very difficult to have further expansion for the business needs of users. Moreover, at the result display level, existing third-party tools only display query results in a straightforward table form, such as sorting, grouping statistics, etc., which is not intuitive enough. Summary of the Invention

[0004] The object of the present invention is to provide a method for managing data assets based on metadata and its application system that can intuitively display data asset information.

[0005] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] A method for managing data assets based on metadata, the data asset management method includes: accessing a preset database to obtain data asset information and corresponding metadata, and storing the metadata in a local database in a preset meta-model; in response to an operation request, obtaining metadata corresponding to the operation request from the local database, and processing the metadata according to the content of the operation request, so as to output a processing result in a form recognizable by various users.

[0007] Further, the operation request includes querying and / or classifying and / or grouping and / or annotating and / or lineage analysis of metadata.

[0008] Further, the data asset management method further includes: when the operation request is the lineage analysis of metadata, by parsing the processing process, obtaining the temporary data created during the processing process, and obtaining the source data through the temporary data, thereby constructing the lineage relationship according to the metadata, and outputting the lineage relationship in a form recognizable by various users.

[0009] Further, the data asset management method further includes: when the operation request is any one of classification, grouping, and annotation of metadata, storing the output processing result in the local database.

[0010] Further preferably, the data asset management method further includes: when the operation request is the query of metadata, also obtaining the query result by accessing a preset data warehouse, and outputting the query result in a form recognizable by various users.

[0011] Further, the data asset management method further includes: in response to an operation addition request, obtaining the data asset information corresponding to the operation addition request, and outputting the data asset information in a form recognizable by various users.

[0012] Preferably, the data asset information includes views and / or indexes and / or table fields and / or stored procedures and / or reports and / or data metrics.

[0013] Preferably, the form recognizable by various users includes metrics and / or reports and / or graphical forms.

[0014] An application system based on the data asset management method described above, the application system includes a control unit, a data unit, and a processing unit, wherein the data unit includes a data input unit and a data output unit, the data input unit is configured to load external data into the application system, and the data output unit is configured to send the data in the application system to the outside;

[0015] The control unit is electrically connected to the data unit and the processing unit respectively. When the application system receives an operation request, the control unit processes it by controlling the processing unit, and outputs the processing result through the data output unit.

[0016] Further, the operation request includes query and / or classification and / or grouping and / or annotation and / or lineage analysis of data.

[0017] Advantages of the present invention: It can perform data analysis according to business requirements and output in a form recognizable by various users, which helps to lower the usage threshold for business personnel, facilitates business personnel to find the data they want at any time, greatly improves work efficiency, and moreover, when the required data cannot be obtained, it can be further implemented by combining with an operation addition function, making it more user-friendly. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0019] Figure 1 It is a schematic flowchart of the data asset management method provided by an embodiment of the present invention;

[0020] Figure 2 It is a schematic flowchart of the lineage analysis in the data asset management method provided by an embodiment of the present invention;

[0021] Figure 3 It is a schematic diagram of the lineage analysis in the application system provided by an embodiment of the present invention;

[0022] Figure 4 It is a schematic diagram of the data application in the application system provided by an embodiment of the present invention;

[0023] Figure 5 It is a schematic diagram of the data report in the application system provided by an embodiment of the present invention;

[0024] Figure 6 It is a schematic diagram of the automatic analysis in the application system provided by an embodiment of the present invention;

[0025] Figure 7 It is a schematic diagram of the indicator list in the application system provided by an embodiment of the present invention;

[0026] Figure 8 It is a schematic diagram of the indicator-based query in the application system provided by an embodiment of the present invention;

[0027] Figure 9 It is a first schematic diagram of the report-based query in the application system provided by an embodiment of the present invention;

[0028] Figure 10 It is a second schematic diagram of the report-based query in the application system provided by an embodiment of the present invention;

[0029] Figure 11It is the third schematic diagram based on report query in the application system provided by the embodiments of the present invention. Detailed implementation manners

[0030] In order to enable those skilled in the art to better understand the solution of the present invention, and to more clearly understand the purpose, technical solution and its advantages of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be noted that the implementation manners not illustrated or described in the drawings are forms known to those of ordinary skill in the art. In addition, although this document may provide examples of parameters including specific values, it should be understood that the parameters do not necessarily have to be exactly equal to the corresponding values, but may approximate the corresponding values within an acceptable tolerance or design constraint. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the scope of protection of the present invention. In addition, the terms "including" and "having" and any variations thereof in the description and claims of the present invention are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0031] The present invention can not only provide further page visualization operations for tables, views, indexes, table fields, etc., generate corresponding SQL statements according to the tables, fields, operators, table association fields, etc. selected by the user on the page for execution, and query the corresponding results; but also directly display business data in the form of metrics, reports, and data visualization according to the needs of business personnel; in addition, it can also avoid the trouble that non-technical personnel cannot write SQL by customizing data asset classification for business personnel, such as data metrics, data visualization, data reports and other data assets, and display the results in a form recognizable by various users, such as graphical and business-oriented (bar charts, pie charts, line charts, etc.). Compared with the display of table results of traditional third-party database management tools, it is more intuitive and easier for ordinary business personnel to understand and use.

[0032] In an embodiment of the present invention, a data asset management method based on metadata is provided. This data asset management method includes: accessing a preset database to obtain data asset information and corresponding metadata, and storing the metadata in a local database in a preset meta-model; in response to an operation request, obtaining the metadata corresponding to the operation request from the local database, and processing the metadata according to the content of the operation request, so as to output the processing result in a form recognizable by various users. For the specific process, please refer to Figure 1Among them, the data asset information includes, but is not limited to, one or more of views and / or indexes and / or table fields and / or stored procedures and / or reports and / or data metrics. The operation requests include, but are not limited to, one or more of querying and / or classifying and / or grouping and / or annotating and / or lineage analysis of metadata. The forms recognizable by various users include, but are not limited to, one or more of metrics and / or reports and / or graphical forms. The protection scope of the present invention is not limited thereto.

[0033] In this embodiment, the collection and subsequent processing work of metadata, that is, the processing process, specifically includes the following processes:

[0034] (1) By accessing a specified database, database tables, views, indexes, table fields, stored procedures, etc. are obtained. As Figure 1 shown, the specified database can also obtain information from a data warehouse;

[0035] (2) Parse the obtained information and save it as a meta-model in the local database;

[0036] (3) Through the system page, for the business, label, mark, group, and classify the metadata;

[0037] (4) Through the page, query business keywords, labels, etc. to search and query the metadata;

[0038] (5) According to the queried metadata, operate query conditions, table field associations, sorting methods, etc. on the page, and the background generates a query sql statement, which is executed in the specified database to obtain the query result;

[0039] (6) According to the page configuration requirements, perform operations such as graphical (pie chart, bar chart, line chart, etc.) display on the query result that can be recognized by various users.

[0040] When the operation request is the lineage analysis of metadata, by parsing the data processing process, the temporary data created during the processing process is obtained, and the source data is obtained through the temporary data. Thus, the lineage relationship is constructed based on the metadata and the lineage relationship is output in a form recognizable by various users. For the specific process, see Figure 2 In this embodiment, the lineage analysis of data specifically includes the following processes:

[0041] (1) Parse the stored procedures in the database;

[0042] (2) Parse the intermediate construction of temporary tables and the derivative part of the sql statement (the insert into select part);

[0043] (3) Parse the insert derivative target table statement in the sql statement to obtain the target table;

[0044] (4) Analyze the SELECT part in the SQL statement, which is a hierarchical recursive process. The SELECT part can be a direct table query to obtain the source data table, or a nested subquery. Analyze all the nested subqueries in the SELECT part to obtain the source data tables therein.

[0045] (5) The SELECT part uses the temporary tables constructed in the stored procedure. When filtering the source data tables, it is necessary to judge the temporary tables to obtain the real source data tables.

[0046] (6) Construct the lineage relationship based on the target table and the source data tables.

[0047] When the operation request is any one of metadata classification, grouping, and annotation, store the output processing result in the local database.

[0048] When the operation request is metadata query, in addition to querying through accessing the local database, also obtain the query result by accessing a preset data warehouse, and output the query result in a form recognizable by various users.

[0049] In addition, the data asset management method further includes: in response to an operation addition request, obtain the data asset information corresponding to the operation addition request, and output the data asset information in a form recognizable by various users.

[0050] In an embodiment of the present invention, an application system based on the above-mentioned data asset management method is provided. This application system includes a control unit, a data unit, and a processing unit. Among them, the data unit includes a data input unit and a data output unit. The data input unit is configured to load external data into the application system, and the data output unit is configured to send the data in the application system to the outside; the control unit is electrically connected to the data unit and the processing unit respectively. When the application system receives an operation request, the control unit processes it by controlling the processing unit, and outputs the processing result through the data output unit. Among them, the operation request includes but is not limited to one or more of data query and / or classification and / or grouping and / or annotation and / or lineage analysis, and the protection scope of the present invention is not limited thereto.

[0051] This application system processes the basic elements of the database (tables, fields, views, etc.), operations, etc. accordingly to enhance the readability for non-technical personnel and reduce the difficulty of using the database for non-technical business personnel. At the same time, it can redefine the data through data standards, metrics, reports, data visualization, etc., and display the data to business personnel in a more intuitive way. During the process of page operation through the data asset portal, by checking the table names, fields, associated fields, operators, etc. required by the business, the background generates SQL statements for querying, and the results are displayed in a way that can be recognized by various users (pie charts, bar charts, line charts, etc.), which is more intuitive than the table display of the SQL statement query results of traditional tools and is easier for business personnel to understand.

[0052] This application system collects database-related information (tables, views, indexes, table fields, stored procedures, etc.), combines with a custom metadata model to build a local data asset catalog; moreover, combined with the characteristics of small and medium-sized banks, it defines an index meta-model and a report meta-model to provide convenient data services for business personnel; furthermore, it can also maintain and expand the meta-model and metadata classification, grouping, tagging, etc. for business personnel to query, search, display, etc.; at the same time, it realizes data lineage analysis by parsing the data warehouse stored procedure to obtain the tables used in the data processing process; in addition, it also uses metadata to build custom query metrics and displays the query results of data in a more intuitive way through various business-recognizable methods.

[0053] Specifically, as Figure 3 shown, through the lineage analysis function of this application system, it can be seen that there are Table 2, Table 3, and Table 4 downstream of Table 1; as Figure 4 shown, through the data application function of this application system, when business personnel cannot obtain the required data assets, they can use the data application function to connect with R & D personnel, put forward data requirements, and obtain the relevant data assets they need. After receiving the data application proposed by business personnel, R & D personnel can conduct an inventory query through the existing data assets. If there are no relevant data assets, they will develop and go live the applied data assets and then submit them to business personnel for query, which is more user-friendly.

[0054] This application system can also define data assets through the meta-model. When business personnel view relevant data, they can search and query the required data assets through reports, BI, metrics, and data visualization. Specifically, the query and usage situation of reports by business personnel is the fixed report function of this application system as Figure 5 shown, the query and usage situation of BI by business personnel is the automatic analysis function of this application system as Figure 6 shown, the viewing of metrics by business personnel, that is, the situation of the metric list function of this application system is as Figure 7 shown, the situation of business personnel querying data based on metrics is asFigure 8 as shown

[0055] This application system can also provide technicians with data asset - report knowledge query to analyze whether existing data assets can be reused. The specific situation is as Figure 9 shown. Further, technicians obtain relevant processing calibers. The specific situation is as Figure 10 shown. Further, technicians obtain the hierarchical classification of fields to obtain relevant query permissions and desensitization information. The specific situation is as Figure 11 shown

[0056] In an embodiment of the present invention, the application system displays the data query results graphically. The specific process is as follows:

[0057] (1) The user queries and searches for business - related metadata (tables, fields, etc.) through the page;

[0058] (2) According to the obtained metadata, the user adds operations such as query conditions, display fields, and inter - table management on the page. The application system automatically generates the query sql statement and the query result;

[0059] (3) The application system visually displays the query results in the form of statistical charts (pie charts, bar charts, etc.).

[0060] In addition, it should be noted that for the metadata part, the maintenance of metadata can also be achieved through manual entry and manual update, which does not limit the protection scope of the present invention; for the lineage analysis of data, the lineage relationship of table data can also be manually maintained and added when adding stored procedures, which also does not limit the protection scope of the present invention.

[0061] The idea of the embodiment of this application system is the same as the working process of the data asset management method in the above - mentioned embodiment. By way of full - text reference, all the content of the embodiment of the above - mentioned data asset management method is incorporated into the embodiment of this application system, and will not be elaborated here.

[0062] The present invention assembles various types of information such as data warehouse tables, views, indexes, table fields, and stored procedures into various meta - models, combines certain business requirements, customizes meta - models for database tables, fields, views, indexes, stored procedures, etc., and then performs sorting, induction, adding custom tags and other sorting operations, and defines them in the form of indicators, reports, and data visualization, which is convenient for business personnel to search and find, reduces the usage threshold of business personnel, improves efficiency, and can solve the problem that current business personnel in small and medium - sized banks have weak professional technology and limited ability to write sql statements for querying required data.

[0063] In addition, after the data warehouse is established, the data in it will all be reprocessed through stored procedures. Traditional database management tools can only display the SQL statements and execution results of stored procedures. Since the business personnel in small and medium-sized banks do not have excellent database professional capabilities, it is inconvenient to locate the data source through the results in the data traceability process. However, through the analysis of data lineage and intuitive display in a way that can be recognized by various users, the present invention facilitates operations such as business personnel's understanding of the data processing process and data quality traceability.

[0064] The above are only the preferred embodiments of the present invention, and do not limit its patent scope accordingly. Any equivalent structural or equivalent process transformation made by using the content of the specification and drawings of the present invention, directly or indirectly applied in other related technical fields, shall be similarly included in the patent protection scope of the present invention.

Claims

1. A data asset management method based on metadata, characterized in that The described data asset management method includes: accessing a preset database to obtain data asset information and corresponding metadata, and storing the metadata in a local database in a preset meta-model; in response to an operation request, obtaining the metadata corresponding to the operation request from the local database, and processing the metadata according to the content of the operation request, including: according to the retrieved metadata, the user adds query conditions, display fields, and inter-table management operations on the page, and the system automatically generates a query SQL statement to obtain data asset information corresponding to the added operations, so as to output the data asset information in a form recognizable by various users, such as metrics and / or reports and / or graphical forms; When the operation request is the lineage analysis of metadata, by parsing the processing process of the metadata, temporary data created during the processing process is obtained, and source data is obtained through the temporary data, so as to construct a lineage relationship according to the metadata, and output the lineage relationship in a form recognizable by various users, where the lineage relationship is constructed through the following steps: (1) Parse the stored procedures in the database; (2) Parse the intermediate construction of temporary tables and the insert into select part of the SQL statement; (3) Parse the insert derivative target table statement in the SQL statement to obtain the target table; (4) Parse the hierarchical recursive process in the select part of the SQL statement, where the select part is a direct table query to obtain the source data table or parse all nested subqueries in the select part to obtain the source data tables therein; (5) When screening the source data table, determine whether the table used in the select part is a temporary table to obtain the real source data table; (6) Construct the lineage relationship of the metadata according to the target table and the source data table; If the user cannot obtain the required data assets, use the data application function to connect with R & D personnel to put forward data requirements; after receiving the data application, the R & D personnel conduct an inventory query through the existing data assets. If there are no relevant data assets, the applied data assets are developed and launched and then submitted to the user for query.

2. The data asset management method according to claim 1, wherein The operation request further includes querying and / or classifying and / or grouping and / or annotating the metadata.

3. The data asset management method according to claim 2, characterized in that The data asset management method further includes: when the operation request is any one of classifying, grouping, and annotating the metadata, storing the output processing result in the local database.

4. The data asset management method according to claim 2, wherein The data asset management method further includes: when the operation request is querying the metadata, also accessing a preset data warehouse to obtain the query result, and outputting the query result in the form recognizable by various users.

5. The data asset management method according to claim 1, characterized in that The data asset information includes views and / or indexes and / or table fields and / or stored procedures and / or reports and / or data metrics.

6. An application system for the data asset management method according to claim 1, characterized in that, The application system includes a control unit, a data unit, and a processing unit. Among them, the data unit includes a data input unit and a data output unit. The data input unit is configured to load external data into the application system, and the data output unit is configured to send the data in the application system to the outside; The control unit is electrically connected to the data unit and the processing unit respectively. When the application system receives an operation request, the control unit processes it by controlling the processing unit and outputs the processing result through the data output unit.

7. The application system according to claim 6, wherein The operation request includes querying and / or classifying and / or grouping and / or annotating of data.

Images