Asset metadata acquisition method and device, electronic equipment and computer storage medium
By creating and processing the meta-information table of data assets in the metadatabase and writing it to the asset library and search engine index, the problem of inconsistent data asset acquisition cost and efficiency is solved, and efficient and accurate data acquisition and storage.
Patent Information
- Application Number
- CN202311038676.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-16
- Publication Date
- 2025-02-21
AI Technical Summary
In the prior art, the cost of data asset acquisition is inconsistent with efficiency, mainly due to the high operation and maintenance costs of canal components, internationalization problems and inefficient network transmission efficiency.
By creating the basic meta information table and business meta information table of the data assets from the metadatabase, obtaining the basic table and attribute table, writing database statements to write meta information in the asset database, generating foreign key information, and transferring meta information through the target connection Merge and write to the target search engine index.
It reduces deployment costs, improves the efficiency of asset metadata acquisition, realizes read and write separation, reduces data errors, leaks and duplication abnormalities, improves the accuracy of acquisition, and saves computing resources.
Smart Images
Figure CN119493807A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of big data technology, and specifically to an asset metadata collection method, device, electronic device and computer storage medium. Background Art
[0002] Currently, when the big data platform asset module collects data warehouse metadata, it consumes the binlog logs of the target data warehouse (HMS, Hive Meta Store) through the consumption component (such as the canal component), and then writes it to ES (Elastic Search, search engine), and then displays the collection on the page. With the development of business, more and more asset metadata has been stored in ES.
[0003] However, the canal component is an independent server-side program that needs to interact with the MySQL server to capture incremental data and then transmit it to consumers over the network. The canal component needs to be deployed separately, which has high operation and maintenance costs. It also faces problems in applying for machine resources when internationalizing and delivering products to the outside world. In addition, the canal component sends events to consumers through network transmission, which has low transmission efficiency, which ultimately leads to the problem that the cost of data asset collection is inconsistent with the efficiency. Summary of the invention
[0004] The embodiments of the present application provide an asset metadata collection method, device, electronic device and computer storage medium, which can solve the technical problem that the cost and efficiency of data asset collection are inconsistent.
[0005] The present application embodiment provides an asset metadata collection method, including:
[0006] Creating a first basic meta information table and a first business meta information table of data assets from the meta database;
[0007] Acquire a first-type basic table and a second-type attribute table, write a target database statement for introducing the first-type basic table, and write the first-type basic table into the first basic meta information table in the asset library based on the target database statement to obtain a second basic meta information table;
[0008] Performing format declaration on the second type attribute table to obtain a target attribute table, generating foreign key information based on the first basic meta information table and the target attribute table, and writing the foreign key information into the first business meta information table in the asset library to obtain a second business meta information table;
[0009] Obtain a total database statement of the second basic meta information table and the second business meta information table, create a target connection, and merge the first basic meta information table and the first business meta information table into an index of a target search engine through the target connection and the total database statement.
[0010] Further, the obtaining of the first type basic table and the second type attribute table includes:
[0011] Accessing a first type processing table and a second type processing table of a target data warehouse, and writing the first type processing table and the second type processing table into a distributed message system;
[0012] The service topics corresponding to the first type processing table and the second type processing table in the distributed message system are obtained, and the service topics are introduced into the data asset center respectively to form the first type basic table and the second type attribute table.
[0013] Furthermore, the step of writing the first type basic table into the first basic meta information table in the asset library based on the target database statement to obtain a second basic meta information table includes:
[0014] Determine whether there is a multi-cluster service. When it is determined that there is no multi-cluster service, write the first type of basic table into the first basic meta information table in the asset library through a preset introduction method based on the target database statement to obtain the second basic meta information table;
[0015] When it is determined to be a multi-cluster service, the cluster name is obtained, and the cluster name is added to the first write statement corresponding to the target database statement to obtain the second write statement, and the first type of basic table is written to the first basic meta information table in the asset library through a preset introduction method and the second write statement to obtain the second basic meta information table.
[0016] Further, the format declaration of the second type attribute table is performed to obtain a target attribute table, foreign key information is generated based on the first basic meta information table and the target attribute table, and the foreign key information is written into the first business meta information table in the asset library to obtain a second business meta information table, including:
[0017] Acquire a target format, and perform format declaration on the second type attribute table according to the target format to obtain the target attribute table;
[0018] The first basic meta information table is used as an external dimension table, the foreign key information is generated according to the association between the external dimension table and the target attribute table, and the foreign key information is written into the first business meta information table in the asset library to obtain the second business meta information table.
[0019] Further, after writing the foreign key information into the first business meta information table to obtain the second business meta information table, the method further includes:
[0020] The second basic meta information table and the second business meta information table are soft deleted, row-to-column transferred and authorized according to the data table control logic statement, and the data table control logic statement is the data table control logic statement of the second basic meta information table and the second business meta information table.
[0021] Further, the data table control logic statement includes a first data table control logic statement and a second data table control logic statement, and before the second basic meta information table and the second business meta information table are soft-deleted, row-to-column transferred, and authorized according to the data table control logic statement, the following steps are included:
[0022] Compile the logic functions of the first field business meta information wide table logic, the first table business information wide table logic and the first library business meta information wide table logic corresponding to the second basic meta information table to obtain the first data table control logic statement corresponding to the second basic meta information table;
[0023] Write the logical functions of the second field business meta information wide table logic, the second table business information wide table logic and the second library business meta information wide table logic corresponding to the second business meta information table to obtain the second data table control logic statement corresponding to the second business meta information table.
[0024] Furthermore, after merging the first basic meta information table and the first business meta information table into the index of the target search engine through the target connection and the total database statement, the method further includes:
[0025] Determining whether a modification parameter sent by an external interface is received, and when it is determined that the modification parameter exists, creating a service interface;
[0026] Based on the service interface, the first service meta information table is modified through a transaction modification method and the modification parameters.
[0027] Accordingly, an embodiment of the present application provides an asset metadata collection device, comprising:
[0028] A creation module, used to create a first basic meta information table and a first business meta information table of a data asset from a meta database;
[0029] A writing module, used for obtaining a first type basic table and a second type attribute table, writing a target database statement for introducing the first type basic table, and writing the first type basic table into the first basic meta information table in the asset library based on the target database statement to obtain a second basic meta information table;
[0030] a declaration module, configured to perform format declaration on the second type attribute table to obtain a target attribute table, generate foreign key information based on the first basic meta information table and the target attribute table, write the foreign key information into the first business meta information table in the asset library, and obtain a second business meta information table;
[0031] A writing module is used to obtain the total database statement of the second basic meta information table and the second business meta information table, and create a target connection, and merge the first basic meta information table and the first business meta information table into the index of the target search engine through the target connection and the total database statement.
[0032] In addition, an embodiment of the present application also provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and the processor is used to run the computer program in the memory to implement the asset metadata collection method provided in the embodiment of the present application.
[0033] In addition, an embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. The computer program is suitable for loading by a processor to execute any asset metadata collection method provided in the embodiment of the present application.
[0034] In addition, an embodiment of the present application also provides a computer program product, including a computer program, and when the computer program is executed by a processor, any one of the asset metadata collection methods provided in the embodiments of the present application is implemented.
[0035] In the embodiment of the present application, a first basic meta information table and a first business meta information table of a data asset are created from a meta database; then, a first type base table and a second type attribute table are obtained, a target database statement for introducing the first type base table is written, and based on the target database statement, the first type base table is written to the first basic meta information table in the asset library to obtain the second basic meta information table, thereby achieving efficient writing of the first type base table to the first basic meta information table in the asset library; a format declaration is performed on the second type attribute table to obtain a target attribute table, foreign key information is generated based on the first basic meta information table and the target attribute table, and the foreign key information is written to the first business meta information table, obtain the second business meta information table, and realize the efficient writing of the second type attribute table in the first business meta information table in the asset library; finally, obtain the total database statement of the second basic meta information table and the second business meta information table, and create a target connection, and merge the first basic meta information table and the first business meta information table into the index of the target search engine through the target connection and the total database statement, thereby reducing the deployment cost, improving the efficiency of asset metadata collection, and realizing the separation of reading and writing, reducing the probability of abnormal situations such as asset metadata errors, omissions and duplications, further improving the accuracy of asset metadata collection, and saving computing resources. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0037] Figure 1 It is a flowchart of the asset metadata collection method provided in the embodiment of the present application;
[0038] Figure 2 It is a structural diagram of an asset metadata collection device provided in an embodiment of the present application;
[0039] Figure 3 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0040] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.
[0041] The embodiment of the present application provides an asset metadata collection method, device, electronic device and computer storage medium. The asset metadata collection device can be integrated in an electronic device, which can be a server or a terminal.
[0042] Among them, the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, network acceleration services (Content Delivery Network, CDN), as well as big data and artificial intelligence platforms.
[0043] The terminal may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, and this application does not limit this.
[0044] In addition, the term "plurality" in the embodiments of the present application refers to two or more than two. The terms "first" and "second" in the embodiments of the present application are used to distinguish descriptions and should not be understood as implying relative importance.
[0045] It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments.
[0046] See also Figure 1 , Figure 1 : is a flowchart of an asset metadata collection method provided by an embodiment of the present application. The asset metadata collection method may include:
[0047] S101: Create a first basic meta information table and a first business meta information table of a data asset from a meta database.
[0048] In this embodiment, data assets mainly include two types of library, table, field and partition meta-information data, which can be divided into basic type and business type, and the meta-information under these two types can be divided into basic meta-information and business meta-information. Specifically, basic meta-information includes information data such as table name, creation time, creator, table type, table storage location, field type and field name; business meta-information includes data such as table administrator, table partition type, table level, table directory, field security level and field standard. Among them, the basic meta-information comes from the tbls table of the hive meta-database of MySQL, and the business meta-information comes from the table_params table of the hive meta-database of MySQL. The MySQL wide table corresponding to the basic meta-information and the business meta-information is associated through a joint primary key, which is composed of a cluster, a library name and a table name; when processing the real-time stream of field meta-information, it is necessary to obtain the joint primary key, and when obtaining the joint primary key, it is necessary to return to the table for reverse query. By declaring a dimension table of basic metadata of a table in Flink SQL, this dimension table is an external dimension table. Through this external dimension table, the joint primary key can be obtained, thereby collecting the basic metadata and business metadata of the data assets.
[0049] Therefore, when collecting data assets, first create the first basic meta information table and the first business meta information table of the data assets from the MySQL meta database (hive database). Specifically, obtain the basic dimension of the basic meta information of the data assets, create a MySQL table of the basic meta information based on the basic dimension, and obtain the first basic meta information table; obtain the business dimension of the business meta information of the data assets, and create a MySQL table of the business meta information based on the business dimension, and obtain the first business meta information table.
[0050] S102: Obtain a first-type basic table and a second-type attribute table, write a target database statement to introduce the first-type basic table, and write the first-type basic table to the first basic meta information table in the asset library based on the target database statement to obtain a second basic meta information table.
[0051] In this embodiment, the first type of basic table is a basic table of a distributed messaging system (such as Kafka); the second type of attribute table is an attribute table of a distributed messaging system. The first type of basic table and the second type of attribute table are obtained, and a target database statement for introducing the first type of basic table is written. Among them, the target database statement is a flink SQL statement, and flink is a distributed processing engine and framework. Through flink, the flink CDC connector message can be directly written into the MySQL asset library. When the target database statement for introducing the first type of basic table is completed, the first type of basic table is directly written into the first basic meta information table in the asset library based on the target database statement, thereby obtaining a second basic meta information table, which is a wide table obtained by improving the first basic meta information table, and the wide table is stored in the MySQL asset library.
[0052] S103: perform format declaration on the second type attribute table to obtain a target attribute table, generate foreign key information based on the first basic meta information table and the target attribute table, write the foreign key information into the first business meta information table in the asset library, and obtain a second business meta information table.
[0053] In this embodiment, the default format of the second type attribute table is the avro format, which is a data format that marks the addition, deletion, and update status corresponding to the event when an update operation occurs in a table in a relational database, thereby generating positive and negative flow messages. In actual business, there is no need for upstream messages to produce negative flow effects. Therefore, when the second type attribute table is obtained, the format of the second type attribute table is declared, and the second type attribute table is declared as a target attribute table in a target format. Then, foreign key information is generated based on the first basic meta information table and the target attribute table, and the foreign key information is written to the first business meta information table in the asset library to obtain the second business meta information table, wherein the second business meta information table is a wide table obtained by improving the first business meta information table, and the wide table is stored in the MySQL asset library.
[0054] S104. Obtain a total database statement of the second basic meta information table and the second business meta information table, create a target connection, and merge the first basic meta information table and the first business meta information table into an index of a target search engine through the target connection and the total database statement.
[0055] In this embodiment, when the second basic meta information table and the second business meta information table are obtained, a total database statement that merges the second basic meta information table and the second business meta information table is written. The total database statement is a flink SQL statement that merges the second basic meta information table and the second business meta information. The total database statement is obtained, and a target connection (i.e., flink CDC) is created. According to the target connection and the total database statement, the first basic meta information table and the first business meta information table can be merged and written into the index of the target search engine. Among them, the target search engine is ES (Elastic Search, search engine), and the index of the target search engine includes the index of the library, table and field in ES. By merging the first basic meta information table and the first business meta information table and writing them into the index of the target search engine, when the asset metadata is read, it can be read through the target search engine, thereby realizing the read-write separation of the asset metadata.
[0056] This embodiment creates a first basic meta information table and a first business meta information table of a data asset from a meta database; obtains a first type of basic table and a second type of attribute table, writes a target database statement that introduces the first type of basic table, writes the first type of basic table to the first basic meta information table in the asset library based on the target database statement, and obtains the second basic meta information table; performs a format declaration on the second type of attribute table to obtain a target attribute table, generates foreign key information based on the first basic meta information table and the target attribute table, writes the foreign key information to the first business meta information table in the asset library, and obtains the second business meta information table; obtains a total database statement of the second basic meta information table and the second business meta information table, and creates a target connection, and merges the first basic meta information table and the first business meta information table into the index of the target search engine through the target connection and the total database statement, thereby reducing deployment costs, improving the efficiency of asset metadata collection, and realizing read-write separation, reducing the probability of abnormal situations such as asset metadata errors, omissions and duplications, further improving the accuracy of asset metadata collection, and saving computing resources.
[0057] In some embodiments of the present application, the obtaining of the first type basic table and the second type attribute table includes:
[0058] Accessing a first type processing table and a second type processing table of a target data warehouse, and writing the first type processing table and the second type processing table into a distributed message system;
[0059] The service topics corresponding to the first type processing table and the second type processing table in the distributed message system are obtained, and the service topics are introduced into the data asset center respectively to form the first type basic table and the second type attribute table.
[0060] In this embodiment, the first type of processing table is the basic table corresponding to the target data warehouse (HMS, Hive Meta Store), such as tbls, dbs and column_v2; the second type of processing table is the attribute table corresponding to the target data warehouse, such as table_params and database_params. Access the target data warehouse, obtain the first type of processing table and the second type of processing table corresponding to the target data warehouse, and write the first type of processing table and the second type of processing table into the distributed message system, wherein the distributed message system is the kafka message system. Specifically, when the first type of processing table and the second type of processing table are respectively written into the kafka message system, the first type of kafka table and the second type of kafka table, and the first type of kafka table and the second type of kafka table in fafka correspond to the service theme. Therefore, the service theme corresponding to the first type of processing table and the second type of processing table in the distributed message system is obtained, and the service theme is introduced into the data asset center respectively, so as to form the first type of basic table and the second type of attribute table. The first type of basic table and the second type of attribute table correspond to the above-mentioned first type of kafka table and the second type of kafka table, respectively. In addition, when a service theme is introduced into the data asset center and a first-type basic table and a second-type attribute table are formed, management authority can be automatically granted to the table creator so that the table creator can manage the created basic table in a timely manner.
[0061] This embodiment writes the first type processing table and the second type processing table into a distributed message system, and introduces the service topics corresponding to the first type processing table and the second type processing table in the distributed message system into a data asset center to form a first type basic table and a second type attribute table, thereby realizing efficient construction and reference of different types of tables.
[0062] In some embodiments of the present application, the first type basic table is written into the first basic meta information table in the asset library based on the target database statement to obtain the second basic meta information table, including:
[0063] Determine whether there is a multi-cluster service. When it is determined that there is no multi-cluster service, write the first type of basic table into the first basic meta information table in the asset library through a preset introduction method based on the target database statement to obtain the second basic meta information table;
[0064] When it is determined to be a multi-cluster service, the cluster name is obtained, and the cluster name is added to the first write statement corresponding to the target database statement to obtain the second write statement, and the first type of basic table is written to the first basic meta information table in the asset library through a preset introduction method and the second write statement to obtain the second basic meta information table.
[0065] In this embodiment, when the first type of basic table is written into the first basic meta information table according to the target database statement, the number of clusters corresponding to the data asset is obtained, and whether there is a multi-cluster service is determined according to the number of clusters. If the number of clusters is 1, it is determined that there is no multi-cluster service; when it is determined that there is no multi-cluster service, a preset introduction method, such as insert into method, is obtained, and the first type of basic table is written into the first basic meta information table in the asset library through the introduction method and the target database statement to obtain a second basic meta information table.
[0066] If the number of clusters is greater than 1, it is determined that there are multiple cluster services, and the cluster names of all clusters are obtained. All cluster names are added to the first write statement corresponding to the target database statement to obtain a second write statement, wherein the cluster name can be added to the first write statement through an English abbreviation to obtain a second write statement. When the second write statement is obtained, the first type of basic table is written into the first basic meta information table in the asset library through a preset introduction method (such as insert into) according to the second write statement to obtain a second basic meta information table.
[0067] This embodiment determines whether there are multiple cluster services, and writes to the first type of basic table according to different cluster application scenarios, thereby achieving efficient writing of the basic table in the basic metadata table, and further achieves read-write separation of asset metadata through a writing method different from reading, thereby reducing errors and omissions in asset metadata and improving the accuracy of asset metadata collection.
[0068] In some embodiments of the present application, the format declaration of the second type attribute table is performed to obtain a target attribute table, foreign key information is generated based on the first basic meta information table and the target attribute table, and the foreign key information is written into the first business meta information table in the asset library to obtain a second business meta information table, including:
[0069] Acquire a target format, and perform format declaration on the second type attribute table according to the target format to obtain the target attribute table;
[0070] The first basic meta information table is used as an external dimension table, the foreign key information is generated according to the association between the external dimension table and the target attribute table, and the foreign key information is written into the first business meta information table in the asset library to obtain the second business meta information table.
[0071] In this embodiment, in order to avoid the negative flow effect caused by the upstream message, when writing the second type attribute table to the second business meta information table, the target format is obtained, and the format of the second type attribute table is declared according to the target format. Specifically, the target format is a data format that will not produce negative flow effects, such as JSON format. When the target format is obtained, the format of the second type attribute table is declared as the target format to obtain the target attribute table; then, the first basic meta information table is used as an external dimension table, and the external dimension table is associated with the target attribute table to generate foreign key information, and the foreign key information is written to the first business meta information table in the asset library, that is, the second business meta information table is obtained.
[0072] This embodiment declares the second type of attribute table through the target format to obtain the target attribute table. Thereafter, foreign key information is generated based on the association between the target attribute table and the external dimension table. The foreign key information is written to the first business meta information table in the asset library to obtain the second business meta information table, thereby achieving efficient writing of business meta information, avoiding the generation of negative flow effects, and further improving the efficiency of asset metadata collection.
[0073] In some embodiments of the present application, after writing the foreign key information into the first business meta information table to obtain the second business meta information table, the following steps are included:
[0074] The second basic meta information table and the second business meta information table are soft deleted, row-to-column transferred and authorized according to the data table control logic statement, and the data table control logic statement is the data table control logic statement of the second basic meta information table and the second business meta information table.
[0075] In this embodiment, after writing the foreign key information into the first business meta information table to obtain the second business meta information table, the data table control logic statement of the second basic meta information table and the second business meta information table is written, so that the data in the second basic meta information table and the second business meta information table can be logically processed by the data table control logic statement. Specifically, the data table control logic is obtained, and the function statement (such as flowudf) corresponding to the data table control logic is written to obtain the data table control logic statement, and complex logical functions can be realized by the data table control logic statement. Then, according to the data table control logic statement, the second basic meta information table and the second business meta information table are subjected to the logical operations of soft deletion, row to column transfer and authorization. Among them, for the operation of converting rows to columns, the row width table data can be pieced together into multiple rows of key-value format messages through data table control logic statements; for the soft deletion operation, the data position can be marked through data table control logic statements, or it can be moved to the trash can for deletion. The soft deletion is an operation that does not delete the actual data from the data source; for the authorization operation, when the library table meta information is collected into the asset module, the creator is granted relevant administrator permissions (such as ranger permissions) through data table control logic statements.
[0076] This embodiment writes data table control logic statements, and performs soft deletion, row-to-column conversion, and authorization operations on the second basic meta information table and the second business meta information table according to the data table control logic statements, thereby realizing complex logical operations on the table.
[0077] In some embodiments of the present application, the above-mentioned data table control logic statement includes a first data table control logic statement and a second data table control logic statement, and before the second basic meta information table and the second business meta information table are soft-deleted, row-to-column, and authorized according to the data table control logic statement, the following steps are included:
[0078] Compile the logic functions of the first field business meta information wide table logic, the first table business information wide table logic and the first library business meta information wide table logic corresponding to the second basic meta information table to obtain the first data table control logic statement corresponding to the second basic meta information table;
[0079] Write the logical functions of the second field business meta information wide table logic, the second table business information wide table logic and the second library business meta information wide table logic corresponding to the second business meta information table to obtain the second data table control logic statement corresponding to the second business meta information table.
[0080] In this embodiment, the data table control logic includes three types of data logic, namely, field business meta information wide table logic, table business information wide table logic, and library business meta information wide table logic. Among them, according to the logic statement corresponding to the field business meta information wide table logic, the soft delete operation can be implemented; according to the logic statement corresponding to the table business information wide table logic, the table row to column and authorization operations can be implemented; according to the logic statement corresponding to the library business meta information wide table logic, the library row to column operation can be implemented. Write the logic functions corresponding to the first field business meta information wide table logic, the first table business information wide table logic, and the first library business meta information wide table logic corresponding to the second basic meta information table, and the logic function corresponding to each type is the first data table control logic statement corresponding to the second basic meta information table; write the logic functions corresponding to the second field business meta information wide table logic, the second table business information wide table logic, and the second library business meta information wide table logic corresponding to the second business meta information table, that is, obtain the second data table control logic statement corresponding to the second business meta information table.
[0081] This embodiment implements complex logical operations on different types of table data by writing logical functions corresponding to field business meta information wide table logic, table business information wide table logic, and library business meta information wide table logic.
[0082] In some embodiments of the present application, after the first basic meta information table and the first business meta information table are merged and written into the index of the target search engine through the target connection and the total database statement, the method further includes:
[0083] Determining whether a modification parameter sent by an external interface is received, and when it is determined that the modification parameter exists, creating a service interface;
[0084] Based on the service interface, the first service meta information table is modified through a transaction modification method and the modification parameters.
[0085] In this embodiment, the first business meta information table can be modified based on the business interface by creating a business interface. Specifically, it is determined whether the modification parameters sent by the external interface are received, and the modification parameters include modification parameters for three types of business meta information: library, table, and field. If it is determined that the modification parameters are received, a business interface corresponding to the current asset library is created, wherein the external interface is an interface for other businesses, and the business interface is a restful interface corresponding to the asset library; based on the business interface, the first business meta information table is modified through the transaction modification method and the modification parameters. Specifically, a transaction is an indivisible sequence of operations in a database and is also the basic unit of database concurrency control. When the modification parameters sent by the external interface are received, a business interface corresponding to the asset library is created, and the modification parameters are introduced into the asset library based on the business interface; thereafter, a transaction for modifying the parameters is constructed in the asset library, and the transaction is executed to modify the first business meta information table. Further, when the first business meta information table is modified through the transaction modification method, the first business meta information table is locked to prevent other users from modifying the data in the table. In addition, the business interface may also be pre-created in the asset library. When the modification parameters sent by the external interface are received, the modification parameters are directly introduced through the business interface to modify the first business meta information table.
[0086] This embodiment modifies the first service meta information table based on the service interface through a transaction modification method and modification parameters, thereby achieving accurate and efficient modification of the service meta information table.
[0087] In order to facilitate better implementation of the asset metadata collection method provided in the embodiment of the present application, the embodiment of the present application also provides a device based on the above asset metadata collection method. The meanings of the terms are the same as those in the above asset metadata collection method, and the specific implementation details can refer to the description in the method embodiment.
[0088] For example, Figure 2 As shown, the asset metadata collection device may include: a creation module 201, a writing module 202, a declaration module 203 and a writing module 204.
[0089] A creation module 201, used to create a first basic meta information table and a first business meta information table of a data asset from a meta database;
[0090] In this embodiment, data assets mainly include two types of library, table, field and partition meta-information data, which can be divided into basic type and business type, and the meta-information under these two types can be divided into basic meta-information and business meta-information. Specifically, basic meta-information includes information data such as table name, creation time, creator, table type, table storage location, field type and field name; business meta-information includes data such as table administrator, table partition type, table level, table directory, field security level and field standard. Among them, the basic meta-information comes from the tbls table of the hive meta-database of MySQL, and the business meta-information comes from the table_params table of the hive meta-database of MySQL. The MySQL wide table corresponding to the basic meta-information and the business meta-information is associated through a joint primary key, which is composed of a cluster, a library name and a table name; when processing the real-time stream of field meta-information, it is necessary to obtain the joint primary key, and when obtaining the joint primary key, it is necessary to return to the table for reverse query. By declaring a dimension table of basic metadata of a table in Flink SQL, this dimension table is an external dimension table. Through this external dimension table, the joint primary key can be obtained, thereby collecting the basic metadata and business metadata of the data assets.
[0091] Therefore, when collecting data assets, first create the first basic meta information table and the first business meta information table of the data assets from the MySQL meta database (hive database). Specifically, obtain the basic dimension of the basic meta information of the data assets, create a MySQL table of the basic meta information based on the basic dimension, and obtain the first basic meta information table; obtain the business dimension of the business meta information of the data assets, and create a MySQL table of the business meta information based on the business dimension, and obtain the first business meta information table.
[0092] A writing module 202 is used to obtain a first-type basic table and a second-type attribute table, write a target database statement for introducing the first-type basic table, and write the first-type basic table into the first basic meta information table in the asset library based on the target database statement to obtain a second basic meta information table;
[0093] In an embodiment of the present application, the writing module 202 includes:
[0094] An access unit, configured to access a first type processing table and a second type processing table of a target data warehouse, and write the first type processing table and the second type processing table into a distributed message system;
[0095] An introduction unit is used to obtain the service topics corresponding to the first type processing table and the second type processing table in the distributed message system, and introduce the service topics into the data asset center to form the first type basic table and the second type attribute table.
[0096] In an embodiment of the present application, the writing module 202 further includes:
[0097] A first confirmation unit is used to determine whether there is a multi-cluster service. When it is determined that there is no multi-cluster service, based on the target database statement, the first type of basic table is written into the first basic meta information table in the asset library through a preset introduction method to obtain the second basic meta information table;
[0098] The second confirmation unit is used to obtain the cluster name when it is determined to be a multi-cluster service, add the cluster name to the first write statement corresponding to the target database statement to obtain the second write statement, and write the first type of basic table to the first basic meta information table in the asset library through a preset introduction method and the second write statement to obtain the second basic meta information table.
[0099] In this embodiment, the first type of basic table is a basic table of a distributed messaging system (such as Kafka); the second type of attribute table is an attribute table of a distributed messaging system. The first type of basic table and the second type of attribute table are obtained, and a target database statement for introducing the first type of basic table is written. Among them, the target database statement is a flink SQL statement, and flink is a distributed processing engine and framework. Through flink, the flink CDC connector message can be directly written into the MySQL asset library. When the target database statement for introducing the first type of basic table is completed, the first type of basic table is directly written into the first basic meta information table in the asset library based on the target database statement, thereby obtaining a second basic meta information table, which is a wide table obtained by improving the first basic meta information table, and the wide table is stored in the MySQL asset library.
[0100] A declaration module 203 is used to perform a format declaration on the second type attribute table to obtain a target attribute table, generate foreign key information based on the first basic meta information table and the target attribute table, write the foreign key information into the first business meta information table in the asset library, and obtain a second business meta information table;
[0101] In an embodiment of the present application, the declaration module 203 includes:
[0102] an acquiring unit, configured to acquire a target format, and perform a format declaration on the second type attribute table according to the target format to obtain the target attribute table;
[0103] The writing unit is used to use the first basic meta information table as an external dimension table, generate the foreign key information according to the association between the external dimension table and the target attribute table, write the foreign key information into the first business meta information table in the asset library, and obtain the second business meta information table.
[0104] In this embodiment, the default format of the second type attribute table is the avro format, which is a data format that marks the addition, deletion, and update status corresponding to the event when an update operation occurs in a table in a relational database, thereby generating positive and negative flow messages. In actual business, there is no need for upstream messages to produce negative flow effects. Therefore, when the second type attribute table is obtained, the format of the second type attribute table is declared, and the second type attribute table is declared as a target attribute table in a target format. Then, foreign key information is generated based on the first basic meta information table and the target attribute table, and the foreign key information is written to the first business meta information table in the asset library to obtain the second business meta information table, wherein the second business meta information table is a wide table obtained by improving the first business meta information table, and the wide table is stored in the MySQL asset library.
[0105] The writing module 204 is used to obtain the total database statement of the second basic meta information table and the second business meta information table, and create a target connection, and merge the first basic meta information table and the first business meta information table into the index of the target search engine through the target connection and the total database statement.
[0106] In this embodiment, when the second basic meta information table and the second business meta information table are obtained, a total database statement that merges the second basic meta information table and the second business meta information table is written. The total database statement is a flink SQL statement that merges the second basic meta information table and the second business meta information. The total database statement is obtained, and a target connection (i.e., flink CDC) is created. According to the target connection and the total database statement, the first basic meta information table and the first business meta information table can be merged and written into the index of the target search engine. Among them, the target search engine is ES (Elastic Search, search engine), and the index of the target search engine includes the index of the library, table and field in ES. By merging the first basic meta information table and the first business meta information table and writing them into the index of the target search engine, when the asset metadata is read, it can be read through the target search engine, thereby realizing the read-write separation of the asset metadata.
[0107] In one embodiment of the present application, the asset metadata collection device further includes:
[0108] An operation module is used to perform soft deletion, row-to-column conversion and authorization operations on the second basic meta-information table and the second business meta-information table according to the data table control logic statement.
[0109] In one embodiment of the present application, the asset metadata collection device further includes:
[0110] A first logic control module is used to write a logic function of the first field business meta information wide table logic, the first table business information wide table logic and the first library business meta information wide table logic corresponding to the second basic meta information table, and obtain a first data table control logic statement corresponding to the second basic meta information table;
[0111] The second logic control module is used to write the logic functions of the second field business meta information wide table logic, the second table business information wide table logic and the second library business meta information wide table logic corresponding to the second business meta information table, and obtain the second data table control logic statement corresponding to the second business meta information table.
[0112] In one embodiment of the present application, the asset metadata collection device further includes:
[0113] A confirmation module, used to determine whether the modification parameters sent by the external interface are received, and when it is determined that the modification parameters exist, create a service interface;
[0114] A modification module is used to modify the first service meta information table based on the service interface through a transaction modification method and the modification parameters.
[0115] The asset metadata collection device proposed in this application reduces deployment costs, improves the efficiency of asset metadata collection, and realizes read-write separation, thereby reducing the probability of abnormal situations such as asset metadata errors, omissions and duplications, further improving the accuracy of asset metadata collection and saving computing resources.
[0116] In specific implementation, the above modules can be implemented as independent entities, or can be arbitrarily combined and implemented as the same or several entities. The specific implementation methods and corresponding beneficial effects of the above modules can be found in the previous method embodiments, which will not be repeated here.
[0117] The present application also provides an electronic device, which may be a server or a terminal, etc. Figure 3 As shown, it shows a schematic diagram of the structure of the electronic device involved in the embodiment of the present application, specifically:
[0118] The electronic device may include components such as a processor 601 with one or more processing cores, a memory 602 with one or more computer-readable storage media, a power supply 603, and an input unit 604. Those skilled in the art will appreciate that Figure 3 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0119] The processor 601 is the control center of the electronic device, which uses various interfaces and lines to connect various parts of the entire electronic device, and executes various functions of the electronic device and processes data by running or executing computer programs and / or modules stored in the memory 602, and calling data stored in the memory 602. Optionally, the processor 601 may include one or more processing cores; preferably, the processor 601 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 601.
[0120] The memory 602 can be used to store computer programs and modules. The processor 601 executes various functional applications and data processing by running the computer programs and modules stored in the memory 602. The memory 602 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, a computer program required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 602 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 602 may also include a memory controller to provide the processor 601 with access to the memory 602.
[0121] The electronic device also includes a power supply 603 for supplying power to each component. Preferably, the power supply 603 can be logically connected to the processor 601 through a power management system, so as to manage charging, discharging, power consumption and other functions through the power management system. The power supply 603 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components.
[0122] The electronic device may further include an input unit 604, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.
[0123] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 601 in the electronic device will load the executable files corresponding to the processes of one or more computer programs into the memory 602 according to the following instructions, and the processor 601 will run the computer programs stored in the memory 602, thereby realizing various functions, such as:
[0124] Creating a first basic meta information table and a first business meta information table of data assets from the meta database;
[0125] Acquire a first-type basic table and a second-type attribute table, write a target database statement for introducing the first-type basic table, and write the first-type basic table into the first basic meta information table in the asset library based on the target database statement to obtain a second basic meta information table;
[0126] Performing format declaration on the second type attribute table to obtain a target attribute table, generating foreign key information based on the first basic meta information table and the target attribute table, and writing the foreign key information into the first business meta information table in the asset library to obtain a second business meta information table;
[0127] Obtain a total database statement of the second basic meta information table and the second business meta information table, create a target connection, and merge the first basic meta information table and the first business meta information table into an index of a target search engine through the target connection and the total database statement.
[0128] The specific implementation methods and corresponding beneficial effects of the above operations can be found in the detailed description of the asset metadata collection method above, which will not be elaborated here.
[0129] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by a computer program, or by controlling related hardware through a computer program. The computer program may be stored in a computer-readable storage medium and loaded and executed by a processor.
[0130] To this end, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored, and the computer program can be loaded by a processor to execute the steps in any of the asset metadata collection methods provided in the embodiments of the present application. For example, the computer program can execute the following steps:
[0131] Creating a first basic meta information table and a first business meta information table of data assets from the meta database;
[0132] Acquire a first-type basic table and a second-type attribute table, write a target database statement for introducing the first-type basic table, and write the first-type basic table into the first basic meta information table in the asset library based on the target database statement to obtain a second basic meta information table;
[0133] Performing format declaration on the second type attribute table to obtain a target attribute table, generating foreign key information based on the first basic meta information table and the target attribute table, and writing the foreign key information into the first business meta information table in the asset library to obtain a second business meta information table;
[0134] Obtain a total database statement of the second basic meta information table and the second business meta information table, create a target connection, and merge the first basic meta information table and the first business meta information table into an index of a target search engine through the target connection and the total database statement.
[0135] The specific implementation methods and corresponding beneficial effects of the above operations can be found in the previous embodiments, which will not be described in detail here.
[0136] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0137] Since the computer program stored in the computer-readable storage medium can execute the steps in any one of the asset metadata collection methods provided in the embodiments of the present application, the beneficial effects that can be achieved by any one of the asset metadata collection methods provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.
[0138] According to one aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned asset metadata collection method.
[0139] The above is a detailed introduction to an asset metadata collection method, device, electronic device and computer storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A method for collecting asset metadata, characterized in that: include: Creating a first basic meta information table and a first business meta information table of data assets from the meta database; Acquire a first-type basic table and a second-type attribute table, write a target database statement for introducing the first-type basic table, and write the first-type basic table into the first basic meta information table in the asset library based on the target database statement to obtain a second basic meta information table; Performing format declaration on the second type attribute table to obtain a target attribute table, generating foreign key information based on the first basic meta information table and the target attribute table, and writing the foreign key information into the first business meta information table in the asset library to obtain a second business meta information table; Obtain a total database statement of the second basic meta information table and the second business meta information table, create a target connection, and merge the first basic meta information table and the first business meta information table into an index of a target search engine through the target connection and the total database statement.
2. The asset metadata collection method according to claim 1, characterized in that: The obtaining of the first type basic table and the second type attribute table includes: Accessing a first type processing table and a second type processing table of a target data warehouse, and writing the first type processing table and the second type processing table into a distributed message system; The service topics corresponding to the first type processing table and the second type processing table in the distributed message system are obtained, and the service topics are introduced into the data asset center respectively to form the first type basic table and the second type attribute table.
3. The asset metadata collection method according to claim 1, characterized in that: The step of writing the first type basic table into the first basic meta information table in the asset library based on the target database statement to obtain a second basic meta information table includes: Determine whether there is a multi-cluster service. When it is determined that there is no multi-cluster service, write the first type of basic table into the first basic meta information table in the asset library through a preset introduction method based on the target database statement to obtain the second basic meta information table; When it is determined to be a multi-cluster service, the cluster name is obtained, and the cluster name is added to the first write statement corresponding to the target database statement to obtain the second write statement, and the first type of basic table is written to the first basic meta information table in the asset library through a preset introduction method and the second write statement to obtain the second basic meta information table.
4. The asset metadata collection method according to claim 1, characterized in that: The step of performing format declaration on the second type attribute table to obtain a target attribute table, generating foreign key information based on the first basic meta information table and the target attribute table, and writing the foreign key information into the first business meta information table in the asset library to obtain a second business meta information table includes: Acquire a target format, and perform format declaration on the second type attribute table according to the target format to obtain the target attribute table; The first basic meta information table is used as an external dimension table, the foreign key information is generated according to the association between the external dimension table and the target attribute table, and the foreign key information is written into the first business meta information table in the asset library to obtain the second business meta information table.
5. The asset metadata collection method according to claim 1, characterized in that: After writing the foreign key information into the first business meta information table to obtain the second business meta information table, the method includes: The second basic meta information table and the second business meta information table are soft deleted, row-to-column transferred and authorized according to the data table control logic statement, and the data table control logic statement is the data table control logic statement of the second basic meta information table and the second business meta information table.
6. The asset metadata collection method according to claim 5, characterized in that: The data table control logic statement includes a first data table control logic statement and a second data table control logic statement, and before the second basic meta information table and the second business meta information table are soft-deleted, row-to-column transferred, and authorized according to the data table control logic statement, the following steps are included: Compile the logic functions of the first field business meta information wide table logic, the first table business information wide table logic and the first library business meta information wide table logic corresponding to the second basic meta information table to obtain the first data table control logic statement corresponding to the second basic meta information table; Write the logical functions of the second field business meta information wide table logic, the second table business information wide table logic and the second library business meta information wide table logic corresponding to the second business meta information table to obtain the second data table control logic statement corresponding to the second business meta information table.
7. The asset metadata collection method according to claim 1, characterized in that: After merging the first basic meta information table and the first business meta information table into the index of the target search engine through the target connection and the total database statement, the method further includes: Determining whether a modification parameter sent by an external interface is received, and when it is determined that the modification parameter exists, creating a service interface; Based on the service interface, the first service meta information table is modified through a transaction modification method and the modification parameters.
8. An asset metadata collection device, characterized in that: include: A creation module, used to create a first basic meta information table and a first business meta information table of a data asset from a meta database; A writing module, used for obtaining a first type basic table and a second type attribute table, writing a target database statement for introducing the first type basic table, and writing the first type basic table into the first basic meta information table in the asset library based on the target database statement to obtain a second basic meta information table; a declaration module, configured to perform format declaration on the second type attribute table to obtain a target attribute table, generate foreign key information based on the first basic meta information table and the target attribute table, write the foreign key information into the first business meta information table in the asset library, and obtain a second business meta information table; A writing module is used to obtain the total database statement of the second basic meta information table and the second business meta information table, and create a target connection, and merge the first basic meta information table and the first business meta information table into the index of the target search engine through the target connection and the total database statement.
9. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores a computer program, and the processor is used to run the computer program in the memory to execute the asset metadata collection method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the asset metadata collection method according to any one of claims 1 to 7.