Metadata unified management method, system, medium, equipment and program

Abstract

The present application provides a method, system, medium, device and program for unified metadata management. In the present application, any of the multiple data paths of the metadata unified management system first calls the metadata management center during the process of importing data to the target Hbase, and obtains the metadata model corresponding to the target Hbase in the metadata management center, and then processes the data for the data import operation according to the metadata model to write the processed data to the target Hbase. It can be seen that during the data import process, by uniformly managing the metadata model in the metadata management center, any data source path will use this metadata model as the standard when importing data, thereby effectively solving the technical problem of data alignment failure in the synchronization of multiple data paths.

Description

Technical Field

[0001] The present application relates to the field of financial technology (Fintech), and in particular to a metadata unified management method, system, medium, device and program. Background Art

[0002] With the development of computer technology, more and more technologies are applied in the financial field. The traditional financial industry is gradually transforming to Fintech (Finteh), and interface testing technology is no exception. However, due to the security and real-time requirements of the financial industry, higher requirements are also placed on data management technology.

[0003] At present, with the advent of the era of cloud computing and large-scale data, various types of Internet applications are emerging in an endless stream, especially with the rapid development of social networks represented by social networking services (SNS). Traditional relational databases are increasingly unable to meet the requirements of cloud computing and big data for processing massive data, high concurrent read and write speeds, scalability, and high availability. In order to meet the requirements for processing massive data, high concurrent read and write speeds, scalability, and high availability, non-relational distributed storage systems have emerged. Among them, HBase is a highly reliable, high-performance, column-oriented, and scalable distributed storage system. Using HBase technology, a large-scale structured storage cluster can be built on a cheap hardware environment.

[0004] However, the data sources of HBase are usually multi-channel, which can easily cause confusion in the metadata of multiple data sources. Summary of the invention

[0005] The embodiments of the present application provide a unified metadata management method, system, medium, device and program to solve the technical problem of metadata confusion of multiple data sources caused by multi-channel data sources of HBase.

[0006] In a first aspect, an embodiment of the present application provides a metadata unified management method, which is applied to a metadata unified management system, wherein the metadata unified management system includes: an Hbase cluster, a metadata management center, and multiple data paths, wherein each data path is implemented by a different code, and the method includes:

[0007] Any data path among the multiple data paths calls the metadata management center during the process of performing a data import operation to a target Hbase, where the target Hbase is a database in the Hbase cluster performing the data import operation;

[0008] Obtain the metadata model corresponding to the target Hbase in the metadata management center;

[0009] The data for the data import operation is processed according to the metadata model to write the processed data to the target Hbase.

[0010] In one possible design, the plurality of data paths include: a first data path, a second data path, and a third data path;

[0011] The first data path and the third data path are used to input real-time data, and the second data path is used to input data before a preset time, so that the data input through the second data path supplements the data input through the first data path or the third data path.

[0012] In a possible design, the data for the data import operation is processed according to the metadata model to write the processed data to the target Hbase, including:

[0013] If the data path for performing the data import operation is the first data path, after the core online system generates transaction data, the core online system sends the transaction data to the preset stream processing platform through an asynchronous task;

[0014] Consume the transaction data in the preset stream processing platform through a preset data source extraction tool, and write the transaction data to the target Hbase according to the metadata model; or,

[0015] If the data path for the data import operation is the second data path, the core online system generates transaction data to the online library, and extracts the transaction data to the preset data warehouse tool in the form of extraction through the preset data transfer tool;

[0016] Cleaning and processing the transaction data through the data warehouse tool, and reading the metadata model for data assembly;

[0017] Write the assembled transaction data into the target Hbase through a preset data import tool; or,

[0018] If the data path for performing the data import operation is the third data path, after the core online system generates transaction data, a preset remote calling tool is called;

[0019] The transaction data is written to the target Hbase through the preset remote calling tool and according to the metadata model.

[0020] In a possible design, after processing the data for the data import operation according to the metadata model, the method further includes:

[0021] The column definitions corresponding to the data after the data import operation in each data path are checked for consistency according to a preset information summary algorithm, and after confirming that the columns of the data written in each data path have the same order, the processed data are written to the target Hbase.

[0022] In a possible design, the metadata unified management method further includes:

[0023] If the metadata model in the metadata management center changes, the metadata management center sends a metadata model change notification to each data source;

[0024] The metadata model in the metadata management center includes: Hbase row key rules, and the Hbase row key rules include: Hbase column clusters, Hbase column structures, and Hbase column check fields; wherein the Hbase column clusters are used to constrain column clusters in HBase, the Hbase column structure is used to constrain column structures in HBase, and the Hbase column check fields are used to check columns in HBase.

[0025] In a possible design, after processing the data for the data import operation according to the metadata model to write the processed data to the target Hbase, the method further includes:

[0026] Reading metadata from a metadata center of the metadata management center, and generating a data code according to the metadata;

[0027] The data codes are packaged and compiled to generate corresponding function classes, and the function classes are used to provide business capabilities for the metadata unified management system.

[0028] In a possible design, after the data code is packaged and compiled to generate a corresponding function class, the method further includes:

[0029] Acquire a data query instruction, the metadata unified management system is used to provide data support for the data query system, and the function class is used to support the query instruction;

[0030] The corresponding metadata is read from the metadata center of the metadata management center according to the query instruction.

[0031] In a possible design, after reading the corresponding metadata from the metadata center of the metadata management center according to the query instruction, the method further includes:

[0032] According to a preset time period, periodically read new metadata from the metadata center of the metadata management center, and generate a new data code according to the new metadata;

[0033] Verify whether the first mapping relationship is consistent with the second mapping relationship, wherein the first mapping relationship is a mapping relationship between the data code and the metadata in the metadata center, and the second mapping relationship is a mapping relationship between the new data code and the metadata in the metadata center;

[0034] If there is any inconsistency, an alarm message is sent to perform code reconstruction.

[0035] In a second aspect, an embodiment of the present application further provides a unified metadata management system, including: an Hbase cluster, a metadata management center, and multiple data paths, wherein each data path is implemented by a different code;

[0036] Any data path among the multiple data paths calls the metadata management center during the process of performing a data import operation to a target Hbase, where the target Hbase is a database in the Hbase cluster performing the data import operation;

[0037] Obtain the metadata model corresponding to the target Hbase in the metadata management center;

[0038] The data for the data import operation is processed according to the metadata model to write the processed data to the target Hbase.

[0039] In one possible design, the plurality of data paths include: a first data path, a second data path, and a third data path;

[0040] The first data path and the third data path are used to input real-time data, and the second data path is used to input data before a preset time, so that the data input through the second data path supplements the data input through the first data path or the third data path.

[0041] In a possible design, if the data path for performing the data import operation is the first data path, after the core online system generates transaction data, the core online system sends the transaction data to the preset stream processing platform through an asynchronous task;

[0042] Consume the transaction data in the preset stream processing platform through a preset data source extraction tool, and write the transaction data to the target Hbase according to the metadata model; or,

[0043] If the data path for the data import operation is the second data path, the core online system generates transaction data to the online library, and extracts the transaction data to the preset data warehouse tool in the form of extraction through the preset data transfer tool;

[0044] Cleaning and processing the transaction data through the data warehouse tool, and reading the metadata model for data assembly;

[0045] Write the assembled transaction data into the target Hbase through a preset data import tool; or,

[0046] If the data path for performing the data import operation is the third data path, after the core online system generates transaction data, a preset remote calling tool is called;

[0047] The transaction data is written to the target Hbase through the preset remote calling tool and according to the metadata model.

[0048] In one possible design, the column definitions corresponding to the data after the data import operation in each data path are checked for consistency according to a preset information summary algorithm, and after confirming that the columns of the data written in each data path have the same order, the processed data is written to the target Hbase.

[0049] In a possible design, if the metadata model in the metadata management center changes, the metadata management center sends a metadata model change notification to each data source;

[0050] The metadata model in the metadata management center includes: Hbase row key rules, and the Hbase row key rules include: Hbase column clusters, Hbase column structures, and Hbase column check fields; wherein the Hbase column clusters are used to constrain column clusters in HBase, the Hbase column structure is used to constrain column structures in HBase, and the Hbase column check fields are used to check columns in HBase.

[0051] In a possible design, metadata is read from a metadata center of the metadata management center, and a data code is generated according to the metadata;

[0052] The data codes are packaged and compiled to generate corresponding function classes, and the function classes are used to provide business capabilities for the metadata unified management system.

[0053] In one possible design, the system further includes:

[0054] The data query system is used to obtain data query instructions, the metadata unified management system is used to provide data support for the data query system, and the function class is used to support the query instructions;

[0055] The corresponding metadata is read from the metadata center of the metadata management center according to the query instruction.

[0056] In a possible design, new metadata is periodically read from the metadata center of the metadata management center according to a preset time period, and a new data code is generated according to the new metadata;

[0057] Verify whether the first mapping relationship is consistent with the second mapping relationship, wherein the first mapping relationship is a mapping relationship between the data code and the metadata in the metadata center, and the second mapping relationship is a mapping relationship between the new data code and the metadata in the metadata center;

[0058] If there is any inconsistency, an alarm message is sent to perform code reconstruction.

[0059] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0060] processor; and,

[0061] A memory, configured to store executable instructions of the processor;

[0062] The processor is configured to execute any one of the metadata unified management methods in the first aspect by executing the executable instructions.

[0063] In a fourth aspect, an embodiment of the present application further provides a storage medium on which a computer program is stored, and when the program is executed by a processor, any one of the metadata unified management methods in the first aspect is implemented.

[0064] In a fifth aspect, an embodiment of the present application further provides a computer program product, including a computer program, which, when executed by a processor, implements any one of the metadata unified management methods in the first aspect.

[0065] The embodiment of the present application provides a unified metadata management method, system, medium, device and program. When any of the multiple data paths of the unified metadata management system performs a data import operation to a target Hbase, the metadata management center is first called to obtain the metadata model corresponding to the target Hbase in the metadata management center, and then the data for the data import operation is processed according to the metadata model to write the processed data to the target Hbase. It can be seen that during the data import process, by uniformly managing the metadata model in the metadata management center, any data source path will use this metadata model as the standard when importing data, thereby effectively solving the technical problem of data alignment failure in the synchronization of multiple data paths. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0067] Figure 1 It is a schematic diagram of an application scenario of the metadata unified management method shown in an embodiment of the present application;

[0068] Figure 2 It is a flowchart of a unified metadata management method shown in the first embodiment of the present application;

[0069] Figure 3 It is a structural table of the storage model of Hbase in the prior art;

[0070] Figure 4 It is a structural table of the storage model of Hbase in the embodiment of the present application;

[0071] Figure 5 It is a flowchart of a metadata unified management method shown in the second embodiment of the present application;

[0072] Figure 6 It is a schematic diagram of the structure of the electronic device shown in the third embodiment of the present application. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0074] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0075] For the convenience of explanation, the relevant terms are first explained as follows:

[0076] Hive: Hive is a data warehouse tool based on Hadoop. It can organize data, perform special queries, and analyze data sets stored in files on HDFS. It provides a query language similar to SQL - HiveQL. Simple MR statistics can be implemented through HQL statements. Hive converts HQL statements into MR tasks for execution.

[0077] Hadoop: is a software framework that can perform distributed processing on large amounts of data. Hadoop consists of four modules:

[0078] Common: public tools that support other modules;

[0079] HDFS: Hadoop Distributed File System (HDFS), a distributed file system that provides high throughput access performance;

[0080] YARN: provides a framework for job scheduling and cluster resource management;

[0081] MapReduce: A big data parallel computing framework, referred to as MR.

[0082] Hbase: HBase is an open source non-relational distributed database (NoSQL), which is modeled after Google's BigTable and implemented in Java. It is part of the Apache Software Foundation's Hadoop project, runs on the HDFS file system, and provides services similar to BigTable for Hadoop. Therefore, it can store massive amounts of sparse data in a fault-tolerant manner. Hbase's data storage management includes two aspects: metadata management and data management.

[0083] Metadata: Hbase stores the corresponding region information in the meta table. The client reads and writes the meta table first to find the server where the corresponding region is located, and then directly performs read and write operations through RPC.

[0084] Data: Hbase data is stored in HDFS in strict lexicographic order, with hfile as the smallest file unit.

[0085] Spark Streaming: A framework provided by Spark for real-time computing of big data; its bottom layer is also based on Spark Core; its basic computing model is still based on the memory-based big data real-time computing model RDD, but in view of the characteristics of real-time computing, a layer of encapsulation is performed on top of RDD, called DStream (similar to DataFrame in Spark SQL); so RDD is the core of the entire Spark technology ecosystem. Spark Streaming is an extension of Spark CoreApi, which can be used for large-scale, high-throughput, fault-tolerant real-time data stream processing; it supports reading data from many data sources, such as Kafka, Flume, Twitter, ZeroMQ, Kinesis or TCP Socket, and can use complex algorithms similar to high-order functions to process data, such as map, reduce, join, window; the processed data can be saved to the file system, database, Dashboard and other storage.

[0086] Business Hbase metadata: The metadata involved in this application is metadata at the Hbase business level, such as rowkey rules, column clusters, and column values ​​in Hbase. Due to the simultaneous update of multiple data sources, especially when the various data source channels are not from the same department or subsystem, it is difficult to reach a consensus, which leads to chaotic metadata management.

[0087] With the development of computer technology, more and more technologies are applied in the financial field. The traditional financial industry is gradually transforming to Fintech (Finteh), and interface testing technology is no exception. However, due to the security and real-time requirements of the financial industry, higher requirements are also put forward for technology. The traditional financial industry is gradually transforming to Fintech (Finteh), and interface testing technology is no exception. However, due to the security and real-time requirements of the financial industry, higher requirements are also put forward for data management technology. At present, with the advent of the era of cloud computing and large-scale data, various types of Internet applications are emerging in an endless stream, especially with the rapid development of social networks represented by social networking services (SNS). Traditional relational databases have become increasingly unable to meet the requirements of cloud computing and big data for processing massive data, high concurrent read and write speed, scalability, high availability, etc. Among them, taking the foreign FriendFeed as an example, it reached 250 million user dynamics in one month. For relational databases, the efficiency of performing structured query language (SQL) queries in a table with 250 million records is extremely low.

[0088] In order to meet the requirements of processing massive data, high concurrent read and write speed, scalability, high availability, etc., non-relational distributed storage systems came into being. Among them, HBase is a highly reliable, high-performance, column-oriented, scalable distributed storage system. Using HBase technology, a large-scale structured storage cluster can be built on a cheap hardware environment. HBase is based on the Hadoop HDFS file storage system, uses MapReduce to process massive data, and uses Zookeeper as a collaborative service. It uses a simple key-data image relationship to provide a good solution for ultra-large-scale and high-concurrency massive data real-time response systems, and its data sources are multi-channel, such as HIVE batch derivatives, T0 random writes, etc. However, since HBase is a KV (Key-Value) storage, the metadata management confusion of multiple data sources is a major problem.

[0089] In the prior art, Hbase generally writes data in two forms: random writing and batch writing. However, in either case, the data will first pass through the memory of the Hbase regionserver, and then be flushed to the Hadoop distributed file system (HDFS) through Memstore for storage persistence. Therefore, the general data import method will adopt the following solutions:

[0090] Solution 1: Randomly read and write data to the cluster through the client: This solution is to read data from MySQL or Hive through the Java API and write it to the Hbase cluster row by row or in batches.

[0091] However, the above solution 1 is a typical operation on relational databases and non-relational databases, written in the form of APIs, while the metadata (rowkey) of Hbase is generally spliced ​​in the code, and the fields are also controlled by code. In this case, especially for the Hbase multi-data source channel mode, it is very difficult to ensure that the metadata written by all parties are consistent, which is very easy to make mistakes in production.

[0092] Solution 2: Accept real-time input data streams and split the data into multiple batches. For example, collect 1s of data and convert it into a batch. Then, send each batch to Spark's computing engine for processing. Finally, a result data stream will be generated, and then the result data stream will be written to the target Hbase through the Hbase API.

[0093] However, Solution 2 is similar to Solution 1, except that it is encapsulated through the introduced Spark Streaming API, so the data model (metadata) written to Hbase is also implemented through code, so there is also the problem that the business Hbase metadata cannot be managed.

[0094] Solution 3: Import Hive data into a single cluster in bulk load mode, convert Hive data files into final persistent data files in the format of Hbase's final storage file hfile through MR, and then move the files to the corresponding Hbase storage directory.

[0095] However, Solution 3 uses MR tasks to import the cleaned data in Hive into Hbase in bulk load mode, which has the best performance and minimizes the performance impact on the big data Hbase. However, metadata management is also disordered when importing data to Hbase, which is very prone to errors.

[0096] In summary, all three approaches above have a big drawback, which is that Hbase is a Nosql database, that is, a non-relational database, and there is no strong validation for columns. In addition, the big data query system obtains data from Hbase and spits it out to the outside world. Because these multi-party channels rely on Hbase's metadata and cannot guarantee the definition and maintenance of metadata in each system.

[0097] Therefore, the embodiment of the present application aims to provide a unified metadata management method. In the process of importing data to the target Hbase, any data path in the multiple data paths of the metadata unified management system first calls the metadata management center and obtains the metadata model corresponding to the target Hbase in the metadata management center. Then, the data for the data import operation is processed according to the metadata model to write the processed data to the target Hbase. It can be seen that in the process of data import, by uniformly managing the metadata model in the metadata management center, any data source path will use this metadata model as the basis when importing data, thereby effectively solving the technical problem of data alignment failure in the synchronization of multiple data paths.

[0098] Figure 1 Schematic diagram of the application scenario of the metadata unified management method shown in the embodiment of the present application. Figure 1 As shown, in this embodiment, the data import mainly includes three forms, and these three forms complement each other. Mode 1 and Mode 3 are T0 mode storage, but considering that T0 data may have the risk of data loss, the T1 mode of Mode 2 is needed to make up the data and perform data backup processing. It is worth noting that the T0 mode refers to storage through real-time data, while T1 refers to storage of data on the next day, that is, the T1 mode is more complete than the T0 mode, but has a certain lag.

[0099] For method 1, T0 is imported into Hbase in quasi-real time. Here, quasi-real time means that after the core online system generates a transaction, the online system sends the transaction data to the Kafka cluster through an asynchronous task, and then consumes the Kafka data through Spark Streaming and writes it to Hbase according to the model definition.

[0100] For method 2, which refers to T1 push, currently transactions are generated synchronously by the online system to the core online library, and the core metadata layer data is extracted to Hive through SQOOP extraction, and then Hive performs data cleaning and processing using data warehouse technology (Extract-Transform-Load, ETL). At the same time, the metadata center is read to obtain Hbase business metadata, and it is assembled and imported into Hbase through Bulkload. This method should be the most efficient import method. It directly imports Hfile data in Hbase format into Hbase through the exchange of underlying files, which takes effect immediately. However, although this approach is efficient, it will have a certain lag at the data level. The lag time in this solution is greater than or equal to 1 day.

[0101] Method 3 refers to calling the online service provided by the core through the Remote Method Invocation (RMI) technology to return T0 data. In order to ensure the availability and stability of transactions, the core only provides T0 data queries. The core will regularly delete and clean up previous historical data.

[0102] It can be seen that the three methods used in this application synchronize the core data of the element layer to the Hbase cluster of the big data real-time query system. Since there are many subsystems involved and the Hbase data storage definition is flexible enough without strong constraints on the data structure, it is very easy to cause misalignment of multiple parties, resulting in data synchronization disorder and data error production.

[0103] Continue to refer to Figure 2 In this implementation, the above problems are solved by metadata management. Specifically, in the above data synchronization process, since there will be misalignment in the synchronization of multiple data channels, the metadata of the Hbase business layer should be managed uniformly in the metadata center. The metadata management center is based on TDSQL (Tencent Cloud Database), a reliable financial data source, as a storage medium. No matter how many data source channels there are, this model will prevail. For the specific model, please refer to the model details shown below. The metadata management center is modified by developers and notifies each subsystem whether there is a change. Every time the data source channel involves the relevant data source Hbase, it will call the metadata model center to obtain the metadata of the target Hbase table: Hbase rowkey, column cluster, column, etc., and combine it with the code of each channel respectively. Here, it is required to pull the latest metadata from the management center every time.

[0104] Since the column Value returned after pulling is a Json string, T0 RMI and SPARK STREAMING traverse the KV, while ETL reads the field Column, traverses it according to the JSON string, and corresponds one by one with the fields of the Hive Bulkload table.

[0105] In order to ensure a one-to-one correspondence between the data in the Hbase business metadata center and the columns pushed by Hive, Hive deserializes the data in the metadata center into Json Object during ETL Bulkload to Hbase. There may be inconsistencies in the order of Json deserialized by different Json serialization tools. At the same time, for other channel derivatives, there is also concern that the read data may be disordered, leading to mapping errors, and ultimately causing data disorder in Hbase. Therefore, the MD5 Message-Digest Algorithm consistency secondary check will be performed on the obtained column definition in each channel to prevent the read data from being misordered due to secondary deserialization.

[0106] In addition, the column Key returned after pulling is the Hbase column name, and the Value is the Hive column name. After each system reads the model Value, it will first perform Json serialization on the Value. However, due to different serialization methods, such as Java's Fastjson, Gson, or Python's Json dump, the positions of Key and Value may change after the object is serialized. In addition, in Hive data processing, Bulkload table production is strictly spliced ​​in the order of fields.

[0107] Therefore, it is necessary to strictly specify the Dimporttsv.columns=HBASE_ROW_KEY,${columns} format, so Hive needs to check whether the columns read through Json are in consistent order and ensure that there are no missing or redundant data fields. Therefore, in the embodiment of the present application, it is necessary to check each field and convert the Hbase fields.

[0108] After each system gets Hbase_columns, it will be converted into json and compared with the check field, such as the Md5 (hbase_columns_check) value and the Md5 (hbase_columns. keyset) value, to ensure that the data we read is correct and orderly, and multiple subsystems can be unified. All systems are based on this definition. Currently, only columns are guaranteed, but there are other similarities such as rowkey rules. With the definition of the management center and unified metadata management, the metadata disorder caused by the inconsistent pace of multiple systems can be well solved. At the same time, after the version of the metadata center is changed, it will be pushed to each relevant person in charge in a timely manner, so as to achieve unified metadata management and achieve unity of knowledge and action.

[0109] According to the above metadata management steps, the metadata management at the Hbase table level can be achieved by the unified management of the metadata center, and each channel source follows the principle of unified configuration to ensure unique accuracy. However, the big data query system QS is actually an interface spitting service, so the data accuracy of QS directly determines the accuracy of the service. Among them, the traditional way is to fix this mapping relationship directly in the code to represent the meta-field information in Hbase, so it is necessary to manually ensure whether the fields are complete, corresponding, misplaced, etc., so there are many uncertain factors and it is very easy to produce bugs. In this application, the code automatically generates the mapping relationship and checks the accuracy of the fields according to the definition of the metadata center, so as to ensure that our model definition is completely consistent with that of the center. In this application, the Java bean in the big data query system corresponds one-to-one to the table fields of Hbase, and the bean in the Service Data Transfer Object (DTO) is provided after relevant conversion according to the business scenario. Therefore, to ensure the consistency of this DTO and metadata, the mapping relationship in the Hbase Domain is obtained.

[0110] At the same time, when the continuous integration (CI) is built, it will automatically check whether the Hbase Domain in the big data query service QS is consistent with the Hbase column in the metadata center. The big data query system pulls the HbaseColumns field and Rowkey rule field of the metadata center through Java Database Connectivity (JDBC) and serializes it through JsonObj. Then, the corresponding metadata information defined by Hbase and hive is obtained, and the Hbase Column check field is read to verify whether the data serialized by Fastjson is accurate. This ensures that the data read out from the metadata center is completely correct, providing conditions for the subsequent automatic generation of code based on the metadata read out. Then, after packaging and compiling, the relevant mapping Domin class can be automatically generated for subsequent business development and automatically generated functional classes.

[0111] Finally, continue to refer to Figure 1, regular reliability inspections can be performed. According to the above steps, the metadata unification problem of multiple heterogeneous data sources can basically be solved, but the consistency between the code in the subsequent big data query service and the metadata center cannot be guaranteed. Since multiple channel sources will dynamically obtain the corresponding Hbase table-level metadata from the metadata management center each time they are derived to Hbase, there is no regular reliability guarantee problem. It is only necessary in the scenario where the metadata in the metadata is solidified into the system code, such as the big data query system, so it is necessary to configure a scheduled task to periodically check whether the mapping relationship between the Hbase table-level metadata in the metadata and the metadata in the big data query service is normal. If there is an abnormality, a Fata alarm will be sent and code will be built to ensure that the data metadata and the data in the business code in this scenario are consistent.

[0112] Figure 2 FIG. 1 is a flow chart of a metadata unified management method according to the first embodiment of the present application. Figure 2 As shown, the metadata unified management method provided in this embodiment includes:

[0113] Step 101: Any data path among the multiple data paths calls the metadata management center during the process of importing data to the target Hbase.

[0114] In this implementation, the metadata unified management method is applied to the metadata unified management system, wherein the metadata unified management system includes: an Hbase cluster, a metadata management center, and multiple data paths, wherein each data path is implemented by different codes. Any data path in the multiple data paths calls the metadata management center during the process of importing data to a target Hbase, wherein the target Hbase is a database in the Hbase cluster for the data import operation.

[0115] It is worth noting that the above-mentioned multiple data paths may include: a first data path, a second data path and a third data path. The first data path and the third data path are used to input real-time data, and the second data path is used to input data before a preset time, so that the data input through the second data path supplements the data input through the first data path or the third data path.

[0116] In a possible design, data import can be in three forms, that is, data import operations are performed through three different data channels. Moreover, these three are complementary to each other. Mode 1 (i.e., through the first data channel) and Mode 3 (i.e., through the third data channel) are T0 mode storage, but considering that T0 data may have the risk of data loss, Mode 2 (i.e., through the second data channel) T1 mode is needed to supplement the data and perform data backup processing. Among them, it is worth explaining that the T0 mode refers to storage through real-time data, while T1 refers to storage of data on the next day, that is, the T1 mode is more complete than the T0 mode, but there is a certain lag.

[0117] For method 1, that is, if the data path for data import operation is the first data path, after the core online system generates transaction data, the core online system sends the transaction data to the preset stream processing platform through asynchronous tasks, and then consumes the transaction data in the preset stream processing platform through the preset data source extraction tool, and writes the transaction data to the target Hbase according to the metadata model. Specifically, method 1 is that T0 is imported into Hbase through a quasi-real-time approach. Here, quasi-real-time means that after the core online system generates a transaction, the online system sends the transaction data to the Kafka cluster through asynchronous tasks, and then consumes the Kafka data through Spark Streaming, and writes it to Hbase according to the model definition.

[0118] For method 2, that is, if the data path for the data import operation is the second data path, the core online system generates transaction data to the online library, and extracts the transaction data to the preset data warehouse tool in the form of extraction through the preset data transfer tool. The transaction data is cleaned and processed by the data warehouse tool, and the metadata model is read for data assembly. The assembled transaction data is then written to the target Hbase through the preset data import tool. Specifically, method 2 refers to T1 push. Currently, transactions are generated synchronously by the online system to the core online library, and the core metadata layer data is extracted to Hive through SQOOP extraction. Hive then performs data cleaning and processing using data warehouse technology (Extract-Transform-Load, ETL). At the same time, it reads the metadata center to obtain Hbase business metadata, assembles it, and imports it to Hbase through Bulkload. This method should be the most efficient import method. It directly imports Hfile data in Hbase format into Hbase through the exchange of underlying files. It takes effect immediately. However, although this approach is efficient, it will have a certain lag at the data level. The lag period in this solution is greater than or equal to 1 day.

[0119] For method 3, if the data path for data import operation is the third data path, after the core online system generates transaction data, the preset remote call tool is called; through the preset remote call tool, the transaction data is written to the target Hbase according to the metadata model. Specifically, method 3 refers to calling the online service provided by the core to return T0 data through the remote call technology (Remote Method Invocation, RMI). In order to ensure the availability and stability of transactions, the data provided by the core only provides T0 data query, and the core will regularly delete and clean up the previous historical data.

[0120] It can be seen that the three methods used in this application synchronize the core meta-layer data to the Hbase cluster of the big data real-time query system.

[0121] Step 102: Obtain the metadata model corresponding to the target Hbase in the metadata management center.

[0122] Figure 3 It is a structural table of the Hbase storage model in the prior art. Figure 3 As shown in the figure, the storage model of Hbase mainly includes primary key, column cluster, column, and value as basic units. Column cluster and column can be expanded freely. However, the primary key must be unique in the entire table, and the underlying storage is arranged in lexicographic order. Due to the flexibility of this KV storage, various data source channels cannot strictly agree on a certain definition. When the data source channel changes, various abnormal situations such as Hbase data confusion and loss will occur.

[0123] Therefore, in order to manage this part of data in a unified manner, all data sources can be based on the data in the metadata center, and if there is any change, each data source channel will be notified. Figure 4 It is a structural table of the storage model of Hbase in the embodiment of the present application. Figure 4 As shown, the storage model of Hbase, that is, the metadata model, can be Figure 4 Model definition shown. Specifically, the metadata model in the metadata management center includes: Hbase row key rules, Hbase row key rules include: Hbase column clusters, Hbase column structures and Hbase column check fields, where Hbase column clusters are used to constrain column clusters in HBase, Hbase column structures are used to constrain column structures in HBase, and Hbase column check fields are used to check columns in HBase. If the metadata model in the metadata management center changes, the metadata management center sends a metadata model change notification to each data source.

[0124] Step 103: Process the data for data import operation according to the metadata model to write the processed data into the target Hbase.

[0125] After obtaining the metadata model corresponding to the target Hbase in the metadata management center, the data for data import operation is processed according to the metadata model to write the processed data to the target Hbase. Specifically, in this step, the above-mentioned problem is solved by performing metadata management. Specifically, in the above-mentioned data import process, since the synchronization of multiple data channels may be misaligned, the metadata of the Hbase business layer must be uniformly managed in the metadata center. Among them, the metadata management center is based on TDSQL (Tencent Cloud Database), a reliable financial-grade data source, as a storage medium. No matter how many data source channels there are, this model will prevail. For the specific model, please refer to Figure 4 The model details shown in the figure are as follows. The metadata management center is modified by developers and notifies each subsystem whether any changes occur. Every time the data source channel involves the relevant data source Hbase, it will call the metadata model center to obtain the metadata of the target Hbase table: Hbase rowkey, column cluster, column, etc., and combine it with the code of each channel respectively. Here, it is required to pull the latest metadata from the management center every time.

[0126] In this embodiment, when any of the multiple data paths of the metadata unified management system performs a data import operation to the target Hbase, it first calls the metadata management center and obtains the metadata model corresponding to the target Hbase in the metadata management center, and then processes the data for the data import operation according to the metadata model to write the processed data to the target Hbase. It can be seen that during the data import process, by uniformly managing the metadata model in the metadata management center, any data source path will use this metadata model as the basis when importing data, thereby effectively solving the technical problem of data alignment failure in the synchronization of multiple data paths.

[0127] Figure 5 FIG. 1 is a flow chart of a metadata unified management method according to Embodiment 2 of the present application. Figure 5 As shown, the metadata unified management method provided in this embodiment includes:

[0128] Step 201: Any data path among the multiple data paths calls the metadata management center during the process of importing data to the target Hbase.

[0129] In this implementation, the metadata unified management method is applied to the metadata unified management system, wherein the metadata unified management system includes: an Hbase cluster, a metadata management center, and multiple data paths, wherein each data path is implemented by different codes. Any data path in the multiple data paths calls the metadata management center during the process of importing data to a target Hbase, wherein the target Hbase is a database in the Hbase cluster for the data import operation.

[0130] It is worth noting that the above-mentioned multiple data paths may include: a first data path, a second data path and a third data path. The first data path and the third data path are used to input real-time data, and the second data path is used to input data before a preset time, so that the data input through the second data path supplements the data input through the first data path or the third data path.

[0131] In a possible design, data import can be in three forms, that is, data import operations are performed through three different data channels. Moreover, these three are complementary to each other. Mode 1 (i.e., through the first data channel) and Mode 3 (i.e., through the third data channel) are T0 mode storage, but considering that T0 data may have the risk of data loss, Mode 2 (i.e., through the second data channel) T1 mode is needed to supplement the data and perform data backup processing. Among them, it is worth explaining that the T0 mode refers to storage through real-time data, while T1 refers to storage of data on the next day, that is, the T1 mode is more complete than the T0 mode, but there is a certain lag.

[0132] For method 1, that is, if the data path for data import operation is the first data path, after the core online system generates transaction data, the core online system sends the transaction data to the preset stream processing platform through asynchronous tasks, and then consumes the transaction data in the preset stream processing platform through the preset data source extraction tool, and writes the transaction data to the target Hbase according to the metadata model. Specifically, method 1 is that T0 is imported into Hbase through a quasi-real-time approach. Here, quasi-real-time means that after the core online system generates a transaction, the online system sends the transaction data to the Kafka cluster through asynchronous tasks, and then consumes the Kafka data through Spark Streaming, and writes it to Hbase according to the model definition.

[0133] For method 2, that is, if the data path for the data import operation is the second data path, the core online system generates transaction data to the online library, and extracts the transaction data to the preset data warehouse tool in the form of extraction through the preset data transfer tool. The transaction data is cleaned and processed by the data warehouse tool, and the metadata model is read for data assembly. The assembled transaction data is then written to the target Hbase through the preset data import tool. Specifically, method 2 refers to T1 push. Currently, transactions are generated synchronously by the online system to the core online library, and the core metadata layer data is extracted to Hive through SQOOP extraction. Hive then performs data cleaning and processing using data warehouse technology (Extract-Transform-Load, ETL). At the same time, it reads the metadata center to obtain Hbase business metadata, assembles it, and imports it to Hbase through Bulkload. This method should be the most efficient import method. It directly imports Hfile data in Hbase format into Hbase through the exchange of underlying files. It takes effect immediately. However, although this approach is efficient, it will have a certain lag at the data level. The lag period in this solution is greater than or equal to 1 day.

[0134] For method 3, if the data path for data import operation is the third data path, after the core online system generates transaction data, the preset remote call tool is called; through the preset remote call tool, the transaction data is written to the target Hbase according to the metadata model. Specifically, method 3 refers to calling the online service provided by the core to return T0 data through the remote call technology (Remote Method Invocation, RMI). In order to ensure the availability and stability of transactions, the data provided by the core only provides T0 data query, and the core will regularly delete and clean up the previous historical data.

[0135] It can be seen that the three methods used in this application synchronize the core meta-layer data to the Hbase cluster of the big data real-time query system.

[0136] Step 202: Obtain the metadata model corresponding to the target Hbase in the metadata management center.

[0137] Hbase's storage model is as follows Figure 3 As shown in the figure, it mainly includes primary key, column cluster, column, and value as basic units. Column cluster and column can be expanded freely. However, the primary key must be unique in the entire table, and the underlying storage is arranged in lexicographic order. Due to the flexibility of this KV storage, various data source channels cannot strictly agree on a certain definition. When the data source channel changes, various abnormal situations such as Hbase data confusion and loss will occur.

[0138] Therefore, in order to manage this part of data in a unified manner, all data sources can be based on the data in the metadata center, and if there is any change, each data source channel will be notified. Figure 4 It is a structural table of the storage model of Hbase in the embodiment of the present application. Figure 4 As shown, the metadata model in the metadata management center includes: Hbase row key rules, Hbase row key rules include: Hbase column clusters, Hbase column structures and Hbase column check fields, where Hbase column clusters are used to constrain column clusters in HBase, Hbase column structures are used to constrain column structures in HBase, and Hbase column check fields are used to check columns in HBase. If the metadata model in the metadata management center changes, the metadata management center sends a metadata model change notification to each data source.

[0139] Step 203: Process the data for data import operation according to the metadata model.

[0140] After obtaining the metadata model corresponding to the target Hbase in the metadata management center, the data for data import operation is processed according to the metadata model to write the processed data to the target Hbase. Specifically, in this step, the above-mentioned problem is solved by performing metadata management. Specifically, in the above-mentioned data import process, since the synchronization of multiple data channels may be misaligned, the metadata of the Hbase business layer must be uniformly managed in the metadata center. Among them, the metadata management center is based on TDSQL (Tencent Cloud Database), a reliable financial-grade data source, as a storage medium. No matter how many data source channels there are, this model will prevail. For the specific model, please refer to Figure 4 The model details shown in the figure are as follows. The metadata management center is modified by developers and notifies each subsystem whether any changes occur. Every time the data source channel involves the relevant data source Hbase, it will call the metadata model center to obtain the metadata of the target Hbase table: Hbase rowkey, column cluster, column, etc., and combine it with the code of each channel respectively. Here, it is required to pull the latest metadata from the management center every time.

[0141] Step 204: Perform consistency check on the column definitions corresponding to the data after the data import operation in each data path according to the preset information summary algorithm, and after confirming that the columns of the data written in each data path have the same order, write the processed data to the target Hbase.

[0142] Specifically, in order to ensure a one-to-one correspondence between the data in the Hbase business metadata center and the Hive push columns, Hive deserializes the Json Object from the metadata center during ETL Bulkload to Hbase. There may be inconsistencies in the Json order de-parsed by different Json serialization tools. At the same time, for other channel derivatives, there is also a concern that the read data may be disordered, leading to mapping errors, and ultimately causing data disorder in Hbase. Therefore, the MD5 Message-Digest Algorithm consistency secondary check will be performed on the obtained column definition in each channel to prevent the read data from being misordered due to secondary deserialization.

[0143] In addition, the column Key returned after pulling is the Hbase column name, and the Value is the Hive column name. After each system reads the model Value, it will first perform Json serialization on the Value. However, due to different serialization methods, such as Java's Fastjson, Gson, or Python's Json dump, the positions of Key and Value may change after the object is serialized. In addition, in Hive data processing, Bulkload table production is strictly spliced ​​in the order of fields.

[0144] Therefore, it is necessary to strictly specify the Dimporttsv.columns=HBASE_ROW_KEY,${columns} format, so Hive needs to check whether the columns read through Json are in consistent order and ensure that there are no missing or redundant data fields. Therefore, in the embodiment of the present application, it is necessary to check each field and convert the Hbase fields.

[0145] After each system gets Hbase_columns, it will be converted into json and compared with the check field, such as the Md5 (hbase_columns_check) value and the Md5 (hbase_columns. keyset) value, to ensure that the data we read is correct and orderly, and multiple subsystems can be unified. All systems are based on this definition. Currently, only columns are guaranteed, but there are other similarities such as rowkey rules. With the definition of the management center and unified metadata management, the metadata disorder caused by the inconsistent pace of multiple systems can be well solved. At the same time, after the version of the metadata center is changed, it will be pushed to each relevant person in charge in a timely manner, so as to achieve unified metadata management and achieve unity of knowledge and action.

[0146] Step 205: Read metadata from the metadata center of the metadata management center, and generate data codes according to the metadata.

[0147] Step 206: Package and compile the data code to generate corresponding function classes, which are used to provide business capabilities for the unified metadata management system.

[0148] Step 207: periodically read new metadata from the metadata center of the metadata management center according to a preset time period, and generate a new data code according to the new metadata.

[0149] Step 208: Check whether the first mapping relationship is consistent with the second mapping relationship.

[0150] Step 209: If there is inconsistency, send an alarm message to perform code reconstruction.

[0151] In step 205-step 209, according to the above metadata management steps, the metadata management for the Hbase table level can be achieved by the unified management of the metadata center, and each channel source follows the principle of unified configuration to ensure unique accuracy, but the big data query system QS is actually an interface spitting service, so the data accuracy of QS directly determines the accuracy of the service. Among them, the traditional way is to fix this mapping relationship directly in the code to represent the meta-field information in Hbase, so it is necessary to manually ensure whether the fields are complete, corresponding, misplaced, etc., so there are many uncertain factors, and it is very easy to generate bugs. In this application, the code automatically generates the mapping relationship and checks the accuracy of the fields according to the definition of the metadata center, so as to ensure that our model definition is completely consistent with that of the center. In this application, the Java bean in the big data query system corresponds one-to-one to the table field of Hbase, and the bean in the Service Data Transfer Object (DTO) is provided after relevant conversion according to the business scenario. Therefore, to ensure the consistency of this DTO and metadata, the mapping relationship in the Hbase Domain is obtained.

[0152] At the same time, when the continuous integration (CI) is built, it will automatically check whether the Hbase Domain in the big data query service QS is consistent with the Hbase column in the metadata center. The big data query system pulls the HbaseColumns field and Rowkey rule field of the metadata center through Java Database Connectivity (JDBC) and serializes it through JsonObj. Then, the corresponding metadata information defined by Hbase and hive is obtained, and the Hbase Column check field is read to verify whether the data serialized by Fastjson is accurate. This ensures that the data read out from the metadata center is completely correct, providing conditions for the subsequent automatic generation of code based on the metadata read out. Then, after packaging and compiling, the relevant mapping Domin class can be automatically generated for subsequent business development and automatically generated functional classes.

[0153] In addition, regular reliability inspections can be performed. According to the above steps, the metadata unification problem of multiple heterogeneous data sources can basically be solved, but there is no way to ensure that the code in the subsequent big data query service is consistent with the metadata center. Since multiple channel sources will dynamically go to the metadata management center to obtain the corresponding Hbase table-level metadata each time they are derived to Hbase, there is no problem of regular reliability assurance. It is only necessary in the scenario where the metadata in the metadata is solidified into the system code, such as the big data query system, so it is necessary to configure a scheduled task to periodically check whether the mapping relationship between the Hbase table-level metadata in the metadata and the metadata in the big data query service is normal. If there is an abnormality, a Fata alarm will be sent and code will be built to ensure that the data metadata and the data in the business code in this scenario are consistent.

[0154] On the basis of the above embodiment, after the data code is packaged and compiled to generate the corresponding function class, the data query instruction can be obtained, wherein the above metadata unified management system is used to provide data support for the data query system, and the above function class is used to support the query instruction. Then, the corresponding metadata is read from the metadata center of the metadata management center according to the query instruction.

[0155] In this embodiment, during the data import process, by uniformly managing the metadata model in the metadata management center, any data source channel will use this metadata model as the standard when importing data, thereby effectively solving the technical problem of data alignment in the synchronization of multiple data channels, thereby achieving the technical effect of achieving multi-party uniformity of heterogeneous service metadata for business Hbase. In addition, the technical solution provided by this embodiment has a technical effect of rapid migration and replication for various business systems because the solution mode is unified and migration is convenient.

[0156] The embodiment of the present application also provides a unified metadata management system, including: an Hbase cluster, a metadata management center, and multiple data paths, wherein each data path is implemented by a different code;

[0157] Any data path among the multiple data paths calls the metadata management center during the process of performing a data import operation to a target Hbase, where the target Hbase is a database in the Hbase cluster performing the data import operation;

[0158] Obtain the metadata model corresponding to the target Hbase in the metadata management center;

[0159] The data for the data import operation is processed according to the metadata model to write the processed data to the target Hbase.

[0160] In one possible design, the plurality of data paths include: a first data path, a second data path, and a third data path;

[0161] The first data path and the third data path are used to input real-time data, and the second data path is used to input data before a preset time, so that the data input through the second data path supplements the data input through the first data path or the third data path.

[0162] In a possible design, if the data path for performing the data import operation is the first data path, after the core online system generates transaction data, the core online system sends the transaction data to the preset stream processing platform through an asynchronous task;

[0163] Consume the transaction data in the preset stream processing platform through a preset data source extraction tool, and write the transaction data to the target Hbase according to the metadata model; or,

[0164] If the data path for the data import operation is the second data path, the core online system generates transaction data to the online library, and extracts the transaction data to the preset data warehouse tool in the form of extraction through the preset data transfer tool;

[0165] Cleaning and processing the transaction data through the data warehouse tool, and reading the metadata model for data assembly;

[0166] Write the assembled transaction data into the target Hbase through a preset data import tool; or,

[0167] If the data path for performing the data import operation is the third data path, after the core online system generates transaction data, a preset remote calling tool is called;

[0168] The transaction data is written to the target Hbase through the preset remote calling tool and according to the metadata model.

[0169] In one possible design, the column definitions corresponding to the data after the data import operation in each data path are checked for consistency according to a preset information summary algorithm, and after confirming that the columns of the data written in each data path have the same order, the processed data is written to the target Hbase.

[0170] In a possible design, if the metadata model in the metadata management center changes, the metadata management center sends a metadata model change notification to each data source;

[0171] The metadata model in the metadata management center includes: Hbase row key rules, and the Hbase row key rules include: Hbase column clusters, Hbase column structures, and Hbase column check fields; wherein the Hbase column clusters are used to constrain column clusters in HBase, the Hbase column structure is used to constrain column structures in HBase, and the Hbase column check fields are used to check columns in HBase.

[0172] In a possible design, metadata is read from a metadata center of the metadata management center, and a data code is generated according to the metadata;

[0173] The data codes are packaged and compiled to generate corresponding function classes, and the function classes are used to provide business capabilities for the metadata unified management system.

[0174] In one possible design, the system further includes:

[0175] The data query system is used to obtain data query instructions, the metadata unified management system is used to provide data support for the data query system, and the function class is used to support the query instructions;

[0176] The corresponding metadata is read from the metadata center of the metadata management center according to the query instruction.

[0177] In a possible design, new metadata is periodically read from the metadata center of the metadata management center according to a preset time period, and a new data code is generated according to the new metadata;

[0178] Verify whether the first mapping relationship is consistent with the second mapping relationship, wherein the first mapping relationship is a mapping relationship between the data code and the metadata in the metadata center, and the second mapping relationship is a mapping relationship between the new data code and the metadata in the metadata center;

[0179] If there is any inconsistency, an alarm message is sent to perform code reconstruction.

[0180] Figure 6 is a schematic diagram of the structure of an electronic device according to an exemplary embodiment of the present application. Figure 6 As shown, this embodiment provides an electronic device 300, including:

[0181] processor 301; and,

[0182] A memory 302, used to store executable instructions of the processor, and the memory may also be a flash memory;

[0183] The processor 301 is configured to execute each step in the above method by executing the executable instructions.

[0184] Optionally, the memory 302 may be independent or integrated with the processor 301 .

[0185] When the memory 302 is a device independent of the processor 301, the electronic device 300 may further include:

[0186] The bus 303 is used to connect the processor 301 and the memory 302 .

[0187] This embodiment further provides a readable storage medium, in which a computer program is stored. When at least one processor of an electronic device executes the computer program, the electronic device executes each step in the above method.

[0188] This embodiment also provides a program product, which includes a computer program stored in a readable storage medium. At least one processor of the electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device implements each step in the above method.

[0189] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0190] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for unified metadata management, characterized in that: Applied to a unified metadata management system, the unified metadata management system includes: an Hbase cluster, a metadata management center, and multiple data paths, wherein each data path is implemented by a different code, the method includes: Any data path among the multiple data paths calls the metadata management center during the process of performing a data import operation to a target Hbase, where the target Hbase is a database in the Hbase cluster performing the data import operation; Obtain the metadata model corresponding to the target Hbase in the metadata management center; Processing the data for the data import operation according to the metadata model to write the processed data to the target Hbase; Perform consistency check on the column definitions corresponding to the data after the data import operation in each data path according to a preset information digest algorithm, and after confirming that the columns of the data written in each data path have the same order, write the processed data to the target Hbase; If the metadata model in the metadata management center changes, the metadata management center sends a metadata model change notification to each data source; The metadata model in the metadata management center includes: Hbase row key rules, and the Hbase row key rules include: Hbase column clusters, Hbase column structures, and Hbase column check fields; wherein the Hbase column clusters are used to constrain column clusters in HBase, the Hbase column structures are used to constrain column structures in HBase, and the Hbase column check fields are used to check columns in HBase; Wherein, after processing the data for the data import operation according to the metadata model to write the processed data into the target Hbase, it also includes: Reading metadata from a metadata center of the metadata management center, and generating a data code according to the metadata; Packaging and compiling the data code to generate corresponding function classes, where the function classes are used to provide business capabilities for the metadata unified management system; Acquire a data query instruction, the metadata unified management system is used to provide data support for the data query system, and the function class is used to support the query instruction; Reading corresponding metadata from the metadata center of the metadata management center according to the query instruction; Wherein, after reading the corresponding metadata from the metadata center of the metadata management center according to the query instruction, it also includes: According to a preset time period, periodically read new metadata from the metadata center of the metadata management center, and generate a new data code according to the new metadata; Verify whether the first mapping relationship is consistent with the second mapping relationship, wherein the first mapping relationship is a mapping relationship between the data code and the metadata in the metadata center, and the second mapping relationship is a mapping relationship between the new data code and the metadata in the metadata center; If there is inconsistency, an alarm message is sent to perform code reconstruction, that is, to rebuild the data code for generating the function class.

2. The metadata unified management method according to claim 1, characterized in that: The plurality of data paths include: a first data path, a second data path, and a third data path; The first data path and the third data path are used to input real-time data, and the second data path is used to input data before a preset time, so that the data input through the second data path supplements the data input through the first data path or the third data path.

3. The metadata unified management method according to claim 2, characterized in that: Processing the data for the data import operation according to the metadata model to write the processed data to the target Hbase includes: If the data path for performing the data import operation is the first data path, after the core online system generates transaction data, the core online system sends the transaction data to the preset stream processing platform through an asynchronous task; Consume the transaction data in the preset stream processing platform through a preset data source extraction tool, and write the transaction data to the target Hbase according to the metadata model; or, If the data path for the data import operation is the second data path, the core online system generates transaction data to the online library, and extracts the transaction data to the preset data warehouse tool in the form of extraction through the preset data transfer tool; Cleaning and processing the transaction data through the data warehouse tool, and reading the metadata model for data assembly; Write the assembled transaction data into the target Hbase through a preset data import tool; or, If the data path for performing the data import operation is the third data path, after the core online system generates transaction data, a preset remote calling tool is called; The transaction data is written to the target Hbase through the preset remote calling tool and according to the metadata model.

4. A unified metadata management system, characterized in that: include: Hbase cluster, metadata management center and multiple data paths, where each data path is implemented by different codes; Any data path among the multiple data paths calls the metadata management center during the process of performing a data import operation to a target Hbase, where the target Hbase is a database in the Hbase cluster performing the data import operation; Obtain the metadata model corresponding to the target Hbase in the metadata management center; Processing the data for the data import operation according to the metadata model to write the processed data to the target Hbase; Perform consistency check on the column definitions corresponding to the data after the data import operation in each data path according to a preset information digest algorithm, and after confirming that the columns of the data written in each data path have the same order, write the processed data to the target Hbase; If the metadata model in the metadata management center changes, the metadata management center sends a metadata model change notification to each data source; The metadata model in the metadata management center includes: Hbase row key rules, and the Hbase row key rules include: Hbase column clusters, Hbase column structures, and Hbase column check fields; wherein the Hbase column clusters are used to constrain column clusters in HBase, the Hbase column structures are used to constrain column structures in HBase, and the Hbase column check fields are used to check columns in HBase; Wherein, after processing the data for the data import operation according to the metadata model to write the processed data into the target Hbase, it also includes: Reading metadata from a metadata center of the metadata management center, and generating a data code according to the metadata; Packaging and compiling the data code to generate corresponding function classes, where the function classes are used to provide business capabilities for the metadata unified management system; Acquire a data query instruction, the metadata unified management system is used to provide data support for the data query system, and the function class is used to support the query instruction; Reading corresponding metadata from the metadata center of the metadata management center according to the query instruction; Wherein, after reading the corresponding metadata from the metadata center of the metadata management center according to the query instruction, it also includes: According to a preset time period, periodically read new metadata from the metadata center of the metadata management center, and generate a new data code according to the new metadata; Verify whether the first mapping relationship is consistent with the second mapping relationship, wherein the first mapping relationship is a mapping relationship between the data code and the metadata in the metadata center, and the second mapping relationship is a mapping relationship between the new data code and the metadata in the metadata center; If there is inconsistency, an alarm message is sent to perform code reconstruction, that is, to rebuild the data code for generating the function class.

5. An electronic device, characterized in that: include: processor; as well as a memory for storing a computer program for the processor; The processor is configured to implement the metadata unified management method according to any one of claims 1 to 3 by executing the computer program.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the metadata unified management method according to any one of claims 1 to 3 is implemented.

7. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the metadata unified management method described in any one of claims 1 to 3 is implemented.

Images