Metadata acquisition system based on CDC file

Abstract

The invention discloses a metadata acquisition system based on a CDC file, which comprises a file analysis module for analyzing a change event operation log of a database by using a CDC component, and packaging and writing analyzed operation information into a Kafka message queue; the real-time transmission and analysis module is used for reading the analyzed change event message in real time, identifying and extracting metadata change information in the message, and actively acquiring latest metadata information and update logs in an incremental synchronization mode; the full-text retrieval and alarm module is used for checking whether the current table is still used or not by comparing the table name, the table description, the field name and the field description in the original resource platform; and the service scene and application module is used for providing a RESTful interface and externally sharing metadata information. According to the method, the metadata collection link can be efficiently constructed, the database change event is captured in real time, active collection and updating of the metadata information are achieved, and then the efficiency and accuracy of metadata management are improved.

Description

Technical Field

[0001] The present invention relates to a metadata collection system, in particular to a metadata collection system based on a CDC file. Background Art

[0002] CDC (Change Data Capture) technology is a general term for technologies that capture incremental data. It is mainly used to capture changes in database data, including changes triggered by statements such as DDL (Data Definition Language), DML (Data Manipulation Language), and DCL (Data Control Language). CDC technology is widely used in scenarios such as data backup and disaster recovery, data distribution, and data integration for data warehouses.

[0003] There are many ways to implement CDC technology, including log-based consumption, trigger-based capture, etc. Common CDC technology implementations include open source technologies such as Canal, Maxwell, Debezium, and Flink CDC. These technologies have their own characteristics and can be selected according to specific application scenarios and requirements.

[0004] Canal: An incremental data subscription and consumption component based on MySQL database binlog, providing incremental data subscription and consumption services; Maxwell: It can also read MySQL binlog in real time, generate JSON information, and synchronize data to MQ (such as Kafka); Debezium: It was originally designed as a Kafka connector. It consumes binlog data and delivers it to Kafka. It then relies on Kafka's connector capabilities to output it to other storages. FlinkCDC: A CDC tool based on the Flink ecosystem that integrates Debezium for binlog collection, omitting the MQ part and directly performing calculations and storage through Flink.

[0005] CDC technology is widely used in data backup, disaster recovery, data warehouse construction, real-time data analysis, etc. Through CDC technology, enterprises can efficiently capture and synchronize changed data in the database, realize real-time data update and synchronization, and thus support faster and more accurate business decisions.

[0006] With the rapid development of information technology, the scale of enterprise data has exploded, especially in the era of big data and cloud computing, the amount of data can easily reach tens of billions or even higher. Metadata, as data describing data, is becoming increasingly important. However, in the face of such a huge amount of data, especially when tens of thousands of metadata are updated every day and hundreds of thousands of process tables are generated every month, traditional metadata collection methods are difficult to achieve real-time collection, and affect subsequent data analysis and application. There are mainly the following problems.

[0007] (1) Massive data collection efficiency issues: Faced with the data platform's massive data of tens of billions, the metadata that needs to be updated every day is estimated to reach about 70,000. Traditional metadata collection methods (such as full table scans, scheduled tasks, etc.) are difficult to achieve real-time collection due to processing capacity and efficiency limitations. This not only increases data latency, but may also affect the accuracy of subsequent data analysis and applications; (2) The integrity of data collection: In a massive data environment, how to ensure that the collected metadata is both real-time and highly consistent and complete is another important challenge. Data loss or errors may lead to deviations in subsequent analysis results; (3) Data authorization and access control issues: Metadata applications, such as data authorization and access control, have high requirements for data timeliness. If metadata collection is not timely, data authorization may be delayed, affecting the user's data access experience and data security; (4) Synchronization of metadata application: In the process of metadata application, how to ensure the synchronization and consistency of metadata between different systems or modules is also a key issue. Inconsistent data may lead to data conflicts or analysis errors.

[0008] In addition to the above main issues, we also need to pay attention to the accuracy, security, scalability and other aspects of data collection. Therefore, how to build an efficient and real-time metadata collection system has become a technical problem that needs to be solved in the current data governance field. Summary of the invention

[0009] The technical problem to be solved by the present invention is to provide a metadata collection system based on CDC files, which can efficiently build a metadata collection link, capture database change events in real time, realize active collection and update of metadata information, and thus improve the efficiency and accuracy of metadata management.

[0010] The technical solution adopted by the present invention to solve the above technical problems is to provide a metadata collection system based on CDC files, including a file parsing module: using the CDC component to parse the change event operation log of the database, and encapsulating the parsed operation information into a message of a unified format, and writing it into the Kafka message queue; a real-time transmission and parsing module: reading the parsed change event message from the Kafka message queue in real time, identifying and extracting the metadata change information in the message through the built-in parsing engine, and actively obtaining the latest metadata information and update log in an incremental synchronization manner; a full-text retrieval and alarm module: after obtaining the latest metadata information, comparing it with the table name, table description, field name, and field description in the original resource platform to verify whether the current table is still in use; a service scenario and application module: providing a RESTful interface, sharing metadata information externally, supporting table-level and field-level data authorization, and establishing data exchange tasks.

[0011] Furthermore, the file parsing module obtains the log file that can be consumed in a streaming manner, and the parsed operation information includes the newly added table, deleted table, and modified table structure related to the metadata.

[0012] Furthermore, the log file is a binlog of MySQL, a wal file of PostgreSQL or a redolog of Oracle.

[0013] Furthermore, the metadata change information extracted by the real-time transmission and analysis module includes table name, field name and data type.

[0014] Furthermore, the full-text search and alarm module automatically issues an alarm message for a detected deletion operation and records relevant operation content; and further performs a cleanup or archiving operation on an operation table that has no corresponding resources and is not in use.

[0015] Furthermore, the service scenario and application module also includes data exploration based on metadata information, establishment of a data standard system, and verification of data compliance operations.

[0016] Compared with the prior art, the present invention has the following beneficial effects: the metadata collection system based on CDC files provided by the present invention, by introducing an active collection mechanism, will no longer rely on traditional scheduled tasks or full table scanning methods, but can perceive data changes in real time and immediately trigger collection actions, ensuring that metadata updates can be reflected in the system in real time, significantly improving the efficiency of metadata collection. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic diagram of the metadata collection system based on CDC files of the present invention. DETAILED DESCRIPTION

[0018] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0019] Figure 1 It is a schematic diagram of the metadata collection system based on CDC files of the present invention.

[0020] See also Figure 1 The metadata collection system based on CDC files provided by the present invention includes four core modules: file parsing, real-time transmission and parsing, full-text retrieval and alarm, and service scenarios and applications, as shown below.

[0021] (1) File parsing module The core of the present invention is to use CDC component technology to directly parse the database change event operation log (such as MySQL binlog, Oracle redolog, etc.). These logs record in detail all DML (data manipulation language) and DDL (data definition language) operation changes that occur in the database, including metadata change events such as adding tables, deleting tables, and modifying table structures.

[0022] At the same time, since log files similar to binlog can be consumed in a streaming manner and provide real-time data, the present invention can capture these change events in real time through a customized CDC component to ensure its real-time performance and parse out operation information related to metadata. The parsed operation information will be encapsulated into a message in a unified format and written into the Kafka message queue for subsequent processing.

[0023] (2) Real-time transmission and analysis module The metadata management platform, as the main processing platform of the present invention, is responsible for reading the parsed change event messages from the Kafka message queue in real time. Through the built-in parsing engine, the platform can identify and extract the metadata change information (such as table name, field name, data type, etc.) in the message, and actively obtain the latest metadata information and update log in an incremental synchronization manner. This method avoids the performance burden brought by traditional full table scanning, while ensuring the real-time and accuracy of metadata information.

[0024] (3) Full-text search and alarm module After obtaining the latest metadata information, the present invention also designs a full-text search mechanism. By comparing with the table name, table description, field name, field description, etc. in the original resource platform, it is verified whether the current table is still in use (that is, whether the corresponding resource exists). For the detected deletion operation, the system will automatically issue an alarm message and record the relevant operation content to remind the administrator to intervene in time. If there is no corresponding resource in the asset platform, it is determined that the table is not in use and can be further cleaned or archived.

[0025] (4) Service scenarios and application modules In order to give full play to the value of metadata information, the present invention also provides a rich set of metadata service scenarios and application interfaces. The metadata management platform provides a RESTful interface to share metadata information externally, and supports advanced functions such as table-level and field-level data authorization and the establishment of data exchange tasks. At the same time, based on metadata information, data exploration, establishment of a data standard system, and verification of data compliance can also be performed, providing strong support for the company's data governance and business development.

[0026] In summary, through the close collaboration of these four modules, the present invention aims to achieve real-time capture, analysis and transmission of database change events, as well as active collection and update of metadata information. This technical solution not only improves the efficiency and accuracy of metadata collection, but also provides enterprises with a wealth of metadata service scenarios and application interfaces, injecting new vitality into data governance and business development.

[0027] The metadata collection system based on CDC files provided by the present invention has the following main innovations: (1) Real-time data capture mechanism Firstly, the core of the present invention is to capture the changed data in the database in real time, which means that the system needs to be able to monitor the changes in the data instantly and transmit it to other systems or storage for subsequent processing or analysis.

[0028] Secondly, the real-time capture mechanism can reduce data latency and ensure the real-time and up-to-dateness of data, which is especially important for business scenarios that require rapid response.

[0029] (2) Data change identification technology In order to correctly capture data changes, the system needs to be able to accurately identify which data has changed (added, modified, deleted, etc.) and extract the specific content of the change. At the same time, by accurately identifying data changes, the system can avoid transmitting unchanged data, thereby reducing data redundancy and transmission costs.

[0030] (3) Transaction log parsing capabilities The present invention provides multi-source support. Many database systems (such as MySQL, PostgreSQL, etc.) support transaction logs (such as binlog, WAL, etc.), which record the change history of the database. By parsing these logs, the system can capture almost all data changes.

[0031] Additionally, parsing the transaction log directly is often more efficient than capturing data through database triggers or query interfaces because of the reduced impact on database performance.

[0032] (4) High availability and fault tolerance mechanism In the user production environment, the system most needs to be able to run stably and reliably, even in the face of abnormal situations such as network interruption, hardware failure, etc. Therefore, the present invention has high availability and fault tolerance mechanism to ensure the continuity and integrity of data capture.

[0033] In addition, in the process of capturing and transmitting data changes, the system needs to ensure the consistency and integrity of the data to avoid data loss or corruption due to network problems, system failures, etc.

[0034] (5) Scalability and flexibility Since different application scenarios may have different requirements for CDC technology (such as different data sources, different target systems, different data formats, etc.), by providing a RESTful interface, the external system can easily access and call the metadata service, and the present invention has scalability and flexibility to adapt to different requirements. At the same time, with the continuous development of technology, new database systems and data formats may continue to emerge, and the system needs to be able to be easily expanded and upgraded to support new data sources and data formats.

[0035] The metadata collection system based on CDC files provided by the present invention has the following beneficial effects: (1) Significantly improve the real-time performance of data synchronization: By optimizing the data capture mechanism and transmission process, the present invention can synchronize the changed data in the database to other systems or storage in real time or near real time, significantly reducing data latency.

[0036] Traditional metadata collection cycle: non-real-time, MySQL metadata needs to be collected once a day; This technical innovation results: metadata collection is completed in real time or semi-real time In addition, the improvement in real-time performance is particularly important for business scenarios that require rapid response (such as data authorization, metadata comparison, model synchronization, asset operation and maintenance, etc.), which can ensure the timeliness and up-to-dateness of data.

[0037] (2) Improve the accuracy of data synchronization: The present invention adopts precise data change identification technology, which can accurately capture data changes (addition, modification, deletion, etc.) and extract the specific content of the changes, reducing data redundancy and errors. Through the Kafka message queue and other methods, the integrity and consistency of data changes are further ensured, and the accuracy of data synchronization is improved.

[0038] Traditional collection technology: If the collection fails, the data will be lost and needs to be collected again, which is time-consuming and laborious; The innovative result of this technology: If consumption fails, you can consume again, and if consumption succeeds, the data will not be lost.

[0039] (3) Enhance the scalability and flexibility of the system: The present invention is designed with scalability and flexibility in mind, and can easily adapt to the requirements of different data sources, target systems, and data formats, which means that the system can be customized and expanded according to business needs without the need for large-scale reconstruction or replacement, greatly reducing the maintenance cost and upgrade difficulty of the system.

[0040] Although the present invention has been disclosed as above with preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art may make some modifications and improvements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be based on the definition of the claims.

Claims

1. A metadata collection system based on CDC files, characterized in that: include: File parsing module: uses the CDC component to parse the database change event operation log, encapsulates the parsed operation information into a message in a unified format, and writes it to the Kafka message queue; Real-time transmission and parsing module: reads parsed change event messages from the Kafka message queue in real time, identifies and extracts metadata change information in the message through the built-in parsing engine, and actively obtains the latest metadata information and update logs in an incremental synchronization manner; Full-text search and alarm module: After obtaining the latest metadata information, it compares the table name, table description, field name, and field description in the original resource platform to verify whether the current table is still in use; Service scenarios and application modules: Provide RESTful interfaces, share metadata information externally, support table-level and field-level data authorization, and establish data exchange tasks.

2. The metadata collection system based on CDC files as claimed in claim 1, characterized in that: The file parsing module obtains the log file that can be consumed in a streaming manner, and the parsed operation information includes the newly added table, deleted table, and modified table structure related to the metadata.

3. The metadata collection system based on CDC files as claimed in claim 2, characterized in that: The log file is a binlog of MySQL, a wal file of PostgreSQL or a redolog of Oracle.

4. The metadata collection system based on CDC files as claimed in claim 1, characterized in that: The metadata change information extracted by the real-time transmission and analysis module includes table name, field name and data type.

5. The metadata collection system based on CDC files as claimed in claim 1, characterized in that: The full-text search and alarm module automatically issues an alarm message for the detected deletion operation and records the relevant operation content; for the operation table without corresponding resources and not being used, further cleaning or archiving operations are performed.

6. The metadata collection system based on CDC files as claimed in claim 1, characterized in that: The service scenarios and application modules also include data exploration based on metadata information, establishment of a data standard system, and verification of data compliance operations.

Images