ETL full-increment dynamic mapping data integration system and method

Abstract

The invention discloses an ETL full-increment dynamic mapping data integration system and method, and the system comprises a data source configuration module which is used for configuring data source connection information and verifying the connectivity of a source database and a target database; the log collection module is used for deploying a log collection tool to monitor file change of the source database and capture a data change event of the source database; the full data updating module is used for triggering full data updating operation, acquiring all logs stored by the log collection tool from the message queue middleware, and updating existing data in the source database to the target database; and the incremental data updating module is used for receiving a new data event from the message queue middleware after the existing data in the source database is updated, comparing change log position information recorded locally, obtaining updated operation information from the new data, obtaining an updating statement by using a dynamic mapping updating engine and executing the updating statement. According to the invention, the real-time performance and the accuracy during data updating are guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and more specifically to an ETL full-increment dynamic mapping data integration system and method. Background Art

[0002] With the rapid development of information technology, data has become an important asset for enterprise decision-making. Enterprises need to extract, transform and load data from various data sources into data warehouses or data lakes, a process called ETL, to support data analysis and business intelligence. Traditional ETL methods usually use full data loading, that is, all data in the data source is copied to the target system regularly, regardless of whether the data has changed. Although this method is simple, it has obvious shortcomings when processing large-scale data sets:

[0003] 1. High resource consumption: Full data loading requires processing and transmitting a large amount of data, consuming a lot of computing resources and storage space.

[0004] 2. Inefficiency: When the data does not change much, full load still needs to process the entire data set, which increases the processing time.

[0005] 3. Poor data timeliness: Since full loading is usually performed periodically, newly generated data cannot be reflected in the target system in a timely manner.

[0006] 4. Data consistency issue: During the data loading process, if the data source changes, the data in the target system may be inconsistent with the source data.

[0007] In order to solve the above problems, the incremental ETL method came into being. Incremental ETL only processes the data that has been added or changed since the last ETL operation, thereby reducing resource consumption, improving processing efficiency, and enhancing the timeliness of data. However, the existing incremental ETL method also faces some challenges:

[0008] 1. Difficulty in identifying incremental data: Accurately identifying incremental data is a challenge without clear change identification.

[0009] 2. Data consistency maintenance: In a distributed system, how to ensure the consistency of incremental data is a technical challenge.

[0010] 3. Performance optimization: When processing large amounts of incremental data, how to optimize performance to meet real-time or near real-time data processing requirements.

[0011] 4. Fault tolerance and recovery: In the ETL process, how to achieve high availability and fault tolerance mechanisms to deal with problems such as node failures.

[0012] Although some technologies have been tried to solve these problems, they usually rely on specific data sources or system architectures and lack versatility and flexibility. In addition, existing methods still face performance bottlenecks and data consistency issues when dealing with large-scale and highly dynamic data sets. Summary of the invention

[0013] In view of this, the present invention provides an ETL full-incremental dynamic mapping data integration system and method, which are used to at least solve some of the technical problems in the background technology.

[0014] In order to achieve the above object, the present invention adopts the following technical solution:

[0015] The present invention discloses an ETL full-incremental dynamic mapping data integration system, comprising:

[0016] The data source configuration module is used to configure the data source connectivity information and verify the connectivity between the source database and the target database;

[0017] The log collection module is used to deploy log collection tools and dynamically generate log collection configuration files to monitor file changes in the source database and capture data change events in the source database;

[0018] The full data update module is used to trigger the full data update operation, obtain all logs saved by the log collection tool from the message queue middleware, and update the existing data in the source database to the target database through the block transmission mechanism;

[0019] The incremental data update module is used to update the existing data in the source database, receive new data events from the message queue middleware, compare the change log location information recorded locally, obtain the updated operation information from the new data, use the dynamic mapping update engine to obtain the update statement, and execute the update statement.

[0020] Preferably, it also includes a visual module, which is used to view the operation records in the data synchronization process and the data in the source database and the target database.

[0021] Preferably, in the data source configuration module, configuring the data source connectivity information specifically includes:

[0022] Configure the source database's host address, port, user name, password, and database name;

[0023] Configure the connection address, user name, password, and name of the target database.

[0024] Preferably, in the log collection module, the following steps are specifically performed:

[0025] Deploy the log collection tool Flume and configure Flume's source component, channel component, and sink component;

[0026] The source component is used to monitor the changes in the change log of the source database and capture data change events;

[0027] The channel component is used to temporarily store captured data change events;

[0028] The sink component is used to transfer the data change events temporarily stored in the channel component to the target database.

[0029] Preferably, the system further comprises a field change processing module, which is used to process the table structure fields before the incremental data update module performs data update, and specifically comprises the following operations:

[0030] Use the dynamic mapping update engine in the data update module to predefine the table structure fields of the source database;

[0031] Based on the mapping configuration of the source database and the target database, the preset table structure fields in the source database are extracted and interpolated into the MongoDB table creation statement to generate an executable table creation statement for the target database.

[0032] Another aspect of the present invention discloses an ETL full-increment dynamic mapping data integration method, comprising the following steps:

[0033] Configure data source connectivity information and verify the connectivity between the source and target databases;

[0034] After the connectivity verification is passed, the log collection tool is deployed and the log collection configuration file is dynamically generated to monitor the file changes of the source database and capture the data change events of the source database;

[0035] The data change events of the source database are transmitted to the data update module through the message queue middleware for data update, including full data update and incremental data update;

[0036] The full data update specifically includes:

[0037] Trigger a full data update operation, obtain all logs saved by the log collection tool from the message queue middleware, and update the existing data in the source database to the target database through the block transmission mechanism;

[0038] Incremental data updates specifically include:

[0039] After the existing data in the source database is updated, new data events are received from the message queue middleware, the change log location information recorded locally is compared, the updated operation information is obtained from the new data, the dynamic mapping update engine is used to obtain the update statement, and the update statement is executed.

[0040] Furthermore, it also includes viewing the operation records during the data synchronization process and the data in the source database and the target database through a visual interface.

[0041] Further, configuring data source connectivity information includes:

[0042] Configure the source database's host address, port, user name, password, and database name;

[0043] Configure the connection address, user name, password, and name of the target database.

[0044] Furthermore, a log collection tool is deployed, and a log collection configuration file is dynamically generated to monitor file changes in the source database and capture data change events in the source database. Specifically, the following steps are included:

[0045] Deploy the log collection tool Flume and configure Flume's source component, channel component, and sink component;

[0046] The source component is used to monitor the changes in the change log of the source database and capture data change events;

[0047] The channel component is used to temporarily store captured data change events;

[0048] The sink component is used to transfer the data change events temporarily stored in the channel component to the target database.

[0049] Furthermore, the data change event of the source database is transmitted to the data update module by using the message queue middleware for data update, which specifically includes the following steps:

[0050] Use the dynamic mapping update engine in the data update module to predefine the table structure fields of the source database;

[0051] Based on the mapping configuration of the source database and the target database, extract the preset table structure fields in the source database and interpolate them into the MongoDB table creation statement to generate an executable table creation statement for the target database;

[0052] Execute the table creation statement of the target database to create the table structure of the target database.

[0053] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses an ETL full-incremental dynamic mapping data integration system and method, which has the following beneficial effects:

[0054] Real-time: The log collection module is used to monitor data source changes in real time, and the changed data is quickly transmitted through the message middleware to ensure real-time synchronization of data.

[0055] Accuracy: The introduction of data version control, transactional update operations and data verification mechanisms ensures the accuracy and consistency of each data update.

[0056] Automation: Automated data source configuration, log collection, data synchronization status tracking, and field change processing reduce manual intervention and operational errors.

[0057] Data consistency maintenance: Data consistency between the source database and the target database is ensured through data snapshot technology, block transmission mechanism, data verification and validation, and data consistency check.

[0058] Fault tolerance and recovery capabilities: An intelligent error handling and retry mechanism, as well as a data backup and recovery strategy, are designed to improve the system's fault tolerance and stability.

[0059] Fine-grained data synchronization: Implements fine-grained data synchronization, such as synchronizing only changed rows or columns instead of the entire table, which improves synchronization efficiency.

[0060] Friendly user interaction: Provides a friendly graphical user interface that enables users to easily configure and manage ETL processes, monitor data synchronization progress and operation records.

[0061] Monitoring and alarm: A real-time monitoring and alarm mechanism is implemented, which can promptly notify the administrator when an abnormality occurs, thereby improving the system's response speed.

[0062] Automatic mapping method: Compared with other existing solutions, the present invention provides a method for automatically mapping table structure, which reduces the workload of manual configuration mapping. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0064] Figure 1 A schematic diagram of the system structure of the ETL full-incremental dynamic mapping data integration provided by the present invention.

[0065] Figure 2 This is an example diagram of the dynamic mapping configuration provided by the present invention. DETAILED DESCRIPTION

[0066] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] On the one hand, the present invention discloses a system for ETL full incremental dynamic mapping data integration, referring to Figure 1 , each module of the system is introduced in detail.

[0068] 1. Data source configuration module:

[0069] Provide a friendly graphical interface for users to input the source database and target database address, log storage path and other configuration information. The system automatically detects the connectivity of the database connection to ensure the accessibility of the data source and target system. Provide an automated tool to scan the table structure in the source database to provide basic information for subsequent data mapping and synchronization.

[0070] 2. Log collection module:

[0071] Use log collection tools such as Flume to remotely deploy and dynamically generate configuration files to monitor file changes in data sources. Enable Flume to monitor the file system in real time, capture data change events, and transmit the change data to downstream processing modules through message middleware such as Kafka. The log receiving module pre-processes the received log data, divides the operations according to different operation types in different data sources, extracts key information, and provides a basis for in-depth processing of subsequent log data.

[0072] 3. Full update module:

[0073] When the first ETL operation or major changes occur in the data source structure, perform a full data scan and synchronization. After the full update, the system records the last synchronization status of the data to provide a benchmark for subsequent incremental updates. In the present invention, major changes occur in the data source structure, generally referring to adding or deleting table fields, and the table structure has a significant change. This situation will only occur if it is manually operated. The next step of the operation can be processed according to the user's choice, for example, choose to clear the existing data for a full update. Or choose to map new fields for incremental updates.

[0074] Full updates often involve large amounts of data and are slow to update. This solution uses the following technologies to ensure data integrity and improve update speed. Data snapshot technology: Before executing a full update, use data snapshot technology to take a consistent snapshot of the source database to ensure the stability and consistency of the source data during the data replication process. Block transmission mechanism: In the face of large amounts of data, the data is transmitted in blocks, and each block of data is processed independently to reduce the risk of failure of the entire data load due to network fluctuations or system failures. Data verification and validation: During data transmission, implement data verification mechanisms, such as CRC verification or MD5 verification, to ensure the integrity and accuracy of data during transmission. Multithreaded parallel processing: Use multithreading technology to process data blocks in parallel to improve data loading speed while reducing the impact on the performance of the source database. Data backup and recovery strategy: During the full update process, implement a data backup strategy to ensure that it can be quickly restored to its original state when the update fails to avoid data loss. Incremental logging: Record the data change log during the full update process so that when an interruption occurs during the update process, it can be restored from the interruption point instead of restarting the full update. Data consistency check: After the full update is completed, a data consistency check is performed to compare the data in the source database and the target database to ensure data consistency. Error handling and retry mechanism: An intelligent error handling and retry mechanism is designed. For recoverable errors encountered during data loading, the system can automatically retry to reduce update failures caused by temporary problems. User notification and log recording: During the full update process, real-time user notifications are provided, including progress updates, error reports, etc., and detailed operation logs are recorded to provide a basis for subsequent problem analysis and system optimization.

[0075] 4. Incremental update module:

[0076] Based on the data change information provided by the log collection module, the incremental update module only processes data changes since the last ETL operation. Through the operation information in the log collection module, the newly added, modified and deleted data records are accurately distinguished to avoid repeated data processing. After the incremental data is converted, efficient data loading technology is used to update the data in the target system to ensure the timeliness of the data. This module also adopts the following solutions to ensure the accuracy of the data update process. Data version control: Introduce a data version control mechanism to ensure that each update is based on the latest state of the data and avoid overwriting the results of other concurrent operations. Transactional update operation: Encapsulate the incremental update operation in a database transaction to ensure the atomicity of each data update, either completely successful or completely failed, thereby maintaining data consistency. Conflict detection and resolution strategy: Design a conflict detection mechanism. When a data version conflict or concurrent update is detected, it automatically or prompts the user to intervene to resolve the conflict according to the predetermined business rules. Data integrity constraints: Implement data integrity constraints in the target database, such as foreign key constraints, check constraints, etc., to ensure that the loaded data meets all business rules. Fine-grained data synchronization: For data tables with complex structures, fine-grained data synchronization is implemented, such as synchronizing only the changed rows or columns instead of the entire table, to improve synchronization efficiency and accuracy. Data verification mechanism: After the data is loaded into the target system, data verification is performed, such as verifying the integrity and accuracy of the data through methods such as data comparison and data summary. Persistent storage of incremental logs: The incremental update logs are persistently stored to record the details of each synchronization, including the amount of synchronized data, timestamp, change type, etc., to provide a basis for data tracking and troubleshooting. Idempotence design of data synchronization: Ensure that the incremental update operation is idempotent, that is, the same update operation has the same result after multiple executions, avoiding data inconsistency caused by repeated execution. Synchronization status tracking of the data source and the target system: The synchronization status between the data source and the target system is recorded using the synchronization status tracking mechanism, so that synchronization can be restored from the correct point in the event of a failure. Abnormal monitoring and alarm mechanism: Implement abnormal monitoring. When abnormal conditions are detected during data synchronization, an alarm is triggered in a timely manner to notify the administrator to intervene. Data backup strategy: Before performing incremental updates, automatically create a backup of the data to prevent errors during the update process and quickly restore to the state before the update.

[0077] 5. Field change processing module:

[0078] After full data synchronization, the incremental update module normally updates the incremental data. However, there may be changes in the table structure, so the field change module is required to synchronize the table structure before the incremental update can run normally. The field change module is independent of the incremental update module and is responsible for providing an accurate table structure environment for incremental updates. Its execution priority is higher than that of incremental updates.

[0079] Based on the operation information in the log module, this module can obtain the metadata change information of the source database, including the addition, deletion or modification of fields. In this regard, this module develops a dynamic mapping update engine. When a field change is detected, the engine can automatically identify the change and update the mapping configuration without manual intervention. The specific implementation ideas are as follows: pre-define the table structure operation statements of common databases, including table creation, table deletion, field addition, field deletion, field attribute modification, etc., and reserve slots to prepare for subsequent interpolation. According to the operation information in the log module, obtain the metadata type of the source database. Table operation type and other information. Match the corresponding table structure operation statement based on this information. Use string interpolation to replace the information to be operated into the table structure operation statement. Obtain the connection information of the target database from the data source module and execute the table structure operation statement in the target database.

[0080] For the operation of adding new fields in field changes, dynamic mapping can be performed only once. For deletion operations, in addition to dynamic mapping, this system adopts the strategy of retaining historical fields for old data to avoid historical data loss, and records operation logs at the same time. Users can process old data according to the logs. For modified fields, in addition to dynamic mapping, it is necessary to analyze the impact on the existing ETL process. Including which conversion logic needs to be updated synchronously.

[0081] Based on the above system, the present invention also discloses a method for ETL full-incremental dynamic mapping data integration. The following takes the synchronization of MySQL source data to MongoDB target database as an example to describe the specific implementation process:

[0082] Step 1: Configure the data source information. The user enters the connection information of the MySQL source database through the graphical interface, including the host address, port, user name, password, and database name. The system automatically executes the detection script to verify the connectivity of the MySQL database and lists all available data tables for the user to select as the data source. Configure the target database and select the database type. Here, select MongoDB. Configure the connection information, including the connection address, user name, password, and target database name. Verify connectivity.

[0083] Step 2: Configure the log collection module, deploy Apache Flume on the log collection server, and configure Flume's agent, including source, channel, and sink components. Configure the source component org.apache.flume.source.mysql.MySQLSource, which is a source specifically used to listen to MySQL binlog. Execute the following configuration through an automated script to enable MySQL binlog by setting log_bin = 'mysql-bin.log' and server_id = 1 (or other unique ID) in the my.cnf or my.ini configuration file. Adjust the binlog format and use binlog_format = 'ROW' to capture row-level changes. Based on the MySQL data source connection information entered by the user, automatically configure Flume's MySQL source, specify the JDBC URL, user name, and password to connect to MySQL, and set the source component's binlog.file and binlog.position to point to the binlog file and position recorded after the last ETL operation. Configure the source component to filter specific databases or tables, capture table structure, data addition, deletion, modification, and other change events, enable Flume's event monitoring mechanism, monitor MySQL binlog file changes in real time, and capture data change events. Temporarily store the captured change events through Flume's channel component. The channel can be configured as memory or file system. Implement preprocessing logic in the sink component, remove unnecessary log header information, add data source MySQL information, read MySQL-related operation configuration from the pre-defined log operation configuration, obtain the operation information in the log configuration according to the MySQL event name match, and append the operation information to the end of the line log to provide a basis for subsequent process processing. Continue to serialize the log data into avro format in the sink component and transmit it to the downstream processing module through Kafka or other message queue systems. In addition, configure relevant security and accuracy configurations, such as configuring Flume's transactional mechanism to ensure data integrity and restore from the last submitted point even in the event of a failure. Set Flume's ack mechanism to ensure that the data is considered processed only after it is successfully received by the downstream sink. Set the data transmission between Flume's source and sink to use SSL encryption to ensure the security of data during transmission. Configure the alarm mechanism to notify the administrator in a timely manner when the amount of captured data is abnormal or the processing is delayed.

[0084] Step 3: Table structure mapping update. Before the full incremental update, the table structure of the target database is mapped first. In this case, the source MySQL database is synchronized to the target MongoDB database. After the dynamic mapping update engine obtains the table structure metadata information, it extracts the table structure fields in the source database through the mapping configuration based on its own basic mapping configuration from MySQL to MongoDB, and interpolates them into the MongoDB table creation statement to generate an executable MongoDB table creation statement. The table creation statement is executed to create the table structure. The subsequent incremental updates are the same. The basic mapping configuration is obtained, and the MongoDB update statement is constructed according to the configuration. The update logic is executed, and the existing ETL process is analyzed to update the process information. A specific dynamic mapping configuration code example is as follows: Figure 2 shown.

[0085] Step 4: Fully update existing data. After the user triggers a full data update, the system obtains all operation logs saved by Flume from Kafka or other message queues, records snapshots, and saves information such as the location of the current binlog to provide a basis for subsequent incremental updates. After obtaining the full operation log, the log is processed in blocks, and the dynamic mapping update engine is used to obtain the update statement. Multi-threaded concurrent data is used to update each block according to the situation. The results of each transaction execution are recorded. If a failure occurs, it is rolled back and re-executed to ensure data accuracy. After multiple execution failures, an alarm is triggered and the alarm information is sent back to the user for processing. After all blocks are executed, data verification is performed to compare the number of rows, checksum, etc. of the source database and the target database to ensure data consistency.

[0086] Step 5: Incrementally update the data. After the full update is completed, the incremental update module receives new data events from message middleware such as Kafka, compares the binlog position information recorded locally, obtains the updated operation information from the new data, uses the dynamic mapping update engine to obtain the update statement, and executes the update statement. During the update process, the incremental update operation is encapsulated in the database transaction to ensure the atomicity of each update. If the update fails, the transaction is re-executed. If the update fails multiple times, an alarm is triggered and the alarm information is sent to the user for processing. Implement a concurrency control mechanism to allow multiple incremental update operations to be executed in parallel while preventing conflicts between them. After the data is updated to the target database, verify the integrity and accuracy of the data through methods such as data comparison and data summary. Record the binlog position of this update, record the version number, and timestamp to ensure that only the latest update is applied.

[0087] In this step, if the user operation interface changes the table structure, the field update module will process the fields according to the information in the log operation, and then incrementally update the data.

[0088] Step 6: Users can view the data synchronization progress and related operation records during the data synchronization process through the interface, and can also roll back data based on the operation records. Data in the source database and target database can be viewed.

[0089] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0090] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An ETL full-incremental dynamic mapping data integration system, characterized in that: include: The data source configuration module is used to configure the data source connectivity information and verify the connectivity between the source database and the target database; The log collection module is used to deploy log collection tools and dynamically generate log collection configuration files to monitor file changes in the source database and capture data change events in the source database; The full data update module is used to trigger the full data update operation, obtain all logs saved by the log collection tool from the message queue middleware, and update the existing data in the source database to the target database through the block transmission mechanism; The incremental data update module is used to update the existing data in the source database, receive new data events from the message queue middleware, compare the change log location information recorded locally, obtain the updated operation information from the new data, use the dynamic mapping update engine to obtain the update statement, and execute the update statement.

2. The ETL full-increment dynamic mapping data integration system according to claim 1, characterized in that: It also includes a visual module, which is used to view the operation records in the data synchronization process and the data in the source database and the target database.

3. The ETL full-incremental dynamic mapping data integration system according to claim 6 is characterized in that: In the data source configuration module, configuring the data source connectivity information specifically includes: Configure the source database's host address, port, user name, password, and database name; Configure the connection address, user name, password, and name of the target database.

4. The ETL full-increment dynamic mapping data integration system according to claim 1, characterized in that: In the log collection module, the following steps are specifically performed: Deploy the log collection tool Flume and configure Flume's source component, channel component, and sink component; The source component is used to monitor the changes in the change log of the source database and capture data change events; The channel component is used to temporarily store captured data change events; The sink component is used to transfer the data change events temporarily stored in the channel component to the target database.

5. The ETL full-increment dynamic mapping data integration system according to claim 1, characterized in that: It also includes a field change processing module, which is used to process the table structure fields before the incremental data update module executes data update, and specifically includes the following operations: Use the dynamic mapping update engine in the data update module to predefine the table structure fields of the source database; Based on the mapping configuration of the source database and the target database, the preset table structure fields in the source database are extracted and interpolated into the MongoDB table creation statement to generate an executable table creation statement for the target database.

6. An ETL full-incremental dynamic mapping data integration method, characterized in that: The following steps are involved: Configure data source connectivity information and verify the connectivity between the source and target databases; After the connectivity verification is passed, the log collection tool is deployed and the log collection configuration file is dynamically generated to monitor the file changes of the source database and capture the data change events of the source database; The data change events of the source database are transmitted to the data update module through the message queue middleware for data update, including full data update and incremental data update; The full data update specifically includes: Trigger a full data update operation, obtain all logs saved by the log collection tool from the message queue middleware, and update the existing data in the source database to the target database through the block transmission mechanism; Incremental data updates specifically include: After the existing data in the source database is updated, new data events are received from the message queue middleware, the change log location information recorded locally is compared, the updated operation information is obtained from the new data, the dynamic mapping update engine is used to obtain the update statement, and the update statement is executed.

7. The ETL full-incremental dynamic mapping data integration method according to claim 6 is characterized in that: It also includes viewing the operation records during the data synchronization process, as well as the data in the source and target databases through a visual interface.

8. The ETL full-incremental dynamic mapping data integration method according to claim 6 is characterized in that: Configure data source connectivity information, including: Configure the source database's host address, port, user name, password, and database name; Configure the connection address, user name, password, and name of the target database.

9. The ETL full-incremental dynamic mapping data integration method according to claim 6, characterized in that: Deploy the log collection tool and dynamically generate a log collection configuration file to monitor file changes in the source database and capture data change events in the source database. The specific steps include: Deploy the log collection tool Flume and configure Flume's source component, channel component, and sink component; The source component is used to monitor the changes in the change log of the source database and capture data change events; The channel component is used to temporarily store captured data change events; The sink component is used to transfer the data change events temporarily stored in the channel component to the target database.

10. The ETL full-incremental dynamic mapping data integration method according to claim 6, characterized in that: The data change events of the source database are transmitted to the data update module by using the message queue middleware for data update, which specifically includes the following steps: Use the dynamic mapping update engine in the data update module to predefine the table structure fields of the source database; Based on the mapping configuration of the source database and the target database, extract the preset table structure fields in the source database and interpolate them into the MongoDB table creation statement to generate an executable table creation statement for the target database; Execute the table creation statement of the target database to create the table structure of the target database.

Images