Method for realizing incremental migration by network packet capture analysis mechanism

Abstract

The invention discloses a method for realizing incremental migration by a network packet capture analysis mechanism, which comprises the following steps of: 1, network packet capture: monitoring network communication between a database and an application, and capturing incremental data in real time; and step 2, data change identification: identifying data change by analyzing the captured network packet. And 3, extracting and processing incremental data: analyzing the captured data, establishing a data change record, and further classifying and extracting the incremental data needing to be migrated. And 4, data transmission and write-in: sending and writing the incremental data into a target database or a data storage system through a transmission and write-in processing tool. And 5, logging and monitoring: tracking each link in the data migration process, recording and monitoring system logs, and finding and solving problems in time. According to the method, the intrusion to the source database is reduced, and the operation and use of the original business are not influenced.

Description

Technical Field

[0001] The present invention relates to the technical field of database incremental migration, and in particular to a method for implementing incremental migration through a network packet capture and parsing mechanism. Background Art

[0002] In the prior art, there are generally two technical solutions for database incremental data migration: Patent CN118410027A discloses an efficient method for migrating incremental data from a database. The method obtains incremental data and writes the incremental data into a database, divides the acquired incremental data into different data table groups according to target library tables, and uses multi-threading to traverse and group the data in these data table groups into single-table data. The method extracts individual incremental data from the data table groups one by one. The method distinguishes the type of the extracted individual incremental data, and places the individual incremental data into a newly added group, a deleted group, or a changed group in order according to the type of the individual incremental data. The method repeats the above process. The method can effectively solve the problem that there are multiple data types of incremental data in the existing data incremental migration batches at the same time, and the order of the same data type is relatively scattered. When writing, the database driver switches between different types of precompilers, and the execution efficiency is relatively slow.

[0003] Patent CN116975159A is a processing method for incremental data synchronization, including creating a change record tracking table corresponding to the table to be synchronized; creating a trigger on the table to be synchronized, and writing the change record of the table to be synchronized into the change record tracking table through the trigger; creating an inline table-valued function of SQL Server, based on the change record tracking table and reading the incremental data of the table to be synchronized through the inline table-valued function; generating and running synchronization tasks, synchronizing the incremental data to the message queue; consuming and parsing the data in the message queue, obtaining the schema information of the data, generating the corresponding insert statement according to the schema information, and writing the insert statement into the target database. This patent does not require the deployment of additional agent services, does not require the storage of data before and after data changes, and reduces the pressure on the database; it can synchronize the data addition, deletion, and modification scenarios in quasi-real time, and does not require the existence of a timestamp field in the business table.

[0004] In summary, the above existing technologies all rely on the source database table to obtain and process incremental data. The former is processed through multiple threads and different data table groups, and then single incremental data is extracted one by one. The latter is processed through triggers and tracked through change record tracking tables. They all need to add corresponding operations on the source database and open corresponding database operation permissions, which is somewhat invasive. In addition, when the amount of data is large, this operation will occupy a large amount of source database resources, affecting the original business. Summary of the invention

[0005] The purpose of the present invention is to provide a network packet capture and analysis mechanism to realize an incremental migration method, which does not require additional operations on the source database and directly intercepts and analyzes data on network requests, thereby achieving the purpose of not occupying the resources of the source database, reducing the intrusion into the source database, and not affecting the operation and use of the original business, thereby solving the problems raised in the above-mentioned background technology.

[0006] The technical solution adopted by the present invention is: A method for implementing incremental migration using a network packet capture and parsing mechanism comprises the following steps: Step 1: Network packet capture: monitor the network communication between the database and the application, and capture incremental data in real time; Step 2: Data change identification: Identify data changes by parsing captured network packets; Step 3: Extract and process incremental data: After the captured data is parsed, a data change record is created, and the incremental data that needs to be migrated is further classified and extracted; Step 4: Data transmission and writing: The incremental data is sent and written to the target database or data storage system through the transmission and writing processing tool; Step 5: Log and monitor: Track each step of the data migration process and perform system log recording and monitoring to identify and resolve problems in a timely manner.

[0007] Furthermore, in step 1, based on Java+Pcap4J, by configuring the interface addresses of the application and the database, the network communication between the database and the application is monitored to capture the incremental data.

[0008] Furthermore, in step 2, in combination with the Java.net library, the original communication message is disassembled according to the pre-configured database message template to disassemble the change action executed by the current message and the original data information.

[0009] Furthermore, the specific steps of step 2 are as follows: Step 2-1, parse the captured network packets according to the database communication protocol (such as MySQL Statement, PostgreSQL Query, etc.) to identify specific incremental data changes; Step 2-2, determine whether the current incremental data change is valid data; if so, execute step 3; otherwise, discard and record it in the system log and then execute step 1.

[0010] Furthermore, the data captured in step 3 is parsed and saved in JSON format, and a data change record is created. Duplicate data changes are filtered by using the Java Set framework. The incremental data that needs to be migrated is further extracted according to the data change type classification, and the incremental data set is generated according to the rule of converting the UPDATE action into DELETE+INSERT.

[0011] Furthermore, in step 4, based on the JDBC protocol and database batch processing technology, combined with Springboot concurrency control, the incremental data can be quickly sent and written to the target database or data storage system.

[0012] Furthermore, in step 4, the data migration rate is adjusted according to the load condition of the target system.

[0013] Furthermore, in step 5, the log framework Log4j is used to generate detailed log files for each stage and each link of the operation, recording the timestamp, incremental change type, and key information of the transmission result; at the same time, the Java scheduled task scheduling library is used to regularly check the system incremental change monitoring status and send an alarm when a problem is found.

[0014] Furthermore, in step 5, a unified management platform is constructed to centrally monitor the migration status of all data sources.

[0015] The present invention adopts the above technical solution, which has the following significant advantages and positive effects: 1. Reduce the intrusion to the source database: Compared with the prior art method that relies on the source database table to obtain and process incremental data, the present invention uses network packet capture to capture data changes, without the need to implement additional operations or open permissions on the source database, thereby reducing the intrusion to the original system. 2. Reduce the occupation of database resources: When processing large amounts of data, the prior art will significantly occupy the resources of the source database and affect the original business flow. The present invention captures incremental data in real time and only operates at the network level, which significantly reduces the pressure on the source database. 3. Enhance the flexibility and adaptability of the system: The present invention is not limited by the structure of the source database, can adapt to various types of databases and data changes, and has a high degree of flexibility. In various scenario tests, the technical solution successfully supported data of different formats and types without any modification to the source database. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments; Figure 1 The present invention is a flowchart of a method for implementing incremental migration using a network packet capture and analysis mechanism. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0018] like Figure 1 As shown, the present invention discloses a method for realizing incremental migration by using a network packet capture and parsing mechanism, which comprises the following steps: Step 1: Network packet capture: monitor the network communication between the database and the application, and capture incremental data in real time; Step 2: Data change identification: Identify data changes by parsing captured network packets; Step 3: Extract and process incremental data: After the captured data is parsed, a data change record is created, and the incremental data that needs to be migrated is further classified and extracted; Step 4: Data transmission and writing: The incremental data is sent and written to the target database or data storage system through the transmission and writing processing tool; Step 5: Log and monitor: Track each step of the data migration process and perform system log recording and monitoring to identify and resolve problems in a timely manner.

[0019] Furthermore, in step 1, based on Java+Pcap4J, by configuring the interface addresses of the application and the database, the network communication between the database and the application is monitored to capture the incremental data.

[0020] Furthermore, in step 2, in combination with the Java.net library, the original communication message is disassembled according to the pre-configured database message template to disassemble the change action executed by the current message and the original data information.

[0021] Furthermore, the specific steps of step 2 are as follows: Step 2-1, parse the captured network packets according to the database communication protocol (such as MySQL Statement, PostgreSQL Query, etc.) to identify specific incremental data changes; Step 2-2, determine whether the current incremental data change is valid data; if so, execute step 3; otherwise, discard and record it in the system log and then execute step 1.

[0022] Furthermore, the data captured in step 3 is parsed and saved in JSON format, and a data change record is created. Duplicate data changes are filtered by using the Java Set framework. The incremental data that needs to be migrated is further extracted according to the data change type classification, and the incremental data set is generated according to the rule of converting the UPDATE action into DELETE+INSERT.

[0023] Furthermore, in step 4, based on the JDBC protocol and database batch processing technology, combined with Springboot concurrency control, the incremental data can be quickly sent and written to the target database or data storage system.

[0024] Furthermore, in step 4, the data migration rate is adjusted according to the load condition of the target system.

[0025] Furthermore, in step 5, the log framework Log4j is used to generate detailed log files for each stage and each link of the operation, recording the timestamp, incremental change type, and key information of the transmission result; at the same time, the Java scheduled task scheduling library is used to regularly check the system incremental change monitoring status and send an alarm when a problem is found.

[0026] Furthermore, in step 5, a unified management platform is constructed to centrally monitor the migration status of all data sources.

[0027] Specifically, the present invention provides multi-source and multi-target support, allowing data to be collected and integrated from multiple data sources and migrated to one or more targets. This solution is particularly suitable for modern enterprise environments, where data is often scattered across multiple databases and platforms. Multi-source data integration: When a business system communicates with multiple source databases, the listening program can accurately distinguish the type and specific number of the source database from which the communication message comes based on the interface information and template type configured when the listening process is started; according to the type of source data and the database type of the target system, the incremental data of multiple types of source data types are converted into a unified SQL format based on the data conversion module and stored in the memory; based on the data filtering configuration module, data that does not meet the rule requirements is eliminated; based on the database object mapping configuration module, the database objects (such as table names, column names) in SQL are mapped to the object names in the target business system; finally, the data writing module is called to unify the data to the same one or more target systems. The data integration of the platform is implemented as follows: Unified management and monitoring: Build a unified management platform to centrally monitor the migration status of all data sources and improve the scalability and maintainability of the system.

[0028] Specifically, the present invention provides dynamic flow control, which dynamically adjusts the rate and amount of data flowing to the target according to the current load of the system and the network status. It can effectively reduce the risk of system overload, improve the overall efficiency of data migration, and ensure smooth migration under different conditions. (1) Flow control mechanism: Design a dynamic flow control mechanism to adjust the data migration rate according to the load of the target system to avoid the target database affecting system performance due to too fast data inflow. (2) Intelligent scheduling: Combined with system monitoring and feedback, automatically adjust the processing and transmission strategy of incremental data to achieve optimal performance.

[0029] The present invention adopts the above technical solution, which has the following significant advantages and positive effects: 1. Reduce the intrusion to the source database: Compared with the prior art method that relies on the source database table to obtain and process incremental data, the present invention uses network packet capture to capture data changes, without the need to implement additional operations or open permissions on the source database, thereby reducing the intrusion to the original system. 2. Reduce the occupation of database resources: When processing large amounts of data, the prior art will significantly occupy the resources of the source database and affect the original business flow. The present invention captures incremental data in real time and only operates at the network level, which significantly reduces the pressure on the source database. 3. Enhance the flexibility and adaptability of the system: The present invention is not limited by the structure of the source database, can adapt to various types of databases and data changes, and has a high degree of flexibility. In various scenario tests, the technical solution successfully supported data of different formats and types without any modification to the source database.

[0030] Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of the present application is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians of the art without making creative work are within the scope of protection of the present application.

Claims

1. A method for implementing incremental migration using a network packet capture and analysis mechanism, characterized in that: It includes the following steps: Step 1: Network packet capture: monitor the network communication between the database and the application, and capture incremental data in real time; Step 2: Data change identification: Identify data changes by parsing captured network packets; Step 3: Extract and process incremental data: After the captured data is parsed, a data change record is created, and the incremental data that needs to be migrated is further classified and extracted; Step 4: Data transmission and writing: The incremental data is sent and written to the target database or data storage system through the transmission and writing processing tool; Step 5: Log and monitor: Track each step of the data migration process and perform system log recording and monitoring to identify and resolve problems in a timely manner.

2. According to claim 1, a method for implementing incremental migration using a network packet capture and analysis mechanism is characterized in that: In step 1, based on Java+Pcap4J, the interface addresses of the application and database are configured to monitor the network communication between the database and the application and capture incremental data.

3. According to claim 1, a method for implementing incremental migration using a network packet capture and analysis mechanism is characterized in that: In step 2, the Java.net library is combined to disassemble the original communication message according to the pre-configured database message template to disassemble the change action and original data information executed by the current message.

4. According to claim 1, a method for implementing incremental migration using a network packet capture and analysis mechanism is characterized in that: The specific steps of step 2 are as follows: Step 2-1, parse the captured network packets according to the database communication protocol to identify specific incremental data changes; Step 2-2, determine whether the current incremental data change is valid data; if so, execute step 3; otherwise, discard and record it in the system log and then execute step 1.

5. According to claim 1, a method for implementing incremental migration using a network packet capture and analysis mechanism is characterized in that: The captured data in step 3 is parsed and saved in JSON format to create a data change record. Duplicate data changes are filtered out by using the Java Set framework. The incremental data that needs to be migrated is further extracted based on the data change type classification, and the incremental data set is generated according to the rule of converting the UPDATE action into DELETE+INSERT.

6. A method for implementing incremental migration using a network packet capture and analysis mechanism according to claim 1, characterized in that: In step 4, based on the JDBC protocol and database batch processing technology, combined with Springboot concurrency control, the incremental data can be quickly sent and written to the target database or data storage system.

7. A method for implementing incremental migration using a network packet capture and analysis mechanism according to claim 1, characterized in that: In step 4, the data migration rate is adjusted according to the load of the target system.

8. The method for implementing incremental migration using a network packet capture and analysis mechanism according to claim 1, characterized in that: In step 5, the log framework Log4j is used to generate detailed log files for each stage and each link of the operation, recording the timestamp, incremental change type, and key information of the transmission result; at the same time, the Java scheduled task scheduling library is used to regularly check the system incremental change monitoring status and send an alarm when a problem is found.

9. The method for implementing incremental migration using a network packet capture and analysis mechanism according to claim 1, characterized in that: In step 5, a unified management platform is built to centrally monitor the migration status of all data sources.

Images