Data exchange method and system based on full fragmentation and incremental log analysis

Abstract

The invention relates to the technical field of computer networks, and discloses a data exchange method based on full fragmentation and incremental log analysis, which comprises the following steps: step 1, carrying out data fragmentation according to a database table primary key, and storing fragmented information into a message queue to obtain full fragmentation information; step 2, according toa preset job mapping relationship and the total fragmentation information in the message queue, obtaining data from a database in batches and writing the data into a target end; 3, performing real-time database incremental log analysis according to the write-in time points of the full records, and storing incremental data to a message queue in real time; and step 4, obtaining corresponding incremental data from the message queue by the position point recorded in last incremental data writing, and writing the incremental data into the target end. Batch and incremental acquisition of the data can be realized without modifying a source table structure, online business pause or service downtime is avoided, the source data can be sent to a destination end in real time while the format of the source data is ensured, and timely data service is provided.

Description

technical field [0001] The invention relates to the field of computer network information technology, in particular to a data exchange method and system based on full sharding and incremental log analysis. Background technique [0002] With the progress of society and the development of technology, there are more and more application systems, and their respective data exist in their own isolated networks. There are more and more data islands. The incremental and full data exchange methods break the data islands, and the full volume Reading source database data in slices avoids suspension of online business services, and incremental log parsing protects the source ecosystem from being broken. The traditional data exchange method is generally to directly check the database in full amount. When the amount of data is large, the online business may be down or suspend the service. However, most of the incremental technologies add timestamps at the source or compare the whole table...

AI Technical Summary

Problems solved by technology

The traditional data exchange method is generally to directly check the database in full amount. When the amount of data is large, the online business may be down or suspend the service. However, most of the incremental technologies add timestamps at the source or compare the whole table, and need to modify the source table. At the same time, more useless data is transmitted, resulting in higher network load and destroying the source ecosystem. There is a certain gap in performance and stability compared with the method of data exchange with full sharding and incremental parsing logs

Images