Data synchronization method, device, equipment, system and computer-readable storage medium

Abstract

The invention discloses a data synchronization method. The method comprises: Spark obtaining source data at a source data end according to a data obtaining instruction; carrying out tagging processingon the source data to obtain a partition template; after the partitioning of the source data is completed based on the partitioning template, extracting target synchronous data from the partitioned source data according to a preset rule; writing the target synchronization data into a target data end. The data synchronization method can effectively improve the synchronization efficiency of data between two data synchronization parties and reduce the complexity of data synchronization while reducing the flow waste and the performance loss. The invention further discloses a data synchronizationdevice, equipment and system and a computer readable storage medium which all have the above beneficial effects.

Description

technical field [0001] The present application relates to the field of big data technology, in particular to a data synchronization method, and also to a data synchronization device, device, system and computer-readable storage medium. Background technique [0002] In the prior art, when it is necessary to realize data synchronization between two data terminals, there are usually two ways, namely, data import and export based on external tools, and data import and export based on cache. Among them, the implementation method using third-party tools needs to serialize the data into content readable by the third-party tool, and re-serialize it back to the data format corresponding to the data terminal to be synchronized. During this period, it needs to undergo a network transmission, and there is a possibility of data storage; The implementation scheme that uses caching will sacrifice additional disk resources, and the data will also undergo multiple flushing and serialization,...

AI Technical Summary

Problems solved by technology

Among them, the implementation method using third-party tools needs to serialize the data into content readable by the third-party tool, and re-serialize it back to the data format corresponding to the data terminal to be synchronized. During this period, it needs to undergo a network transmission, and there is a possibility of data storage; The implementation scheme using caching will sacrifice additional disk resources, and the data will also undergo multiple flushing and serialization, which will cause a large loss in performance

In addition, no matter which method is used, the import and export tasks are completed through the way of data first exiting the cluster and then entering the cluster. There is a waste of traffic, and the efficiency of data synchronization is extremely low

Images