Supercharge Your Innovation With Domain-Expert AI Agents!

Data synchronization method, device and system and storage medium

A data synchronization and data synchronization technology, which is applied in the field of information processing, can solve the problems that the uReplicator cluster cannot realize the synchronization of a large amount of data, and achieve the effects of high concurrency, synchronization, and improved utilization

Pending Publication Date: 2021-05-25
BEIJING DAJIA INTERNET INFORMATION TECH CO LTD
View PDF3 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0007] The present disclosure provides a data synchronization method, device, system and storage medium to at least solve the problem that the current uReplicator cluster cannot realize a large amount of data synchronization

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data synchronization method, device and system and storage medium
  • Data synchronization method, device and system and storage medium
  • Data synchronization method, device and system and storage medium

Examples

Experimental program
Comparison scheme
Effect test

example 1

[0187] Example 1. The source address is the address of the Kafka cluster, and the destination address is the address of the Kafka cluster. When it is necessary to synchronize topicA in the source Kafka cluster cluster1 to topicB in the destination Kafka cluster cluster2, topicA and topicB have three partitions, two The data synchronization process between Kafka clusters is as follows:

[0188] After the control server receives the data synchronization request, it sends a synchronization task request. The synchronization task request includes: the address of the source Kafka cluster cluster1, the address of the destination Kafka cluster cluster2; source topicA, the number of partitions is 3, respectively partition0, partition1, partition2 , purpose topicB.

[0189] After the control server receives the task request, it assigns the task, and assigns the three partitions of topicA to different workers for data synchronization.

[0190] In this step, you can assign partition0 to ...

example 2

[0192] Example 2, the source address is the Kafka cluster address, and the destination address is HDFS. When it is necessary to synchronize topicA in the source Kafka cluster cluster1 to the HDFS directory corresponding to the destination HDFS, topicA has 3 partitions, and data synchronization between the Kafka cluster and HDFS The process is as follows:

[0193] After the control server receives the data synchronization request, it sends a synchronization task request. The synchronization task request includes: the address of the source Kafka cluster cluster1, the destination HDFS address; the source topicA, the number of partitions is 3, which are partition0, partition1, partition2, and destination HDFS Table of contents.

[0194] After the control server receives the task request, it assigns the task, and assigns the three partitions of topicA to different workers for data synchronization.

[0195] In this step, you can assign partition0 to worker1, partition1 to worker2, ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a data synchronization method, device and system and a storage medium, which are applied to a synchronization server, the method comprises the steps of reading a plurality of synchronization tasks in a distributed application program coordination system, wherein each synchronization task comprises a task allocation object and a source address and a destination address of synchronization data; when the task allocation objects of the plurality of synchronization tasks are the synchronization server, establishing a plurality of groups of consumer threads and producer threads for each synchronization task; according to the source address of each piece of synchronous data, starting a corresponding consumer thread to connect a source synchronization object to read the synchronous data; and according to the destination address of each piece of synchronous data, starting a corresponding producer thread to connect a destination synchronous object to write the synchronous data, so as to at least solve the problem that data synchronization among a plurality of clusters cannot be realized by the current uReplicator cluster.

Description

technical field [0001] The present disclosure relates to the technical field of information processing, and in particular to a data synchronization method, device, system and storage medium. Background technique [0002] Kafka is a high-throughput distributed publish-subscribe messaging system that can handle all consumer actions streaming data in a website and is a key factor in many social functions on the modern web. [0003] In practical applications, when a message system Kafka cluster mirrors topic data to another Kafka cluster, it needs to use the MirroMaker tool in the Kafka cluster. Since the MirroMaker tool is not a clustered service, the above tools cannot solve a large amount of data mirroring task scene. [0004] In order to achieve a large number of data mirroring tasks, uReplicator clusters are usually used to implement, specifically, [0005] Before the Worker in the uReplicator cluster starts, write a source Kafka cluster address and a destination Kafka cl...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F16/178G06F16/182G06F9/50
CPCG06F16/178G06F16/182G06F9/5038
Inventor 洪亮赵健博陈林赵博于海洋
Owner BEIJING DAJIA INTERNET INFORMATION TECH CO LTD
Features
  • R&D
  • Intellectual Property
  • Life Sciences
  • Materials
  • Tech Scout
Why Patsnap Eureka
  • Unparalleled Data Quality
  • Higher Quality Content
  • 60% Fewer Hallucinations
Social media
Patsnap Eureka Blog
Learn More

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2025 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com