A data conversion system and method across kafka clusters

Abstract

The present invention relates to a data conversion system and method across Kafka clusters, comprising the following steps: 1) building a data conversion system across Kafka clusters, including a data conversion module, a first cluster and a second cluster, a first cluster and a second cluster Both include producers and consumers, so that the producers of the first cluster produce data, and the producers of the second cluster receive data; 2) Start the Kafka services of the first cluster and the second cluster, and produce in the producers of the first cluster data; 3) the consumer of the first cluster extracts the message data block from the producer of the first cluster; 4) converts the format of the extracted message data to obtain data that meets the data format requirements of the second cluster; 5) will meet the requirements of the first cluster The data required by the format of the second cluster is sent to the producer of the second cluster for the message flow of the Kafka service of the second cluster. The invention can be widely applied to the field of Kafka data transmission.

Description

technical field [0001] The invention relates to a data conversion system and method across Kafka clusters, and belongs to the technical field of big data. Background technique [0002] Kafka, originally developed by Linkedin, is a high-throughput distributed publish-subscribe message system that can process all action stream data of consumers in the website. Kafka runs as a cluster and can consist of one or more services. The Kafka cluster is a distributed, partitionable, and replicable messaging system that can provide the functionality of a common messaging system. In layman's terms, Kafka is a log cluster. Various servers send their own logs to the cluster for unified aggregation and storage, and then other machines pull messages from the cluster for analysis and processing, such as ELT and data mining. [0003] However, there is no problem with data transfer within the Kafka cluster. However, due to the problems of the Kafka architecture itself, it does not support cr...

AI Technical Summary

Problems solved by technology

[0003] However, there is no problem with data transfer within the Kafka cluster

However, due to the problems of the Kafka architecture itself, it does not support cross-node data replication

Therefore, when the consumer of the A message system and the producer of the B message system connect data, the producer of the B message system will not receive the message due to the inconsistency of the data format.

Images