Unlock instant, AI-driven research and patent intelligence for your innovation.

Data transmission method and device for Kafka cluster

A data transmission and clustering technology, applied in the field of data processing, can solve problems such as data loss, achieve the effect of improving reliability, reducing the risk of data loss, and ensuring accuracy

Active Publication Date: 2017-07-07
BEIJING DIGITAL UNION WEB SCI & TECH CO LTD
View PDF3 Cites 8 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

That is, in the Kafka client, the data sent to the Kafka server is generally cached in the local message queue and then sent to the KAFKA server cluster in batches. When all the KAFKA servers are down, the data cached in the local message queue is risk of loss

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data transmission method and device for Kafka cluster
  • Data transmission method and device for Kafka cluster
  • Data transmission method and device for Kafka cluster

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0025] The embodiment of the present invention modifies the Kafka client. The Kafka client judges whether all Kafka cluster servers in the Kafka cluster can no longer be connected. In this case, the server status is reported to the caller. You can stop transmitting data to the Kafka cluster to minimize the risk of data loss.

[0026] figure 1 A hardware architecture diagram to which the data transmission device 200 of the Kafka cluster according to an embodiment of the present invention is applicable. In this embodiment, the characteristics of Kafka are used to realize a distributed, partitionable, multi-subscriber, redundant and durable service. Producer, broker and consumer are all distributed architectures. The Kafka client 120 (producer) caches the generated data to a local message queue and sends it to the KAFKA cluster 100 (broker) in a batch asynchronous sending manner. The KAFKA cluster 100 includes multiple servers 110. The Kafka cluster 100 can distribute the data tra...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a data transmission method and device for a Kafka cluster. The data transmission method for the Kafka cluster comprises the steps of calling a preset Kafka connection state interface, and obtaining a connection state identifier of the Kafka cluster through utilization of the Kafka connection state interface; identifying a data connection state of the Kafka cluster corresponding to the connection state identifier; and stopping transmitting data to the Kafka cluster under the condition that the data connection state indicates that all servers in the Kafka cluster cannot be connected. The connection state of the Kafka cluster can be judged through the Kafka connection state interface, the abnormal connection condition of the Kafka cluster can be discovered timely and the data transmission is stopped, so a data loss risk in a local message queue of a cache is reduced, and the reliability of the Kafka cluster is improved.

Description

Technical field [0001] The present invention relates to data processing, in particular to a data transmission method and device of a Kafka cluster. Background technique [0002] Message middleware is a software widely used in distributed systems for communication between nodes. In the large-scale high-concurrency background service architecture system, KAFKA, as a commonly used message middleware, is widely used. [0003] Kafka is a messaging middleware developed and open sourced by LinkedIn. Kafka message middleware is mainly composed of a producer, a broker, and a consumer. The producer publishes a message, the broker forwards the message from the producer to the consumer, and the consumer receives and processes the message. The producer and the broker serve as the client and server of the message respectively. In practical applications, multiple Kafka agents are generally run in a cluster to form a Kafka cluster. [0004] Since sending a message every time it needs to establis...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): H04L29/08
CPCH04L67/14H04L67/143
Inventor 杨玉奇杨从安张宇平
Owner BEIJING DIGITAL UNION WEB SCI & TECH CO LTD