Unlock instant, AI-driven research and patent intelligence for your innovation.

Data transmission method and device for kafka cluster

A data transmission and clustering technology, applied in the field of data processing, can solve problems such as data loss, achieve the effect of improving reliability, reducing the risk of data loss, and ensuring accuracy

Active Publication Date: 2019-06-25
BEIJING DIGITAL UNION WEB SCI & TECH CO LTD
View PDF3 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

That is, in the Kafka client, the data sent to the Kafka server is generally cached in the local message queue and then sent to the KAFKA server cluster in batches. When all the KAFKA servers are down, the data cached in the local message queue is risk of loss

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data transmission method and device for kafka cluster
  • Data transmission method and device for kafka cluster
  • Data transmission method and device for kafka cluster

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0025] Embodiments of the present invention modify the Kafka client, and judge whether all Kafka cluster servers in the Kafka cluster have been disconnected at the Kafka client. In this case, the server status is fed back to the caller, and the caller You can stop sending data to the Kafka cluster to minimize the risk of data loss.

[0026] figure 1 A hardware architecture diagram applicable to the data transmission device 200 of the Kafka cluster according to an embodiment of the present invention. In this embodiment, the characteristics of Kafka are used to implement distributed, partitionable, multi-subscriber, and persistent services with redundant backup. The producer producer, broker (broker) and consumer consumer are all distributed architectures. The Kafka client 120 (producer) caches the generated data to a local message queue and sends it to the KAFKA cluster 100 (broker) by means of batch asynchronous sending. The KAFKA cluster 100 includes multiple servers 110 . ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a data transmission method and device for a Kafka cluster. The data transmission method for the Kafka cluster comprises the steps of calling a preset Kafka connection state interface, and obtaining a connection state identifier of the Kafka cluster through utilization of the Kafka connection state interface; identifying a data connection state of the Kafka cluster corresponding to the connection state identifier; and stopping transmitting data to the Kafka cluster under the condition that the data connection state indicates that all servers in the Kafka cluster cannot be connected. The connection state of the Kafka cluster can be judged through the Kafka connection state interface, the abnormal connection condition of the Kafka cluster can be discovered timely and the data transmission is stopped, so a data loss risk in a local message queue of a cache is reduced, and the reliability of the Kafka cluster is improved.

Description

technical field [0001] The invention relates to data processing, in particular to a data transmission method and device of a Kafka cluster. Background technique [0002] Message middleware is a software widely used in distributed systems for communication between nodes. In the large-scale and high-concurrency background service architecture system, KAFKA, as a commonly used message middleware, is widely used. [0003] Kafka is a messaging middleware developed and open source by LinkedIn. Kafka message middleware is mainly composed of producer, broker and consumer. The producer publishes the message, the broker forwards the message from the producer to the consumer, and the consumer receives and processes the message. The producer producer and the broker broker act as the client and server of the message respectively. In practice, multiple Kafka brokers are generally run in a cluster to form a Kafka cluster. [0004] Since every time a message is generated, it needs to es...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): H04L29/08
CPCH04L67/14H04L67/143
Inventor 杨玉奇杨从安张宇平
Owner BEIJING DIGITAL UNION WEB SCI & TECH CO LTD