Kafka message uniqueness method based on Redis

Abstract

The invention relates to a Kafka message uniqueness method based on Redis. Redis is utilized to solve the problem of message repeated consumption in Kafka. The method comprises the steps that: Kafka receives message data; a distributed vote algorithm is adopted to filter the received message data; business logic processing is carried out on the message data after the filtering; and whether the business logic processing is wrong is judged, if yes, the message data wrong in business logic processing is deleted, and if not, the process is ended normally. Compared with the prior art, the Kafka message uniqueness method has the advantages that the message repeated consumption is avoided, the handling capacity is high, and the reliability is high.

Description

technical field [0001] The invention relates to a distributed message publishing-subscribing system, in particular to a method for uniqueness of Kafka messages based on Redis. Background technique [0002] Apache Kafka is a distributed message publish-subscribe system. Compared with the traditional message service RabbitMQ and Apache ActiveMQ, the throughput has increased by several orders of magnitude. At present, Kafka has been widely used in log processing and other systems. [0003] In the use of Kafka, the consumer of the message must maintain the location of the currently consumed message by itself, such as saving the location in Zookeeper. After the consumer finishes processing a batch of messages, it submits the message storage location first, and then reads and processes the next batch of messages. This creates a problem. When the consumer crashes, the location information has not been saved, and the next read location will be the same as the location after the l...

AI Technical Summary

Problems solved by technology

This greatly limits the scope of use of Kafka

Images