Kafka message uniqueness method based on Redis

A message and unique technology, applied in electrical components, transmission systems, etc., can solve problems such as limiting the use of Kafka, and achieve high throughput, reliability, and high reliability.

Active Publication Date: 2016-11-30
上海浦东发展银行股份有限公司信用卡中心
View PDF6 Cites 4 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

This greatly limits th

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Kafka message uniqueness method based on Redis

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0029] The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is carried out on the premise of the technical solution of the present invention, and detailed implementation and specific operation process are given, but the protection scope of the present invention is not limited to the following embodiments.

[0030] Such as figure 1 As shown, the present invention provides a method for uniqueness of Kafka messages based on Redis, and Redis is a key-value storage system. It supports a variety of value type storage, including string (string), list (linked list), set (set), zset (ordered set) and hash (hash type). Using Redis to solve the problem of repeated consumption of messages in Kafka, the method includes the following steps:

[0031] 1) Kafka receives a set of message data.

[0032] 2) Use the distributed election algorithm and use the SETNX command of Redis to filter the message da...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a Kafka message uniqueness method based on Redis. Redis is utilized to solve the problem of message repeated consumption in Kafka. The method comprises the steps that: Kafka receives message data; a distributed vote algorithm is adopted to filter the received message data; business logic processing is carried out on the message data after the filtering; and whether the business logic processing is wrong is judged, if yes, the message data wrong in business logic processing is deleted, and if not, the process is ended normally. Compared with the prior art, the Kafka message uniqueness method has the advantages that the message repeated consumption is avoided, the handling capacity is high, and the reliability is high.

Description

technical field [0001] The invention relates to a distributed message publishing-subscribing system, in particular to a method for uniqueness of Kafka messages based on Redis. Background technique [0002] Apache Kafka is a distributed message publish-subscribe system. Compared with the traditional message service RabbitMQ and Apache ActiveMQ, the throughput has increased by several orders of magnitude. At present, Kafka has been widely used in log processing and other systems. [0003] In the use of Kafka, the consumer of the message must maintain the location of the currently consumed message by itself, such as saving the location in Zookeeper. After the consumer finishes processing a batch of messages, it submits the message storage location first, and then reads and processes the next batch of messages. This creates a problem. When the consumer crashes, the location information has not been saved, and the next read location will be the same as the location after the l...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): H04L29/08
CPCH04L67/10
Inventor 马自谦范桢
Owner 上海浦东发展银行股份有限公司信用卡中心
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap