Message processing system and processing method based on kafka

Abstract

The invention discloses a message processing system and processing method based on kafka and belongs to the technical field of message processing. The message processing system and processing method based on kafka are aimed to solve the problems of the prior art that repeated messages appear in an order system frequently, the processing for each message consumes much system resource, the system resource consumption would greatly increase the database visitor volume at the consumer end, the message processing ability is lowered, and the system stability and service effectiveness are lowered. The message processing system based on kafka comprises a message producer cluster composed of one or more message producers, a kafka cluster, and a consumer cluster composed of one or more consumers, the kafka cluster is composed of one or more servers, and the consumer cluster comprises a filter module, a local message queue caching module and a business processing module. The message processing method realized through the system is used for the system especially the order system which needs to process a large number of messages.

Description

technical field [0001] The invention specifically relates to a kafka-based message processing system and processing method, and belongs to the technical field of message processing. Background technique [0002] Kafka is a distributed message queue system (Message Queue) developed by Linkedin. [0003] The main goal of kafka development is to build a data processing framework for processing massive logs, user behavior and website operation statistics. In the case of combining data mining, behavior analysis, operation monitoring and other requirements, it needs to be able to meet the requirements of various real-time online and batch offline processing applications for low latency and batch throughput performance. Fundamentally speaking, high throughput is the first requirement, followed by real-time and persistence. [0004] The Kafka cluster consists of multiple Broker servers. Each type of message is defined as a topic. Messages within the same topic are partitioned an...

Images