A message processing device and method

Abstract

The invention discloses a message processing device and method, including a reading module, a backup module, a reporting module and a processing module; the reading module is used to read the messages subscribed by consumers from the server; copy the read messages and store them in The backup module and notify the reporting module; the reporting module is used to receive the notification from the reading module, report the offset information, and notify the processing module; the processing module is used to receive the notification from the reporting module, and read the unprocessed information in the backup module message, create multiple sub-threads, and dispatch unprocessed messages in the backup module to each sub-thread; process the dispatched messages through each sub-thread, and store the processing results in the backup module. The present invention also includes a detection module, which is used to detect whether each message in the backup module has been processed each time the consumer restarts. The invention can solve the problems of repeated consumption of data by consumers in the Kafka system, repeated submission or overlay submission of Offset.

Description

technical field [0001] The invention relates to the technical field of computer data processing, in particular to a message processing device and method. Background technique [0002] Kafka is a high-throughput distributed publish-subscribe messaging system. like figure 1 As shown, the existing Kafka-based message processing system includes a message producer cluster composed of multiple message producers (Producer), a Kafka cluster composed of multiple servers (Broker), and multiple consumers (Consumer) Each message producer pushes (push) messages to one or more Broker servers, each Broker server stores messages according to a certain topic (Topic), and each consumer pulls messages to one or more Broker servers Fetch (pull) subscribed messages, and manage the dynamic joining and leaving of Broker server and consumers through the Zookeeper (administrator) node. Messages within the same topic are partitioned and stored on different Broker servers according to a certain key...

AI Technical Summary

Problems solved by technology

However, consumers may fail to process messages due to downtime, power outages, program exceptions, etc. At this time, the consumer application needs to be restarted to read messages from the Broker server again. This will not only cause the problem of repeated consumption of data , it will also cause the sub-thread to repeatedly submit or overwrite the submission of Offset information

Images