Message consumption method and device and readable storage medium

Abstract

The embodiment of the invention provides a message consumption method and device and a readable storage medium. fThe message consumption method comprises the following steps: caching each message to afirst target message queue corresponding to a target partition, sending the maximum offset of the message in each message to a Kafka server to obtain at least one message after the maximum offsetd, so that the at least one message after the maximum offset can be obtained without waiting for the completion of the processing of a batch of obtained messages, and the rate of obtaining the messages from the Kafka server is increased to a certain extent. The messages can be acquired from the first target message queue through the plurality of first threads in the thread pool corresponding to the first target message queue, and the messages are consumed, so that the message consumption efficiency is improved.

Description

technical field

[0001] The present invention relates to the communication field, in particular to a message consumption method, device and readable storage medium. Background technique

[0002] Kafka is a distributed message queuing system developed by Linkedin. The main goal of Kafka development is to build a data processing framework for processing massive log user behavior and website operation statistics. A Kafka cluster contains one or more Kafka servers. Each message published to a Kafka server has a topic, which is called a Topic. A Kafka server can accommodate multiple Topics, and each Topic contains one or more partitions. The production client can send messages to the Kafka server, and the consumer client can consume the messages in the Kafka server.

[0003] At present, before the business accesses the Kafka server, it is necessary to estimate the throughput of the production client and the consumption client to determine the number of partitions of the configur...

Images