Concurrent consumption method and device for message-oriented middleware

Abstract

The invention discloses a message concurrent processing method and device for message-oriented middleware. The specific implementation scheme is that the method comprises the following steps: enablinga consumption thread to extract a message from a message queue, and storing the extracted message in a cache queue; and concurrently processing the messages in the cache queue by adopting the workingthreads in a thread pool, and returning to execute the step of extracting the messages from the message queue by the consumption thread and storing the extracted messages into the cache queue until the messages in the message queue are processed. Since the consumption thread does not execute the service logic and only extracts the message into the cache queue, the performance of a single consumption thread can be significantly improved. The messages are concurrently processed by utilizing a large number of working threads in the thread pool while being stored in the cache queue, so that the time is saved, and the speed and efficiency of concurrently processing the messages are improved. The messages in the cache queue are concurrently processed by utilizing a large number of working threads in the thread pool, so that the concurrent processing capability of a large number of messages is obviously improved.

Description

technical field [0001] This application relates to the field of big data, in particular to the field of concurrent consumption of message middleware. Background technique [0002] Messaging middleware extends inter-process communication in a distributed environment by providing message passing and message queuing models. In the actual application environment, such as SMS notification service, data statistics service, etc., all rely on message middleware to consume this message to complete their own business logic. For message middleware, common roles include producer, consumer, and message queue. Among them, in the thread world, the producer is the thread that produces the message, also called the production thread. A consumer is a thread that consumes messages, also called a consumer thread. Message (Message) refers to data transmitted between applications. Messages can be as simple as just text strings, or more complex and may contain embedded objects. Message queuing...

AI Technical Summary

Problems solved by technology

For example, in the same topic (topic) of the message queue kafka (a high-throughput distributed publish-subscribe message system) cluster, the performance of a single consumer is that each partition can only be consumed by one consumer, and the kafka cluster consumption The maximum number of consumers for an operator depends on the number of partitions. The number of partitions and the performance of a single consumer greatly constrain the consumer performance of the kafka cluster, resulting in poor consumption capabilities of consumers.

Especially when the system encounters high concurrent requests, because the consumption capacity of consumers does not keep up with the production capacity of producers, messages accumulate or even messages are lost

Images