Message processing method, device, device, storage medium and system

Abstract

Embodiments of the present invention provide a message processing method, device, device, storage medium, and system. The method includes: a computing node in a database system receives a message writing request sent by a message producer, and the message writing request includes a message queue identifier and the message body; according to the generated message sequence identifier under the message queue identifier, determine the target message sequence identifier corresponding to the message body, and the message sequence identifier generated under the message queue identifier is continuously increasing; combine the message queue identifier, target message sequence identifier and The message body is converted into the data format required by the first storage engine, so that the first storage engine stores the message queue identifier, the target message sequence identifier and the message body in the cloud storage system in the data format by calling the cloud storage service interface , the computing nodes are separated from the cloud storage system. Through this solution, efficient processing and long-term storage of massive messages can be realized.

Description

technical field [0001] The present invention relates to the technical field of databases, in particular to a message processing method, device, equipment, storage medium and system. Background technique [0002] Message Queue (Message Queue) is an important component in a distributed system. Its general usage scenario can be simply described as: when the result is not needed immediately, but the concurrency needs to be controlled, it is almost necessary to use the Message Queue when. The message queue mainly solves problems such as application coupling, asynchronous processing, and traffic cutting. [0003] The use of message queues such as traditional kafka is: each message published to the kafka cluster needs to have a category, this category is called a topic (topic), each topic can have multiple producers to send messages to it, There can also be multiple consumers to consume the messages. Messages of different topics are stored separately in the Kafka cluster. Physi...

AI Technical Summary

Problems solved by technology

[0004] In the above scheme, the splitting of topic partitions involves more complex operations, and usually the number of partitions under a topic is uncertain, and the expansion of partitions is relatively limited. Moreover, producers need to base their messages on the topic to which they publish The results of the partitioning under the partition, the routing of its published messages, and the complicated user operations make the traditional message system perform poorly in terms of performance and scalability when faced with the processing requirements of massive messages.

Images