Distributed message queue management system based on NVM

Abstract

The invention discloses a distributed message queue management system based on NVM, which adopts a theme-based publish-subscribe mode, and is provided with producer machines, consumer machines, a message server cluster and a cluster manager, each machine is configured with an NVM memory and an RDMA network card, and the NVM memories and the RDMA network cards are interconnected through an RDMA network; the device has the effects that a complex I/O software stack can be bypassed; message data is accessed through the virtual address of the process; the access to the message data can be completedby accessing the message metadata supporting random reading; meanwhile, messages are transmitted based on RDMA unilateral operation; the message data in the memory of the remote server is directly read and written; no redundant data copy is needed, the message transmission with high data throughput and low delay is realized, the remote message writing is ensured to be non-locked through a theme partitioning mechanism, and in the remote message writing process, a message batch processing strategy based on message production speed and message transmission speed self-adaption is adopted, so thatthe transmission delay is reduced, and the transmission bandwidth is improved.

Description

technical field [0001] The invention relates to computer data storage technology, more specifically, a distributed message queue management system based on NVM. Background technique [0002] Message queue, also known as message middleware, is a container for storing messages during message delivery and an important component in a distributed system. It mainly solves problems such as application decoupling and asynchronous consumption. Provide a message persistence mechanism to ensure that messages can be received correctly, that messages will not be lost in case of emergencies, and that messages can be consumed asynchronously, so that the server can cope with high-throughput message processing scenarios. [0003] The existing message queue stores messages in the memory or hard disk of the message server, and uses the traditional TCP / IP network to transmit data. Storing in memory can ensure real-time processing and fast access to messages, but because the memory capacity is ...

AI Technical Summary

Problems solved by technology

Storing in memory can ensure real-time processing and fast access to messages, but because the memory capacity is limited and the data cannot be recovered after power failure, the reliability of the message queue cannot be guaranteed; and the hard disk itself is a low-speed device, when accessing data It also needs to go through a complex I / O software stack, which cannot guarantee fast data reading and writing; the traditional Ethernet transmission bandwidth is low and the delay is large, which cannot effectively meet the needs of low latency and high throughput in the era of big data

The existing message queue cannot balance message persistence and high throughput

Images