Distributed message processing method and system

Abstract

The invention discloses a distributed message processing method and system, and the method comprises the following steps: receiving a message sending request sent by a client, generating a message containing a sending timestamp according to the message sending request, adding the message to a corresponding message queue, the sending timestamp having uniqueness in the message queue, and sending the message to the client according to the message queue. Arranging messages in a message queue based on the sending timestamps; judging whether the message is overtime or not, marking an overtime message according to a judgment result, and backing up the overtime message to a distributed file system; and periodically deleting the overtime messages in the message queue according to preset clearing time. According to the invention, the sequence of message transmission is ensured through the design of the sending timestamps; through the design of judging and clearing the overtime message, the distributed file system and the message queue can be matched with each other, so that the data storage pressure brought by the message queue is relieved while the client queries the message.

Description

technical field

[0001] The invention relates to the technical field of computer software, in particular to a distributed message processing method and system. Background technique

[0002] The Hadoop platform is the most famous big data platform. Today's big data processing technology stacks are all based on Hadoop, and most applications based on the big data platform need to process distributed messages;

[0003] However, the existing message processing system is too specialized / complex or too simple, such as Microsoft's MSMQ is an industrial-grade message processing system with powerful functions and hard delivery guarantees, but its deployment requires Microsoft's proprietary system, which is costly; such as Amazon Provides a low-cost Simple Queue Service (Simple Queue Service, SQS), which only has basic message delivery operations (send / receive), and cannot guarantee the order of message delivery.

[0004] Therefore, when providing Hadoop users with a message queue func...

Examples

Embodiment 1

[0049] Embodiment 1. A distributed message processing method, such as figure 1 shown, including the following steps:

[0050] S100. Receive a message sending request sent by the client 200, generate a message including a sending time stamp according to the message sending request, and add the message to a corresponding message queue, where the sending time stamp is unique in the message queue , arranging the messages in the message queue based on the sending timestamp;

[0051] In this embodiment, the message line is written in the corresponding message queue based on the message sending request in advance, and then the corresponding timestamp management node 300 is requested to send the timestamp, and the obtained sending timestamp is inserted into the message line to complete the generation of the message. mark it as sent, so that the sending of the message is completed based on the message sending request, and this process is implemented based on the atomic method.

[005...

Embodiment 2

[0095] Embodiment 2. A distributed message processing system, such as figure 2 As shown, it is connected to the distributed file system 400, including several message management nodes 100, several time stamp management nodes 300 and several clients 200;

[0096] The message management node 100 includes:

[0097] The message publishing module is configured to receive a message sending request sent by the client 200, generate a message including a sending time stamp according to the message sending request, and add the message to a corresponding message queue, and the sending time stamp is determined by the corresponding time stamp Issued by the management node 300, the sending time stamp is unique in the message queue; the message publishing module 110 is also configured to arrange the messages in the message queue based on the sending time stamp;

[0098] Overtime judging module, for judging whether the message is overtime, marking the overtime message according to the judgm...

technical field

[0001] The invention relates to the technical field of computer software, in particular to a distributed message processing method and system. Background technique

[0002] The Hadoop platform is the most famous big data platform. Today's big data processing technology stacks are all based on Hadoop, and most applications based on the big data platform need to process distributed messages;

[0003] However, the existing message processing system is too specialized / complex or too simple, such as Microsoft's MSMQ is an industrial-grade message processing system with powerful functions and hard delivery guarantees, but its deployment requires Microsoft's proprietary system, which is costly; such as Amazon Provides a low-cost Simple Queue Service (Simple Queue Service, SQS), which only has basic message delivery operations (send / receive), and cannot guarantee the order of message delivery.

[0004] Therefore, when providing Hadoop users with a message queue func...

Examples

Embodiment 1

[0049] Embodiment 1. A distributed message processing method, such as figure 1 shown, including the following steps:

[0050] S100. Receive a message sending request sent by the client 200, generate a message including a sending time stamp according to the message sending request, and add the message to a corresponding message queue, where the sending time stamp is unique in the message queue , arranging the messages in the message queue based on the sending timestamp;

[0051] In this embodiment, the message line is written in the corresponding message queue based on the message sending request in advance, and then the corresponding timestamp management node 300 is requested to send the timestamp, and the obtained sending timestamp is inserted into the message line to complete the generation of the message. mark it as sent, so that the sending of the message is completed based on the message sending request, and this process is implemented based on the atomic method.

[005...

Embodiment 2

[0095] Embodiment 2. A distributed message processing system, such as figure 2 As shown, it is connected to the distributed file system 400, including several message management nodes 100, several time stamp management nodes 300 and several clients 200;

[0096] The message management node 100 includes:

[0097] The message publishing module is configured to receive a message sending request sent by the client 200, generate a message including a sending time stamp according to the message sending request, and add the message to a corresponding message queue, and the sending time stamp is determined by the corresponding time stamp Issued by the management node 300, the sending time stamp is unique in the message queue; the message publishing module 110 is also configured to arrange the messages in the message queue based on the sending time stamp;

[0098] Overtime judging module, for judging whether the message is overtime, marking the overtime message according to the judgm...

Images