Message processing method and system, message destination end device and distributed message system

Abstract

The invention discloses a message processing method and system, a message destination end device and a distributed message system. The method comprises the steps that Producer generates a message identifier for a message group to be sent, wherein the message identifier is used for uniquely identifying the message group; the message identifier and the message group are sent to Broker; Broker receives and stores the message identifier and the message group, wherein the message identifier and the message group are sent by Producer; when Producer determines that the message group needs to be sent again, the message identifier of the message group is acquired from the generated message identifier; the message identifier and the message group are sent to Broke; and when Broker determines that the message group corresponding to the message identifier is received again according to the received message identifier, deduplication is carried out on the received message group this time. According to the technical scheme of the invention, miscalculation rate can be removed on a Broker end, which improves the efficiency of message processing.

Description

technical field [0001] The invention relates to the technical field of information processing, in particular to a message processing method and system, a message destination device and a distributed message system. Background technique [0002] In the application of the distributed message system, the message issued by the message source device (Producer) is forwarded to the message destination device (Consumer) by the message broker device (Broker), during which the flow of messages reaches a large amount, because it is also distributed The complexity of the system is high, and the probability of message duplication or loss is magnified. Therefore, in the application of distributed message systems, it is particularly important to deduplicate messages and prevent message loss. [0003] In order to improve the efficiency of message transmission, the distributed message system needs to provide a clear message distribution guarantee level during the message delivery process. T...

AI Technical Summary

Problems solved by technology

[0005] In the process of realizing the present invention, the inventor found that the prior art has at least the following problems: in the prior art, the Bloom Filter scheme is usually used to deduplicate the received message at the Broker side, but the defect of the Bloom Filter is in the middle of the message In this scene, the files will be enlarged and appear extremely prominent

Because Bloom Filter has a serious problem - miscalculation rate, the miscalculation rate is proportional to the number of elements, and the amount of messages stored in the distributed message middleware is on the order of TB, the miscalculation rate will exceed the acceptable range, resulting in message low processing efficiency

Or, in the process of consuming messages on the consumer side, if the process crashes before the processing of step b) or c) is not successfully completed, the calculation result and the position of the message to be read will be in an inconsistent state, resulting in repeated consumption of messages when processing is resumed or message loss, resulting in inefficient message processing

Images