Message queue processing method, medium, equipment and system

A message queue and processing method technology, applied in the message queue processing method, equipment and system, and media fields, can solve the problems of insufficient throughput capacity and not strictly guaranteeing the order of messages, so as to avoid data loss, achieve reliability, and ensure timing Effect

Active Publication Date: 2021-10-15
北京宇信科技集团股份有限公司
View PDF4 Cites 5 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The first type of message queue generally has an accumulation capacity of about 100,000 messages. Under the premise of sequential confirmation of messages, the throughput capacity is insufficient.
[0005] The message accumulation capacity of the second type of message queue can reach hundreds of millions, but RocketMQ does not strictly guarantee the order of messages

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Message queue processing method, medium, equipment and system
  • Message queue processing method, medium, equipment and system
  • Message queue processing method, medium, equipment and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0121] In the financial system's consistency requirements for account systems, it is required that when the previous transaction of the same account is not completed, the subsequent transaction cannot be processed, that is, strict timing is required. When performing asynchronous processing, it is necessary to use message queues for inter-system communication, and strictly guarantee the timing of messages from the same account. At the same time, when performing batch processing, the message queue is required to reach a message throughput capacity of 10,000 levels per second, and to have a message accumulation capacity of more than one million levels.

[0122] The light-weight deployed message queue in the embodiment of the present invention can be applied to the accounting micro-service system, and has the following capabilities: it can support the processing / accumulation capability of more than one million messages within one hour; it supports the specified business associated ...

Embodiment 2

[0155] Figure 2A It is a flow chart of a message queue processing method according to an embodiment of the present invention. Such as Figure 2A As shown, the method is applied to a message queue device, for example including but not limited to a Redis message queue, which includes the following steps:

[0156] S101. Obtain the write segment index value according to the partition name sent by the message producer;

[0157] In this embodiment, a segment is a collection of message contents stored under a storage key, and is a Stream data structure. It is stored in the same Redis in the storage queue as a Hash data structure. After specifying the Hash key name and the partition name as the field name, the write index of the partition can be accessed. The corresponding Redis operation instructions are:

[0158] hget streamreaderWriteIdx partition name.

[0159] In addition, there is a read index (streamreaderReadIdx) corresponding to the write index, which is also stored in ...

Embodiment 3

[0330] The embodiment of the present invention also provides a computer-readable storage medium. A computer program is stored in the computer-readable storage medium. When the computer program is executed by a processor, it realizes:

[0331] Obtain the value of the write segment index according to the partition name sent by the message producer;

[0332] Obtain the segment number from querying the Hash of the segment index according to the partition name, and use the partition name and the segment number to concatenate to obtain the storage key name;

[0333] Write the message content according to the storage key name, and determine the message time sequence number entryId corresponding to the message content;

[0334] Write a mark in the preset data structure of the message queue, the mark is used to indicate that the message content has not been synchronized between the master node and the slave node of the message queue; Whether the number reaches the preset length of the...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a message queue processing method, medium, equipment and system, and the method comprises the steps: obtaining a value of a write segment index according to a partition name sent by a message production end; querying from the Hash of the write segment index according to the partition name to obtain a segment number, and splicing the partition name and the segment number to obtain a storage key name; writing the message content according to the storage key name, and determining a message time serial number entryId corresponding to the message content; writing a mark into the message content, wherein the mark is used for indicating that the message content is not synchronized between the master node and the slave node of the message queue; enabling the message queue to comprise a master node and slave nodes; judging whether the number of the written message contents reaches the preset length of the segment or not; when the number of the written message contents reaches the preset length of the segment, adding 1 to the value of the write segment index; acquiring a main node replication offset of the message queue; and returning the entryId and the main node replication offset of the message queue to the message production end. According to the technical scheme, the time sequence of message processing is ensured.

Description

technical field [0001] The invention relates to the field of asynchronous communication between applications with strict timing requirements, in particular to a message queue processing method, medium, equipment and system. Background technique [0002] There are two main types of existing open source distributed message queues (Message Queue, MQ for short): the first type is memory-based message queues such as ActiveMQ, RabbitMQ, etc., and the second type is disk-based message middleware such as Kafka, RocketMQ, etc. . [0003] In the process of realizing the present invention, the inventor finds that there are at least the following problems in the prior art: [0004] The first type of message queue generally has an accumulation capacity of about 100,000 messages, and the throughput capacity is insufficient under the premise of sequential confirmation of messages. [0005] The message accumulation capacity of the second type of message queue can reach hundreds of million...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/54G06F9/52
CPCG06F9/546G06F9/52G06F2209/548
Inventor 王虓
Owner 北京宇信科技集团股份有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap