Message processing system and method based on distributed queue

Abstract

The present invention discloses a message processing system and method based on a distributed queue, the system comprising: a producer message proxy module, a distributed message queue cluster and a consumer message proxy module; the producer message proxy module is used to forward messages produced by producers to message queue nodes in the distributed message queue cluster, wherein the distributed message queue cluster comprises multiple message queue nodes; the consumer message proxy module is used to push messages in the message queue nodes to corresponding consumers according to the message subscription information of each consumer. The present invention can process multiple types of messages to meet different business needs by setting multiple message queue nodes in the distributed message queue cluster.

Description

Technical Field

[0001] The present invention relates to a message queue, and in particular to a message processing system and method based on a distributed queue. Background Art

[0002] Common message queues such as Kafka can be used for message processing. Kafka is usually selected for message processing based on distributed queues. Kafka is an open source distributed message processing platform with multiple features such as high throughput, persistence, and horizontal scalability. At present, with the increase in the types of messages and the increase in the amount of messages that need to be processed, Kafka has gradually become unable to meet the message processing needs. The existing technology requires a message processing system that can process multiple types of messages at the same time. Summary of the invention

[0003] In order to solve at least one technical problem in the above-mentioned background technology, the present invention proposes a message processing system based on a distributed queue.

[0004] In order to achieve the above object, according to one aspect of the present invention, a message processing system based on a distributed queue is provided, the system comprising: a producer message proxy module, a distributed message queue cluster and a consumer message proxy module;

[0005] The producer message proxy module is used to forward the message produced by the producer to the message queue node in the distributed message queue cluster, wherein the distributed message queue cluster includes multiple message queue nodes;

[0006] The consumer message proxy module is used to push the messages in the message queue node to the corresponding consumers according to the message subscription information of each consumer.

[0007] Optionally, the message queue node includes: a kafka node, a RocketMQ node, and a delayed message node for storing delayed messages.

[0008] Optionally, the message processing system based on the distributed queue further includes: a cluster management module; the cluster management module is connected to the distributed message queue cluster and is used to detect the operating status of each message queue node.

[0009] Optionally, the message processing system based on distributed queues also includes: a user control module; the user control module is connected to the distributed message queue cluster, and is used for users to create topics and query the message consumption status of each message queue node, wherein the messages in each message queue node are classified by topic.

[0010] Optionally, the producer includes: a client, a MySQL log, and a Redis log.

[0011] Optionally, the consumers include: HTTP server, Redis cache, distributed file system and Hbase database.

[0012] In order to achieve the above object, according to another aspect of the present invention, a message processing method based on a distributed queue is provided, the method comprising:

[0013] Forwarding the message produced by the producer to the message queue node in the distributed message queue cluster, wherein the distributed message queue cluster includes multiple message queue nodes;

[0014] The messages in the message queue node are pushed to the corresponding consumers according to the message subscription information of each consumer.

[0015] In order to achieve the above-mentioned purpose, according to another aspect of the present invention, a computer device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the above-mentioned distributed queue-based message processing method when executing the computer program.

[0016] In order to achieve the above-mentioned purpose, according to another aspect of the present invention, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed in a computer processor, the steps in the above-mentioned message processing method based on distributed queues are implemented.

[0017] The beneficial effect of the present invention is that the present invention can process various types of messages to meet different business needs by setting various message queue nodes in a distributed message queue cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0019] Figure 1 It is a structural diagram of a message processing system based on a distributed queue according to an embodiment of the present invention;

[0020] Figure 2 is a flow chart of a message processing method based on a distributed queue according to an embodiment of the present invention;

[0021] Figure 3 It is a schematic diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0023] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0024] It should be noted that the terms "including" and "having" and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or apparatus.

[0025] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0026] Figure 1 is a structural diagram of a message processing system based on a distributed queue according to an embodiment of the present invention. Figure 1 As shown, the message processing system based on distributed queues of this embodiment includes: a producer message agent module, a distributed message queue cluster and a consumer message agent module.

[0027] In an optional embodiment of the present invention, the distributed message queue cluster includes multiple message queue nodes, and each message queue node may have multiple nodes. In an optional embodiment of the present invention, the message queue node may be a kafka node, a RocketMQ node, and a delayed message node (Chronos) for storing delayed messages. In an optional embodiment of the present invention, the delayed message node (Chronos) uses RocksDB as a storage engine for delayed messages. In an optional embodiment of the present invention, the RocketMQ node includes a master node and a slave node, and the slave node is a backup of the master node.

[0028] In an optional embodiment of the present invention, the producer message proxy module has a built-in Thrift Server, which is used to forward the message produced by the producer to each message queue node according to the type of the message and a preset message forwarding rule.

[0029] In an optional embodiment of the present invention, the consumer message proxy module has a built-in Thrift Server, which is used to push messages in the message queue node to the corresponding consumer according to the message subscription information of each consumer. In addition, the consumer message proxy module can also support the SDK pull method to enable consumers to pull messages from the consumer message proxy module.

[0030] In an optional embodiment of the present invention, the messages produced and consumed may be SDK codes, which support mainstream languages ​​such as Java / C / C++ / Go / PHP / Python.

[0031] In an optional embodiment of the present invention, the producer may be a client, a MySQL log (MySQL binlog), a Redis log (Redis binlog), and the like.

[0032] In an optional embodiment of the present invention, the consumer may be an HTTP server, a Redis cache, a distributed file system, an Hbase database, etc. The HTTP server, the Redis cache, the distributed file system, and the Hbase database may subscribe to messages by subscription, and the consumer message proxy module pushes messages according to the message subscription information. In an optional embodiment of the present invention, the consumer may also be a client, and the client may pull messages from the consumer message proxy module by SDK pulling.

[0033] like Figure 1As shown, the message processing system based on the distributed queue of this embodiment includes a cluster management module (Zookeeper). The cluster management module is connected to the distributed message queue cluster and is used to detect the operating status of each message queue node. In an optional embodiment of the present invention, the kafka node in the distributed message queue cluster includes a master node and a slave node, and the cluster management module is also used for the configuration of the master node to achieve high availability of the node.

[0034] like Figure 1 As shown, the message processing system based on the distributed queue of this embodiment also includes a user control module (Console). The user control module (Console) can be a user console developed based on Spring. The user control module is connected to the distributed message queue cluster, and is used for users to create topics and query the message consumption of each message queue node, wherein the messages in each message queue node are classified by topic. In an optional embodiment of the present invention, the user control module facilitates users to apply for resources such as Topic, ConsumerGroup, Subscription, etc. on the console, and provides the function of viewing and resetting consumption progress.

[0035] In an optional embodiment of the present invention, the message processing system based on the distributed queue further includes a monitoring module (monitor) for providing consumption backlog monitoring and cluster health monitoring.

[0036] In an optional embodiment of the present invention, messages in a message queue node are classified by topic, messages produced by producers are sent to corresponding topics, and consumers subscribe to topics and consume messages. In an optional embodiment of the present invention, a topic can be divided into multiple partitions, and a partition belongs to only a single topic. A partition can be regarded as an additional log file at the storage level. When a message is appended to a partition log file, a specific offset is assigned to ensure the order of the message.

[0037] The message processing system based on distributed queues of the present invention provides low-latency, high-concurrency, high-availability, and high-reliability message services for various business lines. The system of the present invention provides a variety of message types including real-time messages, delayed messages, and transaction messages to meet different business needs. Users can easily access the distributed message queue cluster, produce and consume messages, and experience feature-rich and stable message services through a unified Web console and a fool-proof SDK. At the same time, the system also provides operation and maintenance personnel with a message visualization monitoring and alarm function to facilitate operation and maintenance maintenance. It can not only provide runtime data for applications as a reference, but also quickly locate problems, provide prevention and alarm functions, and greatly enhance the robustness of the overall service.

[0038] It can be seen from the above embodiments that the message processing system based on distributed queues of the present invention achieves at least the following beneficial effects:

[0039] 1. The system of the present invention supports P2P, Pub / Sub and other message models;

[0040] 2. The system of the present invention realizes mass message storage and supports message backtracking: Kafka is used as the underlying storage engine of the message;

[0041] 3. The system of the present invention achieves low latency and high throughput, with millisecond latency and a single machine throughput of millions of messages;

[0042] 4. The system of the present invention supports delayed messages, and can set the delay time of a single message accurately to the second level, supports callback interfaces in the form of Thrift and HTTP, and provides a variety of message types, including delayed messages and cyclic delayed messages.

[0043] Based on the same inventive concept, an embodiment of the present invention also provides a message processing method based on a distributed queue, which can be used to implement the message processing system based on a distributed queue described in the above embodiment, as described in the following embodiment. Since the principle of solving the problem by the message processing method based on a distributed queue is similar to that of the message processing system based on a distributed queue, the embodiment of the message processing method based on a distributed queue can refer to the embodiment of the message processing system based on a distributed queue, and the repeated parts will not be repeated.

[0044] Figure 2 is a flow chart of a message processing method based on a distributed queue according to an embodiment of the present invention. Figure 2 As shown, the message processing method based on the distributed queue according to the embodiment of the present invention includes step S101 and step S102.

[0045] Step S101: forwarding a message produced by a producer to a message queue node in the distributed message queue cluster, wherein the distributed message queue cluster includes multiple message queue nodes.

[0046] Step S102: Push the messages in the message queue node to the corresponding consumers according to the message subscription information of each consumer.

[0047] In an optional embodiment of the present invention, the message queue node includes: a kafka node, a RocketMQ node, and a delayed message node for storing delayed messages.

[0048] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0049] In order to achieve the above object, according to another aspect of the present application, a computer device is also provided. Figure 3 As shown, the computer device includes a memory, a processor, a communication interface and a communication bus. A computer program that can be executed on the processor is stored in the memory. When the processor executes the computer program, the steps in the above-mentioned embodiment method are implemented.

[0050] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0051] The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs and units, such as the corresponding program units in the above method embodiments of the present invention. The processor executes various functional applications of the processor and works data processing by running the non-transitory software programs, instructions and modules stored in the memory, that is, implementing the method in the above method embodiments.

[0052] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0053] The one or more units are stored in the memory, and when executed by the processor, the method in the above embodiment is performed.

[0054] The specific details of the above-mentioned computer device can be understood by referring to the corresponding related descriptions and effects in the above-mentioned embodiments, and will not be repeated here.

[0055] In order to achieve the above purpose, according to another aspect of the present application, a computer-readable storage medium is also provided, wherein the computer-readable storage medium stores a computer program, and the computer program implements the steps in the above-mentioned message processing system based on a distributed queue when executed in a computer processor. Those skilled in the art can understand that the implementation of all or part of the process in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and the program can include the process of the embodiment of each method as described above when executed. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (Hard Disk Drive, abbreviated as: HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0056] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0057] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A message processing system based on distributed queues, characterized in that: include: Producer message agent module, distributed message queue cluster, consumer message agent module, cluster management module and user control module; The producer message proxy module is used to forward the message produced by the producer to the corresponding message queue node in the distributed message queue cluster according to the type of the message and the preset message forwarding rule, wherein the distributed message queue cluster includes multiple message queue nodes, and the message queue nodes include: kafka nodes, RocketMQ nodes and delayed message nodes for storing delayed messages; the delayed message nodes use RocksDB as the storage engine for delayed messages; the RocketMQ nodes include master nodes and slave nodes, and the slave nodes are backups of the master nodes; the produced messages include SDK codes; The consumer message proxy module is used to push messages in the message queue node to the corresponding consumer according to the message subscription information of each consumer; the consumer message proxy module supports the SDK pull method so that the consumer pulls messages from the consumer message proxy module; the consumer includes: HTTP server, Redis cache, distributed file system, Hbase database and client; the client pulls messages from the consumer message proxy module through the SDK pull method; The cluster management module is connected to the distributed message queue cluster and is used to detect the running status of each message queue node. The Kafka node in the distributed message queue cluster includes a master node and a slave node. The cluster management module is also used to configure the master node to achieve high availability of the node; The user control module is connected to the distributed message queue cluster, and is used for users to create topics and query the message consumption of each message queue node, wherein the messages in each message queue node are classified by topic, a topic is divided into multiple partitions, a partition belongs to only a single topic, and a partition is regarded as an appended log file at the storage level. When a message is appended to a partition log file, a specific offset is assigned to ensure the sequence of the message. The user control module is a user console developed based on Spring, which facilitates users to apply for Topic, ConsumerGroup, and Subscription resources on the console, and provides the functions of viewing and resetting consumption progress.

2. The message processing system based on distributed queue according to claim 1, characterized in that: The producers include: client, MySQL log and Redis log.

3. A message processing method based on a distributed queue, characterized in that: include: The producer message proxy module forwards the message produced by the producer to the corresponding message queue node in the distributed message queue cluster according to the type of the message and the preset message forwarding rules, wherein the distributed message queue cluster includes multiple message queue nodes, and the message queue nodes include: kafka nodes, RocketMQ nodes and delayed message nodes for storing delayed messages; the delayed message nodes use RocksDB as the storage engine for delayed messages; the RocketMQ nodes include master nodes and slave nodes, and the slave nodes are backups of the master nodes; the produced messages include SDK codes; The consumer message proxy module pushes the messages in the message queue node to the corresponding consumers according to the message subscription information of each consumer; the consumer message proxy module supports the SDK pull method so that the consumer pulls messages from the consumer message proxy module; the consumer includes: HTTP server, Redis cache, distributed file system, Hbase database and client; the client pulls messages from the consumer message proxy module through the SDK pull method; The cluster management module detects the running status of each message queue node. The Kafka nodes in the distributed message queue cluster include master nodes and slave nodes. The cluster management module is also used for the configuration of the master node to achieve high availability of the node. Users create topics and query the message consumption of each message queue node through the user control module. The messages in each message queue node are classified by topic. A topic is divided into multiple partitions. A partition belongs to only a single topic. The partition is regarded as an appended log file at the storage level. When the message is appended to the partition log file, a specific offset will be assigned to ensure the sequence of the message. The user control module is a user console developed based on Spring. The user control module facilitates users to apply for Topic, ConsumerGroup, and Subscription resources on the console, and provides the functions of viewing and resetting consumption progress.

4. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to claim 3 is implemented.

5. A computer-readable storage medium storing a computer program, characterized in that: The computer program implements the method according to claim 3 when executed in a computer processor.

Images