A message queue agent device
By designing the message queue proxy device, the interfaces and protocols of multiple message queues are unified, the development complexity and error problems caused by client differences in the prior art are solved, and the indiscriminate proxy and message processing of message queues are achieved. Reduces costs and risks.
Patent Information
- Application Number
- CN202210335428.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-31
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2042-03-31
AI Technical Summary
The client differences in existing message queue middleware have large differences, which leads to developers need to adapt to different client encodings, which are prone to errors, which increases maintenance and learning costs, and the introduction of multiple message queues in large Internet companies increases complexity and cost.
Design a message queue proxy device, including a sending agent, a receiving agent and an internal protocol of the proxy. It realizes a unified interface and protocol for multiple message queues through a sending adapter and a receiving processor, and provides a submit message proxy. Factory and message acquisition and processing starter simplifies message processing flow and ensures that messages are not lost.
It realizes indiscriminate proxy for multiple message queues, reduces the learning and maintenance costs of developers, avoids errors and message loss problems caused by different personnel levels, simplifies the message processing process, and supports flexible switching of message queues.
Smart Images

Figure CN114979259B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the field of software, and in particular to a message queue agent device. Background Art
[0002] Message queue middleware is an important component in distributed systems, mainly solving problems such as application coupling, asynchronous messages, and traffic clipping. There are two steps to use various message queues. One is to submit the message queue to the message queue middleware, and the other is to obtain it from the message queue middleware, and then call the business processing method to process the message. Currently on the market, the more commonly used message queues include ActiveMQ, RabbitMQ, ZeroMQ, Kafka, MetaMQ, RocketMQ, Redis, etc. They all provide clients for message submission and message retrieval. However, different message queue middlewares have huge differences in the use of their clients, which will cause: 1. If developers directly use their client code, they have to adapt to different clients. This undoubtedly puts a huge burden on developers. 2. Developers have different levels. When using the client directly, it is inevitable that there will be many situations, even serious errors, resulting in message loss. 3. Directly using the client of the message queue is not conducive to the management of the message queue, if the call chain log will be disconnected. In addition, when the R&D group of an Internet company grows larger and larger, it is inevitable to introduce different message queues. There are many differences in functions and usage among these message queues, which increases the subsequent maintenance costs of the product and the learning costs of R&D personnel. Summary of the invention
[0003] The main purpose of the present invention is to overcome some of the shortcomings of the prior art and provide a message queue agent device, characterized in that it includes a sender agent, a receiver agent and an agent internal protocol, the sender agent includes a sender agent interface and a plurality of sender adapters, the receiver agent includes a receiver agent interface and a plurality of message queue receiver processors, and the agent internal protocol is provided with a plurality of different message queues; for each message queue in the agent internal protocol, the sender agent and the receiver agent have at least one sender adapter and at least one receiver processor, and the three are associated, interacted or communicated with each other;
[0004] Preferably, the sender adapter can send the message obtained from the sender agent interface to the corresponding topic of the message queue corresponding to the internal protocol of the agent; the receiver processor can take the message subject to be processed by the receiver agent interface from the message queue, and then call the receiver agent interface to process specific business;
[0005] Preferably, after the sending-end agent obtains the message from the sending-end agent interface, it encapsulates the message and then sends the packaged message; the receiving-end agent first parses the encapsulated message and then calls the receiving-end agent interface to process specific services;
[0006] Preferably, the internal protocol of the proxy includes any two or more of ActiveMQ, RabbitMQ, ZeroMQ, Kafka, MetaMQ, RocketMQ or Redis;
[0007] Preferably, the device further comprises a sending-end agent and a receiving-end agent, including a message submission end agent factory and a message acquisition and processing initiator, wherein the message submission end agent factory can automatically generate a message submission agent of a business interface, and the message acquisition and processing initiator can start a specific client for obtaining a message to obtain the message and call a specific business implementation to process the business;
[0008] Preferably, when the message submission end proxy factory can automatically generate a message submission proxy of the business interface, the steps include obtaining the context variables of the current call, obtaining the variables passed by the current call method, encrypting the method variables, packaging all the variables, calling a specific submission end client, and extracting the packaged message into the message queue;
[0009] Preferably, when the initiator of message acquisition and processing works, after acquiring the message through the specific initiator, that is, the message acquisition client, the header information is read from the message and saved in the context of the current call, and then the method parameters are decrypted according to the encryption parameters in the header information; then the business interface is called to specifically implement the processing business; then it is determined whether the processing is completed normally, if not, the current message is re-acquired and processed again, otherwise the message acquisition client removes the current message from the message queue and acquires the next message;
[0010] Preferably, the sender proxy interface and the sender proxy interface are the same interface;
[0011] Preferably, when the message queue agent device is applied, it includes adding a tag to the sending end agent interface or the receiving end agent interface, and configuring the message submission end agent factory and the initiator of message acquisition and processing;
[0012] The beneficial effects of the present invention will be described in detail in the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained according to the structures shown in these drawings without creative work.
[0014] Figure 1 It is a schematic diagram of the overall structure of the message queue agent device;
[0015] Figure 2 This is a schematic diagram of the general agent structure;
[0016] Figure 3 The structure and processing diagram of the proxy factory for submitting messages;
[0017] Figure 4 Schematic diagram of the launcher structure and processing. DETAILED DESCRIPTION
[0018] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention;
[0019] In addition, when only the structure and composition scheme related to solving the technical problem are described in the present invention, although the well-known composition and processing, or the technical features that can be determined by combining the drawings and common knowledge, are not described below, it does not mean that they do not exist in the technical solution, and it should not be a reason for insufficient disclosure;
[0020] refer to Figure 1 : It is a schematic diagram of the overall structure of a message queue agent device in a preferred embodiment. The message queue agent device includes a sending-end agent, a receiving-end agent, and an agent internal protocol. Among them, the sending-end agent is composed of a sending-end agent interface and various sending-end adapters, wherein the various sending-end adapters correspond one-to-one to the agent internal protocol or message queue. The sending-end agent interface is provided to developers for use, and the sending-end adapter sends the message obtained from the sending-end agent interface to the corresponding topic of the corresponding message queue. The sending-end adapter is written in advance according to different message queues. Developers only need to write the specific message queue name in the configuration file;
[0021] The receiving agent consists of the receiving agent interface and the receiving processors of various message queues. The receiving processor is responsible for taking the message subject to be processed by the receiving agent interface from the message queue, and then calling the receiving agent interface to handle specific business. The receiving processor is written in advance according to different message queues. Developers only need to write the specific message queue name in the configuration file;
[0022] Preferably, the sender agent interface of the sender agent and the receiver agent interface of the receiver agent can be the same interface, but the implementation of the agent interface at the sender end is to call the send adapter of various message queues. At the receiver end, the agent interface is a specific business process;
[0023] Internal protocol of the agent, the communication protocol between the sending agent and the receiving agent. After the sending agent receives the message from the sending agent interface, it will encapsulate the message and then send the packaged message. The receiving agent must first parse the encapsulated message, and then call the receiving agent interface to handle specific business. The format of this protocol is: {message header: {requestid, message creation time, method name signature, message encryption method}, message body: "message ciphertext"}. Among them: 1. requestid is the call chain id, which is obtained by the sender from the log context (MDC), and the receiver saves this id in the log context;
[0024] In this way, through the message queue proxy device, developers do not need to directly use different client codes, do not need to adapt to different clients, and avoid the problem of message loss caused by different levels of developers directly using the client, which is inevitable in various situations, even serious errors. R&D personnel only need to modify the configuration of the proxy to switch / use different message queues, which reduces the code maintenance cost and the learning cost of R&D personnel, and realizes indifferent message queue proxy;
[0025] refer to Figure 2, is a general proxy structure used by the sending-end proxy and the receiving-end proxy in the preferred embodiment. It includes a business interface A, a message submission end proxy factory E and a message acquisition and processing initiator D. Specifically, for the sending-end proxy, it is the sending-end proxy interface, and for the receiving-end proxy, it is the receiving-end proxy interface. The message submission end proxy factory, after the project is started, will automatically generate a message submission agent of the business interface. The message acquisition and processing initiator, after the project is started, the initiator will start the specific client that gets the message to get the message and call the specific business implementation B to process the business. The developer only needs to design the business interface A and the specific business implementation B. Taking Worker as an example, the variable service originally injected is an object of type HelloServiceImpl. Now, as long as the configuration of the project is modified, the injected variable will become an object of type message queue submission agent. At this time, calling any method of service in Worker will become submitting messages to the queue. Then, add the configuration of the initiator for HelloServiceImpl. The initiator will automatically start the client that gets the message, take the message from the queue and call the corresponding method of HelloServiceImpl to process it. For the sending-end agent, the message-end agent factory E and the message acquisition and processing initiator D can be integrated / included in various sending-end adapters. For the receiving-end agent, the message-end agent factory E and the message acquisition and processing initiator D can be integrated / included in the receiving-end processor of various message queues;
[0026] refer to Figure 3 , which is the structure and processing diagram of the message submission proxy factory. The message submission proxy factory includes reading interface information, and the specific sub-factory generates the submission proxy. The specific sub-factory has written the implementation of each message queue in advance. If you need to use a specific message queue in the project, you can directly add the corresponding configuration. When the project is started, the factory will generate the corresponding submission proxy for the interface and inject it into the variable through spring (such as the service variable in Worked). Here, the variables passed by the method are not submitted directly as messages, but will be wrapped in a layer. So that the variables required for management can be added to the submitted message. The final message structure submitted is: {message header: {requestid, message creation time, method name signature, message encryption method}, message body: "message ciphertext"}. In the process of generating the submission proxy by the specific sub-factory, it includes obtaining the context variables of the current call, obtaining the variables passed by the current calling method, encrypting the method variables, packaging all variables, calling the specific submission client, and extracting the packaged message into the message queue;
[0027] refer to Figure 4, is a schematic diagram of the launcher structure and processing. The internal structure of the launcher for message acquisition and processing is shown in the figure. The specific launcher needs to be implemented in advance for each message queue. If the specific message queue in the project is, just add the corresponding configuration and start the project. The launcher will call the message acquisition client to obtain the message and process it. As can be seen from the figure, the developer's specific business implementation will be included in the launcher. If the launcher exits unexpectedly (such as machine crash) while the message is currently being processed. The currently processed message will not be lost, and the message will be processed again at the next startup. This avoids the problem of message loss due to insufficient developer experience. When the launcher for message acquisition and processing works, after the specific launcher, that is, the message acquisition client obtains the message, it reads the header information from the message and saves it to the context of the current call, and then decrypts the method parameters according to the encryption parameters in the header information; then calls the business interface to specifically implement the processing business; then determines whether the processing is completed normally. If not, re-acquire the current message and process it again, otherwise the message acquisition client removes the current message from the message queue and obtains the next message;
[0028] When the message queue proxy device is actually used, you only need to add the required tags to the interface and add the corresponding configuration. Take Worker as an example, as follows: Add tags to the interface:
[0029] @SenderFactory("kafkaSenderFactory")
[0030] publicinterface HelloService {
[0031] @MqTopic("sayHello")
[0032] publicvoid sayHello(String param1,String param2,String param3);
[0033] @MqTopic("sayworld")
[0034] publicvoid sayworld(String paramX,String paramY,String paramZ);
[0035] }
[0036] Submit the message broker factory configuration:
[0037] <bean class= "cn.qtone.zf.common.mq.MqProduceConfig" >
[0038] <property name= "producers" >
[0039] <list>
[0040] <value> HelloService< / value>
[0041] < / list>
[0042]
[0043]
[0044] Starter configuration for message acquisition and processing
[0045] <bean class= "cn.qtone.zf.common.mq.MqReceiverBootConfig" >
[0046] <property name= "receiverBoot" ref= "KafkaReceiverBoot" / >
[0047] <property name= "impls" >
[0048] <map>
[0049] <entry key= "org.xft.comon.HelloService" value-ref= "helloServiceImpl" / >
[0050] < / map>
[0051]
[0052]
[0053] In this way, through the application of message queue proxy device in software development, general developers do not need to write message queue call code when using message queue, so they can concentrate on business. This reduces the use and learning costs of developers. At the same time, it eliminates the differences in use caused by different levels of personnel, and can also save complete logs and better ensure that messages will not be lost due to downtime. The message queue service used can be switched at any time without any impact on the business code.
Claims
1. A message queue agent device, characterized in that: The invention comprises a sending-end agent, a receiving-end agent and an agent internal protocol, wherein the sending-end agent comprises a sending-end agent interface and a plurality of sending-end adapters, the receiving-end agent comprises a receiving-end agent interface and a plurality of receiving-end processors of message queues, and the agent internal protocol is provided with a plurality of different message queues; for each message queue in the agent internal protocol, at least one sending-end adapter and at least one receiving-end processor are associated, interacted or communicated with each other in the sending-end agent and the receiving-end agent, wherein various sending-end adapters correspond to the agent internal protocol or the message queue respectively, the sending-end agent interface is provided for developers to use, and the sending-end adapter sends the message obtained from the sending-end agent interface to the corresponding topic of the corresponding message queue; The sender adapter is pre-written according to different message queues. Developers only need to write the specific message queue name in the configuration file. The sending-end agent and the receiving-end agent include a message submission agent factory and a message acquisition and processing initiator. The message submission agent factory can automatically generate a message submission agent of a business interface. The message acquisition and processing initiator can start a specific client to obtain a message and call a specific business implementation to process the business. The submission message proxy factory can automatically generate the submission message proxy of the business interface, including obtaining the context variables of the current call, obtaining the variables passed by the current call method, encrypting the method variables, packaging all variables, calling the specific submission client, and extracting the packaged message into the message queue.
2. The message queue agent device according to claim 1, characterized in that: The sending-end adapter can send the message obtained from the sending-end agent interface to the corresponding topic of the message queue corresponding to the internal protocol of the agent; the receiving-end processor can take the message topic to be processed by the receiving-end agent interface from the message queue, and then call the receiving-end agent interface to handle specific business.
3. The message queue agent device according to claim 2, characterized in that: After the sending end agent obtains the message from the sending end agent interface, it encapsulates the message and then sends the packaged message; the receiving end agent first parses the encapsulated message and then calls the receiving end agent interface to process specific services.
4. The message queue agent device according to claim 1, characterized in that: The internal protocol of the proxy includes any two or more protocols among ActiveMQ, RabbitMQ, ZeroMQ, Kafka, MetaMQ, RocketMQ or Redis.
5. The message queue agent device according to claim 1, characterized in that: When the initiator of message acquisition and processing is working, after acquiring the message through the specific initiator, that is, the client of the message acquisition end, the header information is read from the message and saved in the context of the current call, and then the method parameters are decrypted according to the encryption parameters in the header information; then the business interface is called to specifically implement the processing business; then it is determined whether the processing is completed normally. If not, the current message is re-acquired and processed again, otherwise the client of the message acquisition end removes the current message from the message queue and acquires the next message.
6. The message queue agent device according to claim 1, characterized in that: The sending end proxy interface and the receiving end proxy interface are the same interface.
7. The message queue agent device according to claim 1, characterized in that: When the message queue agent device is used, it includes adding a tag to the sending end agent interface or the receiving end agent interface, and configuring the message submission end agent factory and the initiator for message acquisition and processing.
Citation Information
Patent Citations
Multi protocol adapter and method for multi business to implement adapting treatment
CN101207568A
User mode network protocol stack system and method for processing message
CN101867558A
A message passing method based on supporting multiple message queues
CN109710435A
Method and device for supporting multi-protocol access by RCS, and electronic equipment
CN111294317A
Messaging proxy system
CN1456005A
