A message processing method and system

Abstract

An embodiment of the present invention provides a message processing method and system. The method includes that a message server stores a request message sent by a first message end to a first storage area of a first subscription topic according to a first subscription topic in the request message. A second message end obtains the request message from the message server through a first subscription mechanism, processes the request message to obtain a response message, and sends the response message to the message server through a second subscription mechanism. The message server stores the response message in a second storage area of a second subscription topic, and the first message end obtains the response message from the message server through the second subscription mechanism. Since a second storage area for receiving and storing the response message is created in the message server, the loss of the response message can be avoided, and thus the replay of the response message can be realized. In addition, the method can avoid changing the code involved in the message server during the message processing process, so that subsequent services can be used in a timely and effective manner.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of financial technology (Fintech), and in particular, to a message processing method and system. Background Art

[0002] With the development of computer technology, more and more technologies are applied in the financial field. The traditional financial industry is gradually transforming into financial technology. However, due to the security and real-time requirements of the financial industry, higher requirements are also put forward for technologies.

[0003] The existing message processing method is mainly that a message producer (Producer) produces a message and marks the REPLY_TO identifier and the BROKER identifier on the message header of the message. Then the marked message is transmitted to the message server (Broker), and the message server stores the marked message in the topic. The message consumer can pull the marked message corresponding to the subscription requirement from the topic of the message server according to the subscription requirement, and perform business processing based on the marked message. After the business processing is completed, the message indicating that the business processing has been completed, the REPLY_TO identifier, and the BROKER identifier are assembled into a response message. Then, based on the BROKER identifier, the response message is sent to the corresponding message server, and the message server queries the SocketChannel corresponding to the marked message from the memory mapping table based on the REPLY_TO identifier, and directly returns the response message to the corresponding message producer through the SocketChannel. However, this processing method requires many modifications to the message server, which is not conducive to the subsequent use of the service; at the same time, the message server does not store the response message, resulting in an increased probability of loss of the response message and inability to replay the response message.

[0004] In summary, there is an urgent need for a message processing method to avoid the loss of response messages, so as to achieve the replay of response messages. Summary of the Invention

[0005] Embodiments of the present invention provide a message processing method and system to avoid the loss of response messages, so as to achieve the replay of response messages.

[0006] In a first aspect, embodiments of the present invention provide a message processing method, including:

[0007] A first message end sends a request message to a message server through a first subscription mechanism; in the first subscription mechanism, the first message end is a production end, and the second message end is a consumption end;

[0008] The message server stores the request message in a first storage area of the first subscription topic according to the first subscription topic in the request message;

[0009] The second message end obtains the request message from the message server through the first subscription mechanism and processes it to obtain a response message; the response message includes the second subscription topic in the request message.

[0010] The second message end sends the response message to the message server through a second subscription mechanism; in the second subscription mechanism, the first message end is the consumer end and the second message end is the producer end.

[0011] The message server stores the response message in the second storage area of the second subscription topic.

[0012] The first message end obtains the response message from the message server through the second subscription mechanism.

[0013] In the above technical solution, the first message end sends a request message to the message server through the first subscription mechanism. After receiving the request message, the message server stores the request message in the first storage area of the first subscription topic according to the first subscription topic in the request message. Then, the second message end obtains the request message from the message server through the first subscription mechanism, processes it to obtain a response message, and sends the response message to the message server through the second subscription mechanism. After receiving the response message, the message server stores the response message in the second storage area of the second subscription topic so that the first message end can obtain the response message from the message server in a timely and accurate manner through the second subscription mechanism. Since a second storage area for receiving and storing the response message is created in the message server, the loss of the response message can be avoided, and thus the replay of the response message can be realized. In addition, in the process of message processing, since the response message carries the second subscription topic, it can be ensured that the response message is stored in the corresponding second storage area in a timely and accurate manner for the first message end to receive normally, without the need to expand the sending message processor and add a return package message processor by changing the code in the message server, thus avoiding changes to the code involved in the message processing process of the message server, which is convenient for subsequent services to use in a timely and effective manner and can improve the efficiency of message processing.

[0014] Optionally, before sending the request message to the message server, it further includes:

[0015] The first message end creates a first subscription topic for the request message and a second subscription topic for the response message of the request message in the message server.

[0016] The first message end determines the queue information under the second subscription topic through the service registry.

[0017] The first message end determines the response queue information of the response message from the queue information under the second subscription topic; the response queue information serves as the second storage area of the second subscription topic.

[0018] In the above technical solution, the first message end creates a first subscription topic for the request message and a second subscription topic for the response message of the request message in the message server, and determines the response queue information of the response message from the queue information under the second subscription topic. In this way, a second storage area for receiving and storing the response message can be implemented in the message server, so as to ensure that the first message end can receive the response message corresponding to the request message in a timely and accurate manner, and the loss of the response message can be avoided to achieve the replay of the response message.

[0019] Optionally, the first message end determines the response queue information of the response message from the queue information under the second subscription topic, including:

[0020] The first message end determines the response queue information of each application instance in an evenly distributed manner according to the number of its own application instances and the number of queues in the queue information under the second subscription topic; the response queue information is used to store the response message corresponding to the request message generated by the application instance.

[0021] In the above technical solution, by evenly distributing the queue information under the second subscription topic according to the number of its own application instances and the number of queues in the queue information under the second subscription topic, it can be ensured that each application instance shares the queue information under the second subscription topic on average, and the situation that a certain application instance fails or the data processing is slow due to excessive load pressure can be avoided.

[0022] Optionally, the second message end obtains the request message from the message server through the first subscription mechanism, including:

[0023] The second message end generates a message pull request corresponding to the first subscription topic;

[0024] The second message end sends the message pull request to the message server through the first subscription mechanism; the message pull request is used to instruct the message server to determine the request message from the first storage area based on the first subscription topic.

[0025] In the above technical solution, the second message end sends the message pull request to the message server through the first subscription mechanism, so that the message server can query the corresponding request message from the first storage area in a timely and accurate manner based on the first subscription topic in the message pull request.

[0026] Optionally, the second message end processes the request message to obtain a response message, including:

[0027] Based on the request message, the second message end processes the service corresponding to the request message, and after determining that the service processing is completed, generates the response message based on the message indicating that the service has been processed and the second subscription topic corresponding to the request message.

[0028] In the above technical solution, by generating a response message based on the message indicating that the service has been processed and the second subscription topic corresponding to the request message, it is convenient for the message server to effectively and accurately store the response message in the second storage area of the second subscription topic, and helps the first message end to accurately obtain the response message from the message server through the second subscription mechanism.

[0029] Optionally, after storing the request message in the first storage area of the first subscription topic, it further includes:

[0030] The message server determines the positions of the messages to be parsed in the first storage area through a playback thread; the messages to be parsed are the request messages to be parsed in the first storage area;

[0031] Based on the positions of the messages to be parsed, the message server parses and processes the messages to be parsed to obtain the message offsets of the messages to be parsed, and stores the message offsets of the messages to be parsed in the offset cache queue of the first subscription topic; the message offset is used for the message server to query the corresponding request message from the first storage area according to the message offset corresponding to the message pull request sent by the second message end.

[0032] In the above technical solution, the message server determines the positions of the messages to be parsed in the first storage area through a playback thread, and based on the positions of the messages to be parsed, parses and processes the messages to be parsed to obtain the message offsets of the messages to be parsed, so that when the second message end pulls a request message from the message server, the message server can quickly and accurately locate the corresponding request message based on the message offset corresponding to the message pull request.

[0033] In a second aspect, an embodiment of the present invention further provides a message processing system, including a first message end, a second message end, and a message server;

[0034] The first message end includes a first sending unit and a first processing unit;

[0035] The first sending unit is configured to send a request message to the message server through a first subscription mechanism; in the first subscription mechanism, the first message end is a production end, and the second message end is a consumption end;

[0036] The first processing unit is configured to obtain the response message from the message server through the second subscription mechanism;

[0037] The second message end includes a second sending unit and a second processing unit;

[0038] The second sending unit is configured to send the response message to the message server through a second subscription mechanism; in the second subscription mechanism, the first message end is the consumer end and the second message end is the producer end;

[0039] The second processing unit is configured to obtain the request message from the message server through the first subscription mechanism and process it to obtain a response message; the response message includes the second subscription topic in the request message;

[0040] The message server includes a receiving unit and a third processing unit;

[0041] The receiving unit is configured to receive the request message and receive the response message;

[0042] The third processing unit is configured to store the request message into the first storage area of the first subscription topic according to the first subscription topic in the request message; and store the response message into the second storage area of the second subscription topic.

[0043] Optionally, the first processing unit is further configured to:

[0044] Create a first subscription topic for the request message and a second subscription topic for the response message of the request message in the message server;

[0045] Determine the queue information under the second subscription topic through a service registration center;

[0046] Determine the response queue information of the response message from the queue information under the second subscription topic; the response queue information serves as the second storage area of the second subscription topic.

[0047] Optionally, the first processing unit is specifically configured to:

[0048] Determine the response queue information of each application instance in an evenly distributed manner according to the number of application instances of itself and the number of queues in the queue information under the second subscription topic; the response queue information is used to store the response messages corresponding to the request messages generated by the application instances.

[0049] Optionally, the second processing unit is specifically configured to:

[0050] Generate a message pull request corresponding to the first subscription topic;

[0051] Send the message pull request to the message server through the first subscription mechanism; the message pull request is used to instruct the message server to determine the request message from the first storage area based on the first subscription topic.

[0052] Optionally, the second processing unit is specifically configured to:

[0053] Based on the request message, process the service corresponding to the request message, and after determining that the service processing is completed, generate the response message based on the message indicating that the service has been processed and the second subscription topic corresponding to the request message.

[0054] Optionally, the third processing unit is further configured to:

[0055] Determine the positions of the messages to be parsed in the first storage area through a playback thread; the messages to be parsed are the request messages to be parsed in the first storage area;

[0056] Based on the positions of the messages to be parsed, perform parsing processing on the messages to be parsed to obtain the message offsets of the messages to be parsed, and store the message offsets of the messages to be parsed in the offset cache queue of the first subscription topic; the message offset is used for the message server to query the corresponding request message from the first storage area according to the message offset corresponding to the message pull request sent by the second message end.

[0057] In a third aspect, an embodiment of the present invention provides a computing device, including at least one processor and at least one memory, wherein the memory stores a computer program, and when the program is executed by the processor, the processor is caused to execute the message processing method according to any one of the above first aspects.

[0058] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, which stores a computer program executable by a computing device, and when the program runs on the computing device, the computing device is caused to execute the message processing method according to any one of the above first aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can also obtain other drawings based on these drawings without creative efforts.

[0060] Figure 1 Schematic diagram of a message processing system architecture provided by an embodiment of the present invention;

[0061] Figure 2 Flow schematic diagram of a message processing method provided by an embodiment of the present invention;

[0062] Figure 3 Structural schematic diagram of a message processing system provided by an embodiment of the present invention;

[0063] Figure 4 Structural schematic diagram of a computing device provided by an embodiment of the present invention. Detailed implementation manners

[0064] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0065] The following first explains some terms involved in the embodiments of the present invention to facilitate the understanding of those skilled in the art.

[0066] (1) Broker: Responsible for message storage, supporting lightweight queues in terms of topics. A single machine can support a queue scale of tens of thousands, supporting the message push-pull model, with a multi-copy fault tolerance mechanism (2 copies or 3 copies), strong peak shaving and valley filling capabilities, and the ability to stack hundreds of millions of messages. At the same time, it can strictly ensure the orderliness of messages.

[0067] (2) Producer: The message publishing client deployed by users, that is, the message requester.

[0068] (3) Consumer: The message subscription client deployed by users, that is, the message responder. Supports push and pull models, and supports broadcast mode and cluster mode.

[0069] (4) NameServer: A lightweight service registration center. Each NameServer node contains the routing information of topics in all Brokers. In the RocketMQ architecture, the broker maintains long connections with all NameServers, and the Producer and Consumer maintain long connections with any one NameServer.

[0070] (5) ConsumerGroup: The consumer group. Multiple consumer instances consuming the same type of messages form a consumer group, which can also refer to a consumer cluster.

[0071] (6) Topic: It is a logical classification of messages. For example, if there are both order-related messages and inventory-related messages, classification is required. One is the order Topic to store order-related messages, and the other is the inventory Topic to store inventory-related messages.

[0072] (7) Message: It is the carrier of messages. A Message must specify a topic, which is equivalent to the address for sending a letter. When a Producer sends a message and a Consumer receives a message, the topic must be preset.

[0073] (8) MessageQueue: Message queue, a subdivision of the topic. Multiple Queues logically form a topic, and horizontal expansion can be achieved by expanding the MessageQueue. It has three attributes: topic, brokerName, and queueId.

[0074] (9) Tag: Each sent message can have a tag, so that different business scenarios can be distinguished by tags for the same topic. In practice, a business system uses one topic and different tags to distinguish different messages.

[0075] (10) CommitLog: It is the storage body of the message body and metadata at the Broker end, storing the message body content written by the Producer end. The message content is not of fixed length.

[0076] (11) ConsumeQueue: It is the carrier for the Broker end to record the offsets of messages. It does not store the specific messages but only records the offsets of the messages belonging to its Topic in the CommitLog. In this way, when a consumer pulls messages from the Broker, it can quickly locate the messages according to the offsets. Among them, each entry in the ConsumerQueue is 20 bytes, including the offset in the CommitLog file (8 bytes), the byte size of the message (4 bytes), and the HashCode corresponding to the Tags label of the message (8 bytes).

[0077] (12)RemotingCommand: It is the message encapsulation for remote interaction in RocketMQ, which contains attributes such as code, customHeader, extFields, and body. Among them, code refers to the transaction code. For example, there is a corresponding transaction code for heartbeat, and there is a corresponding transaction code for topic management. Different transactions correspond to different transaction codes. When the server (referring to Broker) or the client (referring to Producer or Consumer) starts, it will initialize and start the remoting module simultaneously, and register the message processing class corresponding to the transaction code into the remoting module.

[0078] (13)Rocket-Remoting: The remoting module is the place where the server or the client sends and receives messages. RocketMQ uses netty for Socket interaction by default. Netty is a mature nio processing framework, and using netty can greatly simplify the development of java nio.

[0079] (14)TCP long connection: It means that after the client sends a message to the server, the connection will not be closed, and subsequent new messages will continue to be interacted through this connection. At the same time, the client sends heartbeat messages to the server regularly to ensure that the connection will not be closed due to timeout.

[0080] The above introduces some terms involved in the embodiments of the present invention. Next, the technical features involved in the embodiments of the present invention will be introduced.

[0081] Next, a brief introduction to the design concept of the embodiments of the present invention will be given:

[0082] In the prior art, the Producer side marks the REPLY_TO identifier and the BROKER identifier on the generated message header. Then, the marked message is transmitted to the message server Broker, and the message server stores the marked message in the topic. The message consumer can pull the marked message corresponding to the subscription requirement from the topic of the message server according to the subscription requirement, and perform business processing based on the marked message. After the business processing is completed, the message indicating that the business processing is completed, the REPLY_TO identifier, and the BROKER identifier are assembled into a response message. Then, based on the BROKER identifier, the response message is sent to the corresponding message server, and the message server queries the SocketChannel corresponding to the marked message from the memory mapping table based on the REPLY_TO identifier, and directly returns the response message to the corresponding message producer through the SocketChannel.

[0083] However, in the process of message processing in the prior art, the message server does not store the response messages, resulting in an increased probability of loss of response messages and inability to replay the response messages. Moreover, it is necessary to modify the code multiple times, namely: (1) Modify the Broker startup code to create a special topic of DefaultCluster-REPLY-TOPIC during startup. (2) Modify the Broker code to extend the message sending processor and register it in the remoting module. The remoting module parses the request message received from the Producer module into a RemotingCommand, and obtains the message sending processor corresponding to the transaction code code. This processor first obtains the REPLY_TO identifier of the message header from the extended attributes extFields of the RemotingCommand, and saves this unique identifier and the SocketChannel corresponding to the Producer in the memory mapping of the Broker for subsequent finding of the corresponding SocketChannel through REPLY_TO. (3) Extend the Producer-side MessageQueueSeletor code. When the Producer sends a message to the Broker, obtain the MessageType attribute of the message. When it is determined to be of the reply type (i.e., the response message), preferentially select the MessageQueue with the same name as the message header BROKER for message sending. The Consumer pulls the request message and completes the business processing, obtains the REPLY_TO information and BROKER information of the original request, assembles them into a reply message, and then preferentially sends it to the Broker with the same name through the extended message selector, which is equivalent to returning the packet to the Broker of the original request message. (4) Extend the Broker code to add a reply packet message processor and register it in the remoting module. The remoting module parses the response message received from the Consumer into a RemotingCommand, and obtains the reply packet message processor corresponding to the transaction code code. The reply packet message processor finds the REPLY_TO unique identifier from the extended attributes extFields of the RemotingCommand, then finds the SocketChannel connection information of the original request message from the memory mapping table producerChannelTable through this unique identifier, and directly returns the packet to the message requester through this SocketChannel.

[0084] In view of this, embodiments of the present invention propose a message processing method and system. In the embodiments of the present invention, a first message end sends a request message to a message server through a first subscription mechanism. After receiving the request message, the message server stores the request message in a first storage area of a first subscription topic according to the first subscription topic in the request message. Then, a second message end obtains the request message from the message server through the first subscription mechanism, processes it to obtain a response message, and sends the response message to the message server through a second subscription mechanism. After receiving the response message, the message server stores the response message in a second storage area of a second subscription topic, so that the first message end can obtain the response message from the message server in a timely and accurate manner through the second subscription mechanism. Since a second storage area for receiving and storing the response message is created in the message server, the loss of the response message can be avoided, and thus the replay of the response message can be realized. In addition, in the process of message processing, since the response message carries the second subscription topic, it can be ensured that the response message is stored in the corresponding second storage area in a timely and accurate manner, so that the first message end can receive it normally, without the need to expand the sending message processor and add a return packet message processor by changing the code in the message server, thus avoiding changing the code involved in the message processing process in the message server, which is convenient for subsequent services to be used in a timely and effective manner and can improve the efficiency of message processing.

[0085] To facilitate the understanding of the embodiments of the present invention, first, take the Figure 1 system architecture shown as an example to illustrate the message processing system architecture applicable to the embodiments of the present invention. This message processing system architecture can be applied to message publishing and subscribing in financial transaction scenarios, etc. In actual application scenarios, the present invention does not make any limitations in this regard. As Figure 1 shown, this system architecture may include a first message end 110, a message server 120, and a second message end 130. Among them, the first message end 110 is connected to the message server 120, and the second message end 130 is connected to the message server 120. For example, it can be connected by wire or wirelessly, and the specific connection method is not limited. It should be noted that the first message end 110 can be used as a message producer or a message consumer, and the second message end 130 can be used as a message consumer or a message producer. The embodiments of the present invention do not make any limitations in this regard.

[0086] Exemplarily, taking the first message end 110 as the message production end and the second message end 130 as the message consumption end as an example, the process of message processing is described. When the first message end 110 accesses the message server 120, it first creates a Topic (such as TopicA) for receiving request messages and a replyTopic, that is, SystemA-REPLY-TOPIC, for receiving response messages on the message server 120, and subscribes to the Topics (such as TopicA, TopciB, TopicC, etc.) for the business system and the replyTopic for receiving response messages at startup. Then it produces messages corresponding to the topic (such as messages corresponding to TopicA, messages corresponding to TopciB, messages corresponding to TopicC, etc.), marks the messages with the Topic and replyTopic, and then transmits the marked messages to the message server 120. The message server 120 stores the messages corresponding to the Topic (such as messages corresponding to TopicA, messages corresponding to TopciB, messages corresponding to TopicC, etc.) in the corresponding Topic. The second message end 130 accesses the message server 120 and subscribes to the Topics (such as TopicA, TopciB, TopicC, etc.) for the business system and the replyTopic for receiving response messages at startup. Then, when the second message end 130 wants to obtain the message corresponding to TopicA, it sends a message pull request to the message server 120, so that the message server 120 sends the message stored in TopicA to the second message end 130 based on the message pull request. The second message end 130 performs corresponding business processing based on the message corresponding to TopicA, and after the business processing is completed, generates a response message based on the message whose business has been processed and TopicA, and sends the response message to the SystemA-REPLY-TOPIC response message queue of the message server 120. Finally, when the first message end 110 wants to obtain the response message corresponding to TopicA, it sends a response message pull request to the message server 120, so that the message server 120 sends the response message corresponding to TopicA stored in the SystemA-REPLY-TOPIC response message queue to the first message end 110.

[0087] It should be noted that the structure shown above Figure 1 is only an example, and the embodiments of the present invention are not limited thereto.

[0088] Based on the above description, Figure 2 the flow of a message processing method provided by the embodiments of the present invention is exemplarily shown, and this flow can be executed by a message processing system.

[0089] AsFigure 2 As shown, the process specifically includes:

[0090] Step 201, the first message end sends a request message to the message server through the first subscription mechanism.

[0091] Step 202, the message server stores the request message in the first storage area of the first subscription topic according to the first subscription topic in the request message.

[0092] Step 203, the second message end obtains the request message from the message server through the first subscription mechanism.

[0093] Step 204, the second message end processes the request message to obtain a response message.

[0094] Step 205, the second message end sends the response message to the message server through the second subscription mechanism.

[0095] Step 206, the message server stores the response message in the second storage area of the second subscription topic according to the second subscription topic in the response message.

[0096] Step 207, the first message end obtains the response message from the message server through the second subscription mechanism.

[0097] In the above steps 201 and 202, before sending the request message to the message server, the first message end creates a first subscription topic for the request message and a second subscription topic for the response message of the request message in the message server. Then, the service registration center is used to determine the queue information under the second subscription topic, and the response queue information of the response message is determined from the queue information under the second subscription topic. This response queue information serves as the second storage area of the second subscription topic. In this way, a second storage area for receiving and storing the response message can be implemented in the message server, so as to ensure that the first message end can receive the response message corresponding to the request message in a timely and accurate manner and avoid the loss of the response message to achieve the replay of the response message. Among them, determining the response queue information of the response message from the queue information under the second subscription topic means that the first message end determines the response queue information of each application instance in an evenly distributed manner according to the number of its own application instances and the number of queues in the queue information under the second subscription topic. In this way, it can be ensured that each application instance evenly shares the queue information under the second subscription topic and avoid the situation that a certain application instance fails or has slow data processing due to excessive load pressure. Among them, the response queue information is used to store the response message corresponding to the request message generated by the application instance. Then, based on the first subscription mechanism (the first message end is the producer, the second message end is the consumer, and the second message end subscribes to the first subscription topic. That is, the second message end subscribes to this Topic as the consumer, and the first message end as the producer does not need to subscribe. It connects to all brokers with the Topic and uses a polling mechanism to send the request message to the broker end when sending the request message), the first message end generates a request message corresponding to the first subscription topic and sends the request message to the message server, so that the message server stores the request message in the first storage area of the first subscription topic according to the first subscription topic in the request message.

[0098] Meanwhile, after storing the request message into the first storage area of the first subscription topic, the message server determines the positions of the messages to be parsed in the first storage area through the playback thread, and based on the positions of the messages to be parsed, parses and processes the messages to be parsed to obtain the message offsets of the messages to be parsed, and stores the message offsets of the messages to be parsed into the offset cache queue of the first subscription topic. In this way, when the second message end pulls the request message from the message server, the message server can quickly and accurately locate the corresponding request message based on the message offset corresponding to the message pull request. Among them, each message to be parsed is each request message to be parsed in the first storage area; the message offset is used for the message server to query the corresponding request message from the first storage area according to the message offset corresponding to the message pull request sent by the second message end. Specifically, the message server determines the position of the last parsed request message in the first storage area through the playback thread, and starts parsing and processing multiple request messages located after that position to obtain the offsets of the multiple request messages, and stores the offsets of the multiple request messages in the ConsumeQueue corresponding to the first subscription topic, so that when the consumer end pulls the request message from the message server, the message server can quickly locate the corresponding request message based on the offset of the request message. It should be noted that the playback thread has the existing Offset offset (i.e., the message position at the end of the last parsing) of the CommitLog file. New messages entering the CommitLog will be appended to the end of the CommitLog. If the playback thread determines that the length of the CommitLog file is greater than the existing Offset offset, it starts parsing each message from the Offset and puts it into the ConsumerQueue of the corresponding topic of the message.

[0099] In the above steps 203 and 204, the second message end obtains the request message from the message server through the first subscription mechanism, and processes the request message to obtain the response message. Specifically, the second message end generates a message pull request corresponding to the first subscription topic, and sends the message pull request to the message server through the first subscription mechanism. The message pull request is used to instruct the message server to timely and accurately determine the request message from the first storage area based on the first subscription topic. Then, based on the request message, the service corresponding to the request message is processed. After determining that the service processing is completed, a response message is generated based on the message indicating that the service has been processed and the second subscription topic corresponding to the request message. In this way, it is convenient for the message server to effectively and accurately store the response message in the second storage area of the second subscription topic, and helps the first message end to accurately obtain the response message from the message server through the second subscription mechanism. Specifically, after pulling the request message from the message server, the second message end performs corresponding service processing based on the request message. After the service processing is completed, it obtains the attributes such as REPLY_TOPIC, REPLY_BROKER, and REPLY_QUEUEID in the message header of the request message. Then, a response message is assembled based on the attributes such as REPLY_TOPIC, REPLY_BROKER, and REPLY_QUEUEID and the information indicating that the service processing has been completed, and the response message is sent to the replyTopic response message queue listened by the first message end. In this way, since the response message carries attributes such as REPLY_TOPIC, REPLY_BROKER, and REPLY_QUEUEID, it can ensure that the response message is timely and accurately stored on the replyTopic response message queue listened by the first message end, so that the first message end can receive it normally, without the need to expand the message sending processor and add a return packet message processor by changing the code in the message server, thus avoiding changing the code involved in the message processing process of the message server.

[0100] In the above steps 205, 206, and 207, the second message end uses the second subscription mechanism (the first message end is the consumer end, the second message end is the producer end, and the first message end subscribes to the second subscription topic. That is, the first message end, as the consumer end, subscribes to this replyTopic, and the second message end, as the producer end, does not need to subscribe. It connects to all brokers with the replyTopic and uses a polling mechanism to send the response message to the broker end) to send the response message to the message server. The message server stores the response message in the second storage area of the second subscription topic according to the second subscription topic in the response message, so that the first message end can obtain the corresponding response message from the second storage area of the message server through the second subscription mechanism. Specifically, after receiving the response message, the message server stores the response message on the replyTopic response message queue listened by the first message end, so that the first message end can obtain the corresponding response message in a timely and accurate manner. That is, the first message end generates a response message pull request corresponding to the second subscription topic and sends the response message pull request to the message server, so that the message server queries the response message from the second storage area based on the second subscription topic in the response message pull request and then sends the response message to the first message end. After the first message end obtains the response message, it continues the processing flow of the subsequent received response messages. Among them, the response message corresponds to the request message.

[0101] In view of this, the implementation process of message processing in the embodiments of the present invention will be specifically described below.

[0102] Step1: Processing at the Producer end.

[0103] When the Producer end sends a message to the Broker end, it needs to attach the message queue information of the replyTopic listened by the current system to the message header. Among them, the specific process of attaching the replyTopic to the message header is as follows:

[0104] a. Create a replyTopic for receiving response messages.

[0105] When each system (usually referring to the combination of Producer and Consumer) accesses, a new replyTopic for receiving response messages needs to be created at the Broker end. For example, when SystemA (the combination of Producer and Consumer) accesses, a replyTopic for receiving response messages is created at the Broker end, which is SystemA-REPLY-TOPIC.

[0106] b. When the system starts, subscribe to the replyTopic used to receive the response message.

[0107] A system used in a production environment is usually a combination of a Producer and a Consumer. When the system is started, in addition to the normal subscription to each topic used by the system for the business system, it is also necessary to subscribe to the replyTopic for receiving response messages in order to receive the response messages returned by the downstream subsystem. It should be noted that a topic can be evenly shared by all consumers under the same consumer group ConsumerGroup (also referred to as a consumer cluster). For example, if TopicA has 3 queues, and a consumer group has 3 application instances (meaning that the Producer application is deployed to 3 servers, and each service started refers to an application instance), then each instance is responsible for the consumption of 1 queue. If another instance is started, that is, 4 consumer instances consume 3 topic queue messages, then the last instance started cannot consume the message of this topic, and the request cannot be received. Therefore, the queue messages corresponding to a topic are usually evenly shared by all consumers under the same consumer group ConsumerGroup.

[0108] c. When the Producer sends a message, it will include the replyTopic message queue information to which the instance is subscribed.

[0109] Each message has a topic. The Producer sends a message to the Broker, which saves the message to the CommitLog corresponding to the topic. After saving the message, the playback thread parses each message from the latest point (the point after the last parsed point) and puts it into the ConsumeQueue. Each topic has a corresponding ConsumeQueue. The ConsumeQueue is not responsible for storing messages, but only for recording the offset of the message of the Topic to which it belongs in the CommitLog, so that when the consumer pulls the message from the Broker, it can quickly locate the required message according to the offset. That is, the Consumer subscribed to this topic sends a request to the Broker to pull the message. The Broker finds the message from the ConsumeQueue to which the topic belongs and returns the corresponding message to the Consumer. The Consumer performs the corresponding business processing and returns the response message to the Broker.

[0110] To ensure that the response message can be received by the Producer instance that sent the request message, the Producer needs to attach the MessageQueue information corresponding to the replyTopic subscribed by the Producer to the message header when sending the request message. Specifically, for an application system suitable for a production environment, after the system goes live, the number of application instances is usually fixed. Therefore, the number of message queues MessageQueue of the replyTopic it subscribes to is also unchanged. Based on this principle, the MessageQueue information of the replyTopic subscribed by this instance can be attached when sending the request message, and the downstream receives the request return packet response message on the MessageQueue listened by the current system, so as to ensure that the response message can be normally received by the current system.

[0111] It should be noted that the Consumer client instance is generally DefaultMQPushConsumer, which is initialized using the average message distribution strategy algorithm at startup. Then, through this consumer instance, it obtains all the message queue MessageQueue sets under the replyTopic from the NameServer, that is, mqAll. Then, through this consumer instance, it obtains all the client lists corresponding to the consumer group ConsumerGroup to which the current instance belongs, that is, cidAll. Finally, a MessageQueue for the return message is randomly selected from the list of MessageQueue listened by the current instance through the load balancing algorithm and attached to the REPLY_TOPIC, REPLY_BROKER, and REPLY_QUEUEID fields of the request message header, so that the downstream subsystem that receives this message can parse the message header and send the response message back to the Broker.

[0112] Step2: Processing at the Consumer side.

[0113] After the Consumer pulls the corresponding message from the Broker according to the topic it subscribes to, it performs corresponding business processing. After the business processing is completed, it obtains the attributes such as REPLY_TOPIC, REPLY_BROKER, and REPLY_QUEUEID of the request message header. Then, based on the attributes such as REPLY_TOPIC, REPLY_BROKER, and REPLY_QUEUEID and the information that the business processing has been completed, it encapsulates them into a MessageQueue and specifies this MessageQueue to send the response message, so that the response message can be sent to the replyTopic queue listened by the Producer.

[0114] Step 3: Processing of the response message received by the Producer side.

[0115] The Producer side that sent the original request message subscribes to the replyTopic, pulls the response message corresponding to the original request message from the Broker side, and continues the processing flow of the subsequent received response messages after pulling the response message.

[0116] The above embodiments show that the first message side sends a request message to the message server through the first subscription mechanism. After receiving the request message, the message server stores the request message in the first storage area of the first subscription topic according to the first subscription topic in the request message. Then, the second message side obtains the request message from the message server through the first subscription mechanism, processes it to obtain a response message, and sends the response message to the message server through the second subscription mechanism. After receiving the response message, the message server stores the response message in the second storage area of the second subscription topic so that the first message side can obtain the response message from the message server in a timely and accurate manner through the second subscription mechanism. Since a second storage area for receiving and storing the response message is created in the message server, the loss of the response message can be avoided, and thus the replay of the response message can be realized. In addition, in the process of message processing, since the response message carries the second subscription topic, it can ensure that the response message is stored in the corresponding second storage area in a timely and accurate manner for the first message side to receive normally, without the need to expand the message sending processor and add a return packet message processor by changing the code in the message server, so that the code involved in the message processing process of the message server can be avoided from being changed, which is convenient for subsequent services to be used in a timely and effective manner and can improve the efficiency of message processing.

[0117] Based on the same technical concept, Figure 3 An exemplary message processing system provided by an embodiment of the present invention is shown, and this system can execute the process of the message processing method.

[0118] As Figure 3 shown, the system includes a first message side, a second message side, and a message server;

[0119] The first message side includes a first sending unit 301 and a first processing unit 302;

[0120] The first sending unit 301 is used to send a request message to the message server through the first subscription mechanism; in the first subscription mechanism, the first message side is the production side and the second message side is the consumption side;

[0121] The first processing unit 302 is configured to obtain the response message from the message server through the second subscription mechanism;

[0122] The second message end includes a second sending unit 303 and a second processing unit 304;

[0123] The second sending unit 303 is configured to send the response message to the message server through the second subscription mechanism; in the second subscription mechanism, the first message end is the consumer end and the second message end is the producer end;

[0124] The second processing unit 304 is configured to obtain the request message from the message server through the first subscription mechanism and process it to obtain a response message; the response message includes the second subscription topic in the request message;

[0125] The message server includes a receiving unit 305 and a third processing unit 306;

[0126] The receiving unit 305 is configured to receive the request message and receive the response message;

[0127] The third processing unit 306 is configured to store the request message in the first storage area of the first subscription topic according to the first subscription topic in the request message; store the response message in the second storage area of the second subscription topic.

[0128] Optionally, the first processing unit 302 is further configured to:

[0129] Create a first subscription topic for the request message and a second subscription topic for the response message of the request message in the message server;

[0130] Determine the queue information under the second subscription topic through the service registry;

[0131] Determine the response queue information of the response message from the queue information under the second subscription topic; the response queue information serves as the second storage area of the second subscription topic.

[0132] Optionally, the first processing unit 302 is specifically configured to:

[0133] According to the number of application instances of itself and the number of queues in the queue information under the second subscription topic, determine the response queue information of each application instance in an evenly distributed manner; the response queue information is used to store the response messages corresponding to the request messages generated by the application instances.

[0134] Optionally, the second processing unit 304 is specifically configured to:

[0135] Generate a message pull request corresponding to the first subscription topic;

[0136] Send the message pull request to the message server through the first subscription mechanism; the message pull request is used to instruct the message server to determine the request message from the first storage area based on the first subscription topic.

[0137] Optionally, the second processing unit 304 is specifically configured to:

[0138] Based on the request message, process the service corresponding to the request message, and after determining that the service processing is completed, generate the response message based on the message indicating that the service has been processed and the second subscription topic corresponding to the request message.

[0139] Optionally, the third processing unit 306 is further configured to:

[0140] Determine the positions of the messages to be parsed in the first storage area through a playback thread; the messages to be parsed are the request messages to be parsed in the first storage area;

[0141] Based on the positions of the messages to be parsed, perform parsing processing on the messages to be parsed to obtain the message offsets of the messages to be parsed, and store the message offsets of the messages to be parsed in the offset cache queue of the first subscription topic; the message offset is used for the message server to query the corresponding request message from the first storage area according to the message offset corresponding to the message pull request sent by the second message end.

[0142] Based on the same technical concept, an embodiment of the present invention further provides a computing device, as Figure 4 shown, including at least one processor 401 and a memory 402 connected to at least one processor. In the embodiment of the present invention, the specific connection medium between the processor 401 and the memory 402 is not limited, Figure 4 taking the connection between the processor 401 and the memory 402 through a bus as an example. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0143] In the embodiment of the present invention, the memory 402 stores instructions executable by at least one processor 401, and at least one processor 401 can execute the steps included in the foregoing message processing method by executing the instructions stored in the memory 402.

[0144] Among them, the processor 401 is the control center of the computing device. It can connect various parts of the computing device through various interfaces and circuits. By running or executing the instructions stored in the memory 402 and invoking the data stored in the memory 402, data processing can be achieved. Optionally, the processor 401 may include one or more processing units. The processor 401 may integrate an application processor and a modem processor. Among them, the application processor mainly processes the operating system, user interface, application programs, etc., and the modem processor mainly processes the issued instructions. It can be understood that the above-mentioned modem processor may not be integrated into the processor 401. In some embodiments, the processor 401 and the memory 402 may be implemented on the same chip. In some embodiments, they may also be separately implemented on independent chips.

[0145] The processor 401 may be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application specific integrated circuit (ASIC), a field programmable gate array, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in combination with the message processing embodiments may be directly embodied as being executed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

[0146] The memory 402 is a non-volatile computer-readable storage medium and can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. The memory 402 may include at least one type of storage medium. For example, it may include flash memory, hard disks, multimedia cards, card-type memories, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memories, magnetic disks, optical disks, etc. The memory 402 is any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 402 in the embodiments of the present invention may also be a circuit or any other device capable of implementing a storage function, for storing program instructions and / or data.

[0147] Based on the same technical concept, embodiments of the present invention also provide a computer-readable storage medium storing a computer program executable by a computing device. When the program runs on the computing device, the computing device is caused to execute the steps of the above message processing method.

[0148] Those skilled in the art should understand that the embodiments of the present invention may be provided as a method, a system, or a computer program product. Therefore, the present invention may take the form of an all-hardware embodiment, an all-software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program code.

[0149] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be realized by computer program instructions. These computer program instructions can be provided to the processors of general-purpose computers, special-purpose computers, embedded processors, or other programmable data processing devices to generate a machine, so that the instructions executed by the processors of the computer or other programmable data processing devices generate for realizing in the process Figure 1 one process or multiple processes and / or blocksFigure 1 means for the functions specified in one or more boxes.

[0150] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable memory produce a manufacture including an instruction means that implements the functions specified in one Figure 1 one or more processes and / or boxes Figure 1 or more boxes.

[0151] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, so that the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions specified in one Figure 1 one or more processes and / or boxes Figure 1 or more boxes.

[0152] Although the preferred embodiments of the present invention have been described, additional changes and modifications can be made by those skilled in the art once they learn of the basic creative concept. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications that fall within the scope of the present invention.

[0153] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of this application and their equivalent technologies, the present invention is also intended to include these modifications and variations.

Claims

1. A message processing method, characterized in that, Including: The first message end sends a request message to the message server through a first subscription mechanism; in the first subscription mechanism, the first message end is the producer end, the second message end is the consumer end, and the second message end subscribes to the first subscription topic in the request message; the request message includes a first subscription topic TOPIC, a second subscription topic REPLY_TOPIC, and response queue information under the second subscription topic; The message server stores the request message in the first storage area of the first subscription topic according to the first subscription topic; The second message end obtains the request message from the message server through the first subscription mechanism and processes the service corresponding to the request message based on the request message. After determining that the service processing is completed, a response message is generated based on the message indicating that the service has been processed and the second subscription topic corresponding to the request message; the response message includes the second subscription topic in the request message; the second message end sends the response message to the message server through a second subscription mechanism; in the second subscription mechanism, the first message end is the consumer end, the second message end is the producer end, and the first message end subscribes to the second subscription topic; The message server stores the response message in the second storage area of the second subscription topic according to the second subscription topic in the response message; the second storage area is the queue indicated by the response queue information under the second subscription topic, and the first message end listens to the queue indicated by the response queue information under the second subscription topic through the second subscription mechanism and obtains the response message from the message server; Before sending the request message to the message server, it further includes: The first message end creates a first subscription topic for the request message and a second subscription topic for the response message of the request message in the message server; The first message end determines the queue information under the second subscription topic through a service registry; The first message end determines the response queue information of the response message from the queue information under the second subscription topic; the queue indicated by the response queue information is used as the second storage area of the second subscription topic.

2. The method according to claim 1, characterized in that The first message end determines the response queue information of the response message from the queue information under the second subscription topic, including: The first message end determines the response queue information of each application instance in an evenly distributed manner according to the number of its own application instances and the number of queues in the queue information under the second subscription topic; the response queue information is used to store the response messages corresponding to the request messages generated by the application instances.

3. The method according to claim 1, characterized in that The second message end obtains the request message from the message server through the first subscription mechanism, including: The second message end generates a message pull request corresponding to the first subscription topic; The second message end sends the message pull request to the message server through the first subscription mechanism; the message pull request is used to instruct the message server to determine the request message from the first storage area based on the first subscription topic.

4. The method according to claim 1, characterized in that, After storing the request message into the first storage area of the first subscription topic, it further includes: The message server determines the positions of the messages to be parsed in the first storage area through a playback thread; the messages to be parsed are the request messages to be parsed in the first storage area. The message server parses the messages to be parsed based on the positions of the messages to be parsed, obtains the message offsets of the messages to be parsed, and stores the message offsets of the messages to be parsed into the offset cache queue of the first subscription topic; the message offset is used for the message server to query the corresponding request message from the first storage area according to the message offset corresponding to the message pull request sent by the second message end.

5. A message processing system, characterized in that, It includes a first message end, a second message end and a message server. The first message end includes a first sending unit and a first processing unit. The first sending unit is used to send a request message to the message server through a first subscription mechanism; in the first subscription mechanism, the first message end is the producer end, the second message end is the consumer end, and the second message end subscribes to the first subscription topic in the request message; the request message includes a first subscription topic TOPIC, a second subscription topic REPLY_TOPIC, and response queue information under the second subscription topic. The first processing unit is used to listen to the queue indicated by the response queue information under the second subscription topic through a second subscription mechanism and obtain a response message from the message server; the second message end includes a second sending unit and a second processing unit. The second sending unit is used to send the response message to the message server through a second subscription mechanism; in the second subscription mechanism, the first message end is the consumer end, the second message end is the producer end, and the first message end subscribes to the second subscription topic. The second processing unit is used to obtain the request message from the message server through the first subscription mechanism and process the service corresponding to the request message based on the request message, and after determining that the service processing is completed, generate the response message based on the message indicating that the service has been processed and the second subscription topic corresponding to the request message; the response message includes the second subscription topic in the request message. The message server includes a receiving unit and a third processing unit. The receiving unit is used to receive the request message and receive the response message. The third processing unit is used to store the request message into the first storage area of the first subscription topic according to the first subscription topic; and store the response message into the second storage area of the second subscription topic according to the second subscription topic in the response message; the second storage area is the queue indicated by the response queue information under the second subscription topic. The first processing unit is further configured to: create a first subscription topic for the request message and a second subscription topic for the response message of the request message on the message server; determine queue information under the second subscription topic through the service registry; determine response queue information of the response message from the queue information under the second subscription topic; The queue indicated by the response queue information serves as the second storage area of the second subscription topic.

6. A computing device, characterized in that, It includes at least one processor and at least one memory. Among them, the memory stores a computer program. When the program is executed by the processor, the processor executes the method described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, It stores a computer program executable by a computing device. When the program runs on the computing device, the computing device executes the method described in any one of claims 1 to 4.

8. A computer program product, characterized in that, Instructions are stored in the computer program product. When a computer reads and executes the instructions, the computer executes the method described in any one of claims 1 to 4.

Images