A message consumption method and message system

Abstract

The present invention discloses a message consumption method and a message system, wherein N one-to-one corresponding message storage queues and message consumption queues are respectively set on a message consumption server and a message storage server, and messages are stored in the message storage queue according to a preset queue storage order and a message distribution server's mark on the message, and messages are consumed from the message consumption queue according to a preset queue consumption order and a message distribution server's mark on the message, so that the last message consumption and the current message consumption satisfy that each of the message consumption queues consumes messages in units of a preset number according to the preset queue consumption order, thereby ensuring the order in which the messages are consumed. In addition, since N message pulling processes are set on the message consumption server to pull messages from the message storage server at the same time, compared with a single message pulling process, the message pulling efficiency and the message consumption efficiency and message throughput in high concurrency scenarios are improved.

Description

Technical Field

[0001] The present invention relates to the field of message consumption, and in particular to a message consumption method and a message system. Background Art

[0002] In the message payment field and the industrial field, after the message producer produces the message, the message distribution server will deliver the message produced by the message producer to the message storage server in a certain order. After receiving the message delivered by the message distribution server, the message storage server stores the message in a certain order in the message storage queue of the message storage server. When consuming the message, the message pull process in the message consumption server pulls the message from the message storage queue of the message storage server to the message consumption queue of the message consumption server and consumes it in a certain order. This is sequential consumption. When implementing sequential consumption, the prior art only sets up one message storage queue in the message consumption server and the message storage server, and only has a single message pull process in the message consumption server to pull messages from the message storage server. In a high-concurrency scenario, since the speed at which a single message pull process pulls messages from the message storage server is too slow, the message consumption efficiency in the high-concurrency scenario is very low, which reduces the message throughput. Summary of the invention

[0003] The purpose of the present invention is to provide a message consumption method and a message system, which ensure the order of message consumption, improve the message pulling efficiency and the message consumption efficiency and message throughput in high concurrency scenarios.

[0004] In order to solve the above technical problems, the present invention provides a message consumption method, which is applied to a message system, wherein the message system includes a message distribution server, a message storage server and a message consumption server, and the message consumption method includes:

[0005] When the message storage server receives a message marked and sent by the message distribution server according to the message consumption order, it distributes the message to N message storage queues in sequence according to the mark and the preset queue storage order, so that the last message distribution and the current message distribution meet the requirements that each message storage queue stores messages in units of a preset number according to the preset queue storage order, where N is an integer not less than 2;

[0006] The message consumption server calls N message pulling processes corresponding to N message storage queues one-to-one to pull the messages from the message storage queues corresponding to each of them, and stores the messages in N message consumption queues corresponding to the N message pulling processes one-to-one, and consumes the messages from the message consumption queues according to the tags and the preset queue consumption order, so that the last message consumption and the current message consumption satisfy that each message consumption queue consumes messages in units of the preset quantity according to the preset queue consumption order, and the preset queue consumption order of the N message consumption queues is the same as the preset queue storage order of the N message storage queues corresponding to each of them.

[0007] Preferably, the message is distributed to N message storage queues in sequence according to the mark and the preset queue storage order, so that the last message distribution and the current message distribution satisfy that each message storage queue stores messages in units of a preset number according to the preset queue storage order, including:

[0008] If the number of messages in the last group of the last message distribution is less than the preset number, the message storage queue of the last group of message distribution is determined to be the starting message storage queue, and a first group of messages is selected from the messages according to the mark, the number of the first group of messages being the difference between the preset number and the number of messages in the last group of the last message distribution; the remaining messages are grouped in units of the preset number according to the mark; if the number of messages in the last group of the last message distribution is equal to the preset number, the next message storage queue of the message storage queue of the last group of message distribution is determined to be the starting message storage queue, and the messages are grouped in units of the preset number according to the mark;

[0009] Taking the starting message storage queue as a benchmark, each group of the messages is distributed to each message storage queue in sequence according to the preset queue storage order; wherein, when the last group of messages last time and the first group of messages distributed this time exist in the starting message storage queue, the sum of the number of the last group of messages last time and the first group of messages distributed this time is the preset number.

[0010] Preferably, it also includes: when the message consumption queue corresponding to the message pulling process is full or there is no unpulled message in the message storage queue corresponding to the message pulling process, the message consumption server stops calling the message pulling process to pull the message from the message storage server.

[0011] Preferably, the process of setting the preset queue storage order includes:

[0012] Mark the N message storage queues with serial numbers from small to large;

[0013] The N message storage queues of the message storage server are sorted in descending order of the sequence numbers.

[0014] Preferably, the message distribution server marks and sends the message in the order in which the message is consumed, including:

[0015] When the message distribution server receives a message sent by a message producer, it marks the message in the order in which the message is consumed, and calls the delivery process in the preset queue distribution order to distribute the message to N message distribution queues in sequence. The maximum number of messages that each message distribution queue can store is the preset number, and the messages in each message distribution queue are sent to N message storage queues corresponding one-to-one to the N message distribution queues. The preset queue distribution order of the N message distribution queues is the same as the preset queue storage order of the N message storage queues corresponding one-to-one to each of them.

[0016] Preferably, there are N delivery processes.

[0017] The present invention also provides a message system, comprising:

[0018] Message distribution server;

[0019] A message storage server, configured to, upon receiving a message marked and sent by the message distribution server in accordance with the message consumption order, distribute the message to N message storage queues in sequence according to the mark and the preset queue storage order, so that the last message distribution and the current message distribution satisfy that each of the message storage queues stores messages in units of a preset number in accordance with the preset queue storage order, where N is an integer not less than 2;

[0020] A message consumption server is used to call N message pulling processes corresponding to N message storage queues one by one to pull the messages from the message storage queues corresponding to each of them, and store the messages in N message consumption queues corresponding to the N message pulling processes one by one, and consume the messages from the message consumption queues according to the tags and the preset queue consumption order, so that the last message consumption and the current message consumption satisfy that each of the message consumption queues consumes messages in units of the preset number according to the preset queue consumption order, and the preset queue consumption order of the N message consumption queues is the same as the preset queue storage order of the N message storage queues corresponding to each of them.

[0021] The present invention discloses a message consumption method and a message system, wherein N one-to-one corresponding message storage queues and message consumption queues are respectively set on a message consumption server and a message storage server, and messages are stored in the message storage queue according to a preset queue storage order and a message distribution server's mark on the message, and messages are consumed from the message consumption queue according to a preset queue consumption order and a message distribution server's mark on the message, so that the last message consumption and the current message consumption satisfy that each of the message consumption queues consumes messages in units of a preset number according to the preset queue consumption order, thereby ensuring the order in which the messages are consumed. In addition, since N message pulling processes are set on the message consumption server to pull messages from the message storage server at the same time, compared with a single message pulling process, the message pulling efficiency and the message consumption efficiency and message throughput in high concurrency scenarios are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the prior art and the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 A flow chart of a message consumption method provided by the present invention;

[0024] Figure 2 A structural diagram of a message system provided by the present invention. DETAILED DESCRIPTION

[0025] The core of the present invention is to provide a message consumption method and a message system, which ensures the order in which messages are consumed, improves the message pulling efficiency and the message consumption efficiency and message throughput in high concurrency scenarios.

[0026] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] Please refer to Figure 1 , Figure 1 The present invention provides a flow chart of a message consumption method.

[0028] A message consumption method is applied to a message system, the message system includes a message distribution server, a message storage server and a message consumption server, and the message consumption method includes:

[0029] S1: When the message storage server receives a message marked and sent by the message distribution server according to the message consumption order, it distributes the message to N message storage queues in sequence according to the mark and the preset queue storage order, so that the last message distribution and the current message distribution meet the requirements that each message storage queue stores messages in units of a preset number according to the preset queue storage order, where N is an integer not less than 2;

[0030] S2: The message consumption server calls N message pulling processes corresponding to N message storage queues one by one to pull messages from the message storage queues corresponding to each of them, and stores the messages in N message consumption queues corresponding to the N message pulling processes one by one, and consumes messages from the message consumption queues according to the tags and the preset queue consumption order, so that the last message consumption and the current message consumption satisfy that each message consumption queue consumes messages in units of a preset quantity according to the preset queue consumption order, and the preset queue consumption order of the N message consumption queues is the same as the preset queue storage order of the N message storage queues corresponding to each of them.

[0031] In the prior art, in order to ensure the sequential consumption of messages, there is only a single message storage queue in the message storage server, and a single message pulling process is set in the message consumption server to pull messages from the message storage queue in the message server and consume them in the order of consumption. When the consumption capacity of the message consumption server is very strong and the message producer produces a lot of messages, the efficiency of pulling messages by a single message pulling process is limited, so that the consumption capacity of the consumption server is wasted, reducing the message throughput and consumption efficiency of the system. In order to solve the above problem, in this embodiment, N message storage queues are set in the message storage server, and N message pulling processes and N message consumption queues corresponding to the N message storage queues are set in the message consumption server.

[0032] Specifically, when the message distribution server receives a message sent by a message producer, it will mark the message sent by the message producer with a serial number according to the order of consumption, and then send the message to the message storage server. The message storage server is provided with N message storage queues, and the N message storage queues are sorted according to a preset queue storage order. When the message storage server distributes the message to the N message storage queues, it follows the preset queue storage order and the mark of the message distribution server, that is, according to the consumption order of the message, after distributing a preset number of messages to a queue each time, it distributes a preset number of messages to the next queue according to the preset queue storage order, so that the last message distribution and this message distribution satisfy that each message storage queue stores messages in units of a preset number according to the preset queue storage order.

[0033] For example, when N is 3 and the preset number is 10, when the message storage server receives 13 messages sent by the message distribution server, the message storage server puts the first 10 of the 13 messages into the first message storage queue in the preset queue storage queue according to the preset queue storage order based on the mark of the message distribution server, and puts the remaining 3 messages into the second message storage queue in the preset queue storage queue according to the preset queue storage order. When the message storage server receives a message sent by the message distribution server next time, for example, when it receives 10 messages sent by the message distribution server next time, since 3 messages were stored in the second message storage queue in the last message distribution, in order to satisfy the last message distribution and this message distribution that each message storage queue stores messages in units of a preset number in the preset queue storage order, it is necessary to distribute the first 7 of the 10 messages to the second message storage queue according to the mark of the message distribution server, and then put the remaining 3 messages into the third message storage queue. Among the 23 messages stored in the three message storage queues of the message storage server, the 10 messages stored in the first message storage queue in the preset queue storage order are the 10 messages that should be consumed first according to the mark of the message distribution server, the 10 messages stored in the second message storage queue are the messages that should be consumed after the 10 messages in the first message storage queue, and the 3 messages in the third message storage queue are the three messages that should be consumed last. The three message pulling processes in the message consumption server respectively pull messages from the message storage queue corresponding to themselves to the message consumption queue corresponding to themselves to the message consumption server. Among the 23 messages stored in the three message consumption queues of the message consumption server, the 10 messages stored in the first message consumption queue in the preset queue consumption order are the 10 messages that should be consumed first according to the mark of the message distribution server, the 10 messages stored in the second message consumption queue are the messages that should be consumed after the 10 messages in the first message consumption queue, and the 3 messages in the third message consumption queue are the three messages that should be consumed last. The message consumption server consumes the 10 messages in the first message consumption queue in sequence according to the preset queue consumption order and the mark of the message distribution server, then consumes the 10 messages in the second message consumption queue, and finally consumes the 3 messages in the third message consumption queue, ensuring that the messages are consumed in sequence.

[0034] It should also be noted that in addition to pulling messages from the message storage server by calling the message pulling process, messages can also be pulled from N message storage queues corresponding to themselves by N message pulling threads. This application does not make any special limitations here.

[0035] In summary, in this embodiment, by setting N message storage queues on the message storage server, and setting N message pull processes and N message consumption queues corresponding to the N message storage queues on the message consumption server, and consuming the messages according to the message distribution server's tag, the order of the messages is guaranteed. In addition, since N message pull processes are set on the message consumption server to pull messages from the message storage server at the same time, compared with a single message pull process, the message pull efficiency and the message consumption efficiency and message throughput in high concurrency scenarios are improved.

[0036] Based on the above embodiments:

[0037] As a preferred embodiment, the message is distributed to N message storage queues in sequence according to the mark and the preset queue storage order, so that the last message distribution and the current message distribution satisfy that each message storage queue stores the message in units of a preset number according to the preset queue storage order, including:

[0038] If the number of messages in the last group of the last message distribution is less than the preset number, the message storage queue of the last group of message distribution is determined as the starting message storage queue, and the first group of messages is selected from the messages according to the mark, and the number of the first group of messages is the difference between the preset number and the number of messages in the last group of the last message distribution; the remaining messages are grouped in units of the preset number according to the mark; if the number of messages in the last group of the last message distribution is equal to the preset number, the next message storage queue of the message storage queue of the last group of message distribution is determined as the starting message storage queue, and the messages are grouped in units of the preset number according to the mark;

[0039] Based on the starting message storage queue, each group of messages is distributed to each message storage queue in sequence according to the preset queue storage order; among which, when the last group of messages of the previous time and the first group of messages distributed this time exist in the starting message storage queue, the sum of the number of the last group of messages of the previous time and the first group of messages distributed this time is the preset number.

[0040] In this embodiment, messages are grouped and placed into N message storage queues, so that the last message distribution and the current message distribution satisfy that each message storage queue stores messages in units of a preset number according to a preset queue storage order. For example, when N is 3 and the preset number is 10, if 13 messages are received from the message distribution server, then according to the mark of the message distribution server, that is, according to the consumption order of the messages, the first 10 messages are placed as the first group in the first message storage queue determined according to the preset queue storage order, and the remaining three messages are placed as the second group in the second message storage queue determined according to the preset queue storage order. At the next message distribution, assuming that the next message distribution needs to distribute 15 messages, since the number of messages in the last group of the previous message distribution is 3, which is less than the preset number (10), the message storage queue to which the last group of the previous message distribution is distributed, that is, the second message storage queue determined according to the preset queue storage order is determined as the starting message storage queue, and the number of the first group distributed this time is the preset number minus the number of messages in the last group of the previous message distribution, that is, 10-3=7. According to the consumption order of the messages, the first 15 of the 15 messages distributed this time are determined as the first group and distributed to the second message storage queue determined according to the preset queue storage order, and the remaining 8 messages are distributed to the third message storage queue determined according to the preset queue storage order. The ordered storage of messages is achieved to facilitate the sequential consumption of messages by subsequent consumer servers.

[0041] As a preferred embodiment, it also includes: when the message consumption queue corresponding to the message pulling process is full or there are no unpulled messages in the message storage queue corresponding to the message pulling process, the message consumption server stops calling the message pulling process to pull messages from the message storage server.

[0042] Considering that when the message consumption queue is full, if the message pull process corresponding to the message consumption queue is continued to be called to pull messages from the message storage server, the messages stored in the message consumption queue will be deleted, so that some messages are deleted before they are consumed, thereby causing the order of message consumption to be disordered. In addition, because there are no unpulled messages in the message storage queue corresponding to the message pull process, if the message pull process is still called to pull messages, many invalid pull requests will be generated, thereby affecting the performance of the entire message system. In order to solve the above problems, in this embodiment, when the message consumption queue corresponding to the message pull process is full or there are no unpulled messages in the message storage queue corresponding to the message pull process, the message consumption server stops calling the message pull process to pull messages from the message storage server, further ensuring the sequentiality of message consumption and improving the stability of the message system.

[0043] As a preferred embodiment, the process of setting the preset queue storage order includes:

[0044] Label the N message storage queues with serial numbers from small to large;

[0045] Sort the N message storage queues of the message storage server in descending order of sequence numbers.

[0046] In this embodiment, by marking the N message storage queues with ascending serial numbers and determining the preset queue storage order according to the sequence numbers in ascending order, the N message storage queues are sorted and the implementation method is simple.

[0047] As a preferred embodiment, the message distribution server marks and sends the message in the order of consumption of the message, including:

[0048] When a message distribution server receives a message sent by a message producer, it marks the message in the order in which it is consumed, and calls the delivery process to distribute the message to N message distribution queues in sequence according to the preset queue distribution order. The maximum number of messages that each message distribution queue can store is the preset number, and the messages in each message distribution queue are sent to N message storage queues that correspond one-to-one to the N message distribution queues. The preset queue distribution order of the N message distribution queues is the same as the preset queue storage order of the N message storage queues that correspond one-to-one to each of them.

[0049] In this embodiment, N message distribution queues corresponding to the N message storage queues are also set in the message distribution server, and each message distribution queue can store a preset number of messages at most. After the message distribution server receives the message sent by the message producer, it calls the delivery process to store the received messages in the N message distribution queues in sequence according to the message consumption order and the preset queue distribution order, and sends the messages in the N message distribution queues to the message storage queue corresponding to itself.

[0050] It should also be noted that in order to avoid the delivery process continuing to put messages into the message distribution queue when the message distribution queue is full but the message has not been sent, the delivery process can be stopped from calling the delivery process to send messages to the message distribution queue after the message distribution queue is full, that is, the delivery process enters a blocked state. This embodiment does not make any special limitations on this.

[0051] As a preferred embodiment, there are N delivery processes.

[0052] In order to improve the efficiency of the message distribution server in distributing messages to various message distribution queues when receiving messages produced by message producers, N delivery processes are set in this embodiment, and the N delivery processes correspond one-to-one to the N message distribution queues.

[0053] Please refer to Figure 2 , Figure 2 A structural diagram of a message system provided by the present invention.

[0054] The present invention also provides a message system, comprising:

[0055] Message distribution server 21;

[0056] The message storage server 22 is used to distribute the message to N message storage queues in sequence according to the mark and the preset queue storage order when receiving the message marked and sent by the message distribution server 21 according to the message consumption order, so that the previous message distribution and the current message distribution meet the requirements that each message storage queue stores the message in units of a preset number according to the preset queue storage order, where N is an integer not less than 2;

[0057] The message consumption server 23 is used to call N message pulling processes corresponding to N message storage queues one by one to pull messages from the message storage queues corresponding to each of them, and store the messages in N message consumption queues corresponding to the N message pulling processes one by one, and consume messages from the message consumption queues according to the tags and the preset queue consumption order, so that the last message consumption and the current message consumption satisfy that each message consumption queue consumes messages in units of a preset number according to the preset queue consumption order, and the preset queue consumption order of the N message consumption queues is the same as the preset queue storage order of the N message storage queues corresponding to each of them.

[0058] For the relevant introduction of the message system, please refer to the above embodiment, and this application will not go into details here.

[0059] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0060] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0061] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A message consumption method, characterized in that: Applied to a message system, the message system includes a message distribution server, a message storage server and a message consumption server, and the message consumption method includes: When the message storage server receives a message marked and sent by the message distribution server according to the message consumption order, it distributes the message to N message storage queues in sequence according to the mark and the preset queue storage order, so that the last message distribution and the current message distribution meet the requirements that each message storage queue stores messages in units of a preset number according to the preset queue storage order, where N is an integer not less than 2; The message consumption server calls N message pulling processes corresponding to N message storage queues one-to-one to pull the messages from the message storage queues corresponding to each of them, and stores the messages in N message consumption queues corresponding to the N message pulling processes one-to-one, and consumes the messages from the message consumption queues according to the tags and the preset queue consumption order, so that the last message consumption and the current message consumption satisfy that each message consumption queue consumes messages in units of the preset quantity according to the preset queue consumption order, and the preset queue consumption order of the N message consumption queues is the same as the preset queue storage order of the N message storage queues corresponding to each of them.

2. The message consumption method according to claim 1, characterized in that: Distributing the message to N message storage queues in sequence according to the mark and the preset queue storage order, so that the last message distribution and the current message distribution satisfy that each message storage queue stores messages in units of a preset number according to the preset queue storage order, including: If the number of messages in the last group of the last message distribution is less than the preset number, the message storage queue of the last group of message distribution is determined to be the starting message storage queue, and a first group of messages is selected from the messages according to the mark, the number of the first group of messages being the difference between the preset number and the number of messages in the last group of the last message distribution; the remaining messages are grouped in units of the preset number according to the mark; if the number of messages in the last group of the last message distribution is equal to the preset number, the next message storage queue of the message storage queue of the last group of message distribution is determined to be the starting message storage queue, and the messages are grouped in units of the preset number according to the mark; Taking the starting message storage queue as a benchmark, each group of the messages is distributed to each message storage queue in sequence according to the preset queue storage order; wherein, when the last group of messages last time and the first group of messages distributed this time exist in the starting message storage queue, the sum of the number of the last group of messages last time and the first group of messages distributed this time is the preset number.

3. The message consumption method according to claim 1, characterized in that: Also includes: When the message consumption queue corresponding to the message pulling process is full or there is no unpulled message in the message storage queue corresponding to the message pulling process, the message consumption server stops calling the message pulling process to pull the message from the message storage server.

4. The message consumption method according to claim 1, characterized in that: The process of setting the preset queue storage order includes: Mark the N message storage queues with serial numbers from small to large; The N message storage queues of the message storage server are sorted in ascending order of the sequence numbers.

5. The message consumption method according to any one of claims 1 to 4, characterized in that: The message distribution server marks and sends the message in the order in which the message is consumed, including: When the message distribution server receives a message sent by a message producer, it marks the message in the order in which the message is consumed, and calls the delivery process in the preset queue distribution order to distribute the message to N message distribution queues in sequence. The maximum number of messages that each message distribution queue can store is the preset number, and the messages in each message distribution queue are sent to N message storage queues corresponding one-to-one to the N message distribution queues. The preset queue distribution order of the N message distribution queues is the same as the preset queue storage order of the N message storage queues corresponding one-to-one to each of them.

6. The message consumption method according to claim 5, characterized in that: The number of delivery processes is N.

7. A message system, characterized in that: include: Message distribution server; A message storage server, configured to, upon receiving a message marked and sent by the message distribution server in accordance with the message consumption order, distribute the message to N message storage queues in sequence according to the mark and the preset queue storage order, so that the last message distribution and the current message distribution satisfy that each of the message storage queues stores messages in units of a preset number in accordance with the preset queue storage order, where N is an integer not less than 2; A message consumption server is used to call N message pulling processes corresponding to N message storage queues one by one to pull the messages from the message storage queues corresponding to each of them, and store the messages in N message consumption queues corresponding to the N message pulling processes one by one, and consume the messages from the message consumption queues according to the tags and the preset queue consumption order, so that the last message consumption and the current message consumption satisfy that each of the message consumption queues consumes messages in units of the preset number according to the preset queue consumption order, and the preset queue consumption order of the N message consumption queues is the same as the preset queue storage order of the N message storage queues corresponding to each of them.

Images