Method for solving message accumulation

Abstract

The invention discloses a method for solving message accumulation, and relates to the field of message sending management. Comprising the steps of (1) configuring parameter information in a message processing flow, (2) monitoring the production speed of different types of messages, the consumption speed of the messages, the resource use condition of message consumers and the response condition after the messages are consumed, and (3) performing message flow control according to the message monitoring condition, limiting the message receiving and sending frequency, and controlling the message flow according to the response condition. The method comprises the following steps: step 1, carrying out message flow control by adopting a token bucket mode, and step 2, processing message accumulation: obtaining the accumulation condition of a message queue at the moment according to a message accumulation ratio, and continuously reducing the number of generated tokens in unit time for a message receiving token bucket in an accumulation state type when the message queue is about to be filled; performing retrieval de-duplication operation on the unforwarded messages; and carrying out disk storage on the stacked messages, emptying the number of tokens in a message receiving token bucket of the stacked messages, and not receiving the messages of the type any more.

Description

Technical Field

[0001] The invention discloses a method for solving message accumulation, and relates to the field of message sending management. Background Art

[0002] In complex distributed systems, different services or components are often tightly coupled. When a service needs to call another service, direct communication is usually established between them. However, this direct communication makes the system complex and difficult to maintain. And when the production speed of messages is greater than the consumption speed, a large number of messages will accumulate inside the message service over time, or when there are functional problems in the consumer logic of the message or the message consumption time is too long, it will also cause the messages to accumulate in the queue. Message accumulation takes up a lot of memory, disk space and network resources. When there are too many messages, it will have a negative impact on the performance of the server, resulting in slower processing speed and even exhaustion of system resources. Summary of the invention

[0003] The present invention aims at the problems of the prior art and provides a method for solving message accumulation. The method controls the reception of messages through a token bucket and transfers the messages to disk storage when the messages accumulate, so as to relieve the system pressure.

[0004] The specific scheme proposed by the present invention is:

[0005] The present invention provides a method for solving message accumulation, comprising:

[0006] Step 1: Configure the parameter information in the message processing process, including: message type, message queue length, message priority, message flow control ratio, message accumulation ratio, message maximum survival time and message consumption timeout time.

[0007] Step 2: Monitor the production speed of different types of messages, the consumption speed of messages, the resource usage of message consumers, and the response status after the messages are consumed.

[0008] Step 3: Perform message flow control based on the message monitoring situation and limit the frequency of message receiving and sending:

[0009] Specifically, the token bucket mode is used for message flow control:

[0010] Create two token buckets for each message type, one for sending a message and the other for receiving a message, and preset their maximum capacities.

[0011] For message reception, when the percentage of a certain type of message queue used exceeds the message flow control ratio, the reception speed of this type of message is limited: add tokens to the message receiving token bucket at a fixed rate until the number of tokens in the message receiving token bucket reaches the maximum capacity, and then stop adding tokens; sort the messages by priority, and determine whether there are still tokens of high-priority messages in the message receiving token bucket. If so, transfer the high-priority messages to the message queue, otherwise reject the message sending request;

[0012] For message sending, when the resource utilization rate of message consumers continues to increase, the frequency of message sending is limited: add tokens to the message sending token bucket at a fixed rate until the number of tokens in the message sending token bucket reaches the maximum capacity, and then stop adding tokens; sort the messages by priority, and determine whether there are still tokens of high-priority messages in the message sending token bucket. If yes, send the high-priority messages to the message consumers, otherwise do not send them for the time being;

[0013] Step 4: Process message accumulation: Refer to the message accumulation ratio to obtain the accumulation status of the message queue at this time. When the message queue is about to be filled, the number of tokens generated per unit time for the message receiving token bucket of the type that is already in the accumulation state is continuously reduced; the unforwarded messages are retrieved and deduplicated; the accumulated messages are stored on disk, and the number of tokens in the message receiving token bucket of the accumulated messages is cleared, and no messages of this type are received.

[0014] Further, in step 1 of the method for solving message accumulation, the message type is configured to distinguish different types of messages;

[0015] Configure the message queue length to identify the maximum number of messages of a certain type that can be stored in the system;

[0016] Configure the message priority to control the order of obtaining tokens from the token bucket during the message flow control phase;

[0017] Configure the message flow control ratio. The message flow control ratio indicates the percentage of received messages in the message queue to the total number of messages in the queue when performing message receiving flow control restriction.

[0018] Configure the message accumulation ratio. The message accumulation ratio refers to the ratio of the maximum number of messages stored in the message queue to the total number of messages in the message queue. When the total number of messages of a certain type exceeds the message accumulation ratio, it indicates that the queue of this type of messages is about to be filled.

[0019] Configure the maximum survival time of a message. The maximum survival time of a message refers to the maximum survival time of a certain type of message in the message queue. When the maximum survival time of a message is exceeded, it means that the message is outdated.

[0020] Configure the message consumption timeout to indicate the maximum processing time of this type of message that the consumer is expected to set.

[0021] Furthermore, in step 2 of the method for solving message accumulation, resource reporting requests from message consumers are periodically received, the CPU usage and memory usage of the consumers are recorded, and the message consumers' past resource usage data are combined to infer whether the message consumers are capable of processing the messages that need to be consumed; at the same time, normal consumption status submission requests from message consumers are received. After receiving the normal consumption status request from the message consumer, the message type for restoring the consumption status is restored according to the request, the message flow control is canceled, and the data stored on the disk for this message type is read, and the messages are filled into the message queue of this type to ensure that the message consumers process unconsumed messages in a timely manner.

[0022] Furthermore, in step 2 of the method for solving message backlog, the consumer response time of each message is monitored in real time, and the response request is fed back to the message middleware, and the messages that have not received consumer responses are polled, and the message consumption timeout time of this type of message is obtained according to the message type. If the time difference between the current time and the time when the message consumption starts exceeds the consumption timeout time, it indicates that the consumer cannot process the message, the connection with the message consumer will be disconnected, and the message forwarding of this type of message will be stopped.

[0023] Furthermore, in step 4 of the method for solving message accumulation, a search and deduplication operation is performed on the unforwarded messages, including: traversing all existing messages in the message queue, determining whether the timestamp of the message is in the list of consumed message timestamps, if so, it means that the message has been consumed, deleting the message from the message queue, and at the same time detecting the existence time of the message in the message queue, if the existence time of the message exceeds the maximum survival time of the message, determining that the message is an expired message, and deleting the message from the message queue.

[0024] The present invention also provides a device for solving message accumulation, comprising a message configuration module, a message monitoring module, a message flow control module and a message accumulation processing module.

[0025] The message configuration module configures the parameter information in the message processing process, including: message type, message queue length, message priority, message flow control ratio, message accumulation ratio, message maximum survival time and message consumption timeout time.

[0026] The message monitoring module monitors the production speed of different types of messages, the consumption speed of messages, the resource usage of message consumers, and the response status after the messages are consumed.

[0027] The message flow control module performs message flow control based on the message monitoring situation and limits the frequency of message receiving and sending:

[0028] Specifically, the token bucket mode is used for message flow control:

[0029] Create two token buckets for each message type, one for sending a message and the other for receiving a message, and preset their maximum capacities.

[0030] For message reception, when the percentage of a certain type of message queue used exceeds the message flow control ratio, the reception speed of this type of message is limited: add tokens to the message receiving token bucket at a fixed rate until the number of tokens in the message receiving token bucket reaches the maximum capacity, and then stop adding tokens; sort the messages by priority, and determine whether there are still tokens of high-priority messages in the message receiving token bucket. If so, transfer the high-priority messages to the message queue, otherwise reject the message sending request;

[0031] For message sending, when the resource utilization rate of message consumers continues to increase, the frequency of message sending is limited: add tokens to the message sending token bucket at a fixed rate until the number of tokens in the message sending token bucket reaches the maximum capacity, and then stop adding tokens; sort the messages by priority, and determine whether there are still tokens of high-priority messages in the message sending token bucket. If yes, send the high-priority messages to the message consumers, otherwise do not send them for the time being;

[0032] The message accumulation processing module handles message accumulation: refers to the message accumulation ratio to obtain the accumulation situation of the message queue at this time. When the message queue is about to be filled, the number of tokens generated per unit time for the message receiving token bucket of the type that is in the accumulation state is continuously reduced; the unforwarded messages are retrieved and deduplicated; the accumulated messages are stored on disk, and the number of tokens in the message receiving token bucket of the accumulated messages is cleared, and no messages of this type are received.

[0033] Furthermore, the message configuration module of the device for solving message accumulation configures the message type to distinguish different types of messages;

[0034] Configure the message queue length to identify the maximum number of messages of a certain type that can be stored in the system;

[0035] Configure the message priority to control the order of obtaining tokens from the token bucket during the message flow control phase;

[0036] Configure the message flow control ratio. The message flow control ratio indicates the percentage of received messages in the message queue to the total number of messages in the queue when performing message receiving flow control restriction.

[0037] Configure the message accumulation ratio. The message accumulation ratio refers to the ratio of the maximum number of messages stored in the message queue to the total number of messages in the message queue. When the total number of messages of a certain type exceeds the message accumulation ratio, it indicates that the queue of this type of messages is about to be filled.

[0038] Configure the maximum survival time of a message. The maximum survival time of a message refers to the maximum survival time of a certain type of message in the message queue. When the maximum survival time of a message is exceeded, it means that the message is outdated.

[0039] Configure the message consumption timeout to indicate the maximum processing time of this type of message that the consumer is expected to set.

[0040] Furthermore, the message monitoring module of the device for solving message backlog also periodically receives resource reporting requests from message consumers, records the CPU usage and memory usage of consumers, and infers whether the message consumers are capable of processing the messages that need to be consumed in combination with the message consumers' past resource usage data; at the same time, it receives the message consumer's normal consumption status submission request, and after receiving the message consumer's normal consumption status request, restores the consumption status according to the message type requested, cancels the message flow control, reads the data stored on the disk for this message type, and fills the message into this type of message queue to ensure that the message consumer processes unconsumed messages in a timely manner.

[0041] Furthermore, the message monitoring module of the device for solving message backlog monitors the consumer response time of each message in real time, and feeds back the response request to the message middleware, polls the messages for which no consumer response has been received, and obtains the message consumption timeout time of this type of message according to the message type. If the time difference between the current time and the start time of message consumption exceeds the consumption timeout time, it indicates that the consumer cannot process the message, and the connection with the message consumer will be disconnected, and the message forwarding of this type of message will be stopped.

[0042] Furthermore, a message accumulation processing module of the device for solving message accumulation performs retrieval and deduplication operations on unforwarded messages, including: traversing all existing messages in the message queue, determining whether the timestamp of the message is in the list of consumed message timestamps, if so, it means that the message has been consumed, deleting the message from the message queue, and at the same time detecting the existence time of the message in the message queue, if the existence time of the message exceeds the maximum survival time of the message, determining that the message is an expired message, and deleting the message from the message queue.

[0043] The benefits of the present invention are:

[0044] Using the token bucket mode to control the message receiving mechanism can stabilize the frequency of message receiving and prevent the system from crashing due to a sudden increase in the message middle price. When the pressure on the message consumer server is too high, the token bucket mode is used to limit the number of messages forwarded to the message consumer, reduce the pressure on the consumer server, and ensure its normal operation;

[0045] Ability to obtain performance information of all message consumers in real time, so that administrators can understand the message processing capabilities of message consumers;

[0046] Messages in a backlog state are transferred from the message queue to disk storage to reduce the pressure on the message middleware. After the message consumer repairs the consumption logic, the messages on the disk are transferred to the message queue to ensure that the messages are not lost. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a schematic diagram of the application process of the method of the present invention. DETAILED DESCRIPTION

[0048] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0049] Example 1

[0050] The present invention provides a method for solving message accumulation, comprising:

[0051] Step 1: Configure the parameter information in the message processing process, including: message type, message queue length, message priority, message flow control ratio, message accumulation ratio, message maximum survival time and message consumption timeout.

[0052] The message type is used to distinguish different types of messages. For a certain type of message, when the message consumer function is blocked, the message middleware will no longer forward this type of message to the message consumer until the message consumer actively confirms that the blocking problem of this type of message has been resolved.

[0053] The message queue length is used to identify the maximum number of messages of a certain type that can be stored in the system.

[0054] Message priority is used to control the order of obtaining tokens from the token bucket during the message flow control stage. The message producer will automatically generate message priority information when generating a message.

[0055] Message flow control ratio: The message flow control ratio indicates the percentage of the number of received messages in the message queue to the total number of queues when message receiving flow control is performed. When the percentage of the number of received messages to the total number of queues exceeds the flow control ratio, flow control is performed on the receiving function of this type of messages.

[0056] Message accumulation ratio: The message accumulation ratio refers to the ratio of the maximum number of messages that can be stored in a message queue to the total number of messages in the message queue. When the total number of messages of a certain type exceeds the message accumulation ratio, it indicates that the queue of this type of messages is about to be filled.

[0057] Configure the maximum survival time of a message. The maximum survival time of a message refers to the maximum survival time of a certain type of message in the message queue. When the maximum survival time of a message is exceeded, it means that the message is outdated.

[0058] Configure the message consumption timeout to indicate the maximum processing time of this type of message that the consumer is expected to set.

[0059] Step 2: Monitor the production speed of different types of messages, the consumption speed of messages, the resource usage of message consumers, and the response status after the messages are consumed.

[0060] In step 2, resource reporting requests from message consumers are also periodically received, and the CPU usage and memory usage of consumers are recorded. Combined with the past resource usage data of message consumers, it is inferred whether the message consumers are capable of processing the messages that need to be consumed. At the same time, normal consumption status submission requests from message consumers are received. After receiving the normal consumption status request from the message consumer, the message type that restores the consumption status is restored according to the request, the message flow control is canceled, and the data stored on the disk for this message type is read, and the message is filled into the message queue of this type to ensure that the message consumer processes the unconsumed messages in a timely manner.

[0061] When the message middleware forwards the message to the consumer, in step 2, the timestamp information of the message can also be stored in the list of consumed messages of the corresponding message type, which can be used to filter out duplicate messages caused by network delays or message producer message retry generation mechanisms during subsequent message accumulation processing.

[0062] In step 2, the consumer response time of each message is also monitored in real time, and the response request is fed back to the message middleware. The messages that have not received consumer responses are polled, and the message consumption timeout of this type of message is obtained according to the message type. If the time difference between the current time and the time when the message starts to be consumed exceeds the consumption timeout, it indicates that the consumer cannot process the message, and the connection with the message consumer will be disconnected, and the message forwarding of this type of message will be stopped.

[0063] Step 3: Perform message flow control based on the message monitoring situation to limit the frequency of message reception and sending. For message reception, when the percentage of a certain type of message queue used does not exceed the message flow control ratio, the message middleware receives all messages sent by this type of message producer by default. When the percentage of a certain type of message queue used exceeds the message flow control ratio, the message receiving function is intervened in the flow control process to limit the message receiving speed. For message sending, when the monitoring finds that the CPU and memory usage of the system resources of the message consumer is too high, the message sending function is intervened in the flow control process. By limiting the frequency of message sending, the stability of the message consumer's system resource usage is guaranteed, and the message consumer is prevented from running out of system resources due to the need to process a large number of messages.

[0064] Specifically, the token bucket mode is used for message flow control:

[0065] Create two token buckets for each message type, one for sending a message and the other for receiving a message, and preset their maximum capacities.

[0066] For message reception, when the percentage of a certain type of message queue used exceeds the message flow control ratio, the reception speed of this type of message is limited: add tokens to the message receiving token bucket at a fixed rate until the number of tokens in the message receiving token bucket reaches the maximum capacity, and then stop adding tokens; sort the messages by priority, and determine whether there are still tokens of high-priority messages in the message receiving token bucket. If so, transfer the high-priority messages to the message queue, otherwise reject the message sending request;

[0067] For message sending, when the resource utilization rate of message consumers continues to increase, the frequency of message sending is limited: tokens are added to the message sending token bucket at a fixed rate until the number of tokens in the message sending token bucket reaches the maximum capacity, and then adding tokens is stopped; the messages are prioritized and it is determined whether there are still tokens for high-priority messages in the message sending token bucket. If so, the high-priority messages are sent to the message consumers, otherwise they are not sent for the time being. By controlling the number of tokens in the token bucket, the frequency of message consumption is stabilized.

[0068] Step 4: Process message accumulation: Refer to the message accumulation ratio to obtain the accumulation status of the message queue at this time. When the message queue is about to be filled, the number of tokens generated per unit time for the message receiving token bucket of the type that is already in the accumulation state is continuously reduced; the unforwarded messages are retrieved and deduplicated; the accumulated messages are stored on disk, and the number of tokens in the message receiving token bucket of the accumulated messages is cleared, and no messages of this type are received.

[0069] The retrieval and deduplication operation on the unforwarded messages in step 4 may include: traversing all existing messages in the message queue, determining whether the timestamp of the message is in the list of consumed message timestamps, if so, it means that the message has been consumed, deleting the message from the message queue, and at the same time detecting the existence time of the message in the message queue, if the existence time of the message exceeds the maximum survival time of the message, determining that the message is an expired message, and deleting the message from the message queue.

[0070] Example 2

[0071] The present invention also provides a device for solving message accumulation, comprising a message configuration module, a message monitoring module, a message flow control module and a message accumulation processing module.

[0072] The message configuration module configures the parameter information in the message processing process, including: message type, message queue length, message priority, message flow control ratio, message accumulation ratio, message maximum survival time and message consumption timeout time.

[0073] The message monitoring module monitors the production speed of different types of messages, the consumption speed of messages, the resource usage of message consumers, and the response status after the messages are consumed.

[0074] The message flow control module performs message flow control based on the message monitoring situation and limits the frequency of message receiving and sending:

[0075] Specifically, the token bucket mode is used for message flow control:

[0076] Create two token buckets for each message type, one for sending a message and the other for receiving a message, and preset their maximum capacities.

[0077] For message reception, when the percentage of a certain type of message queue used exceeds the message flow control ratio, the reception speed of this type of message is limited: add tokens to the message receiving token bucket at a fixed rate until the number of tokens in the message receiving token bucket reaches the maximum capacity, and then stop adding tokens; sort the messages by priority, and determine whether there are still tokens of high-priority messages in the message receiving token bucket. If so, transfer the high-priority messages to the message queue, otherwise reject the message sending request;

[0078] For message sending, when the resource utilization rate of message consumers continues to increase, the frequency of message sending is limited: add tokens to the message sending token bucket at a fixed rate until the number of tokens in the message sending token bucket reaches the maximum capacity, and then stop adding tokens; sort the messages by priority, and determine whether there are still tokens of high-priority messages in the message sending token bucket. If yes, send the high-priority messages to the message consumers, otherwise do not send them for the time being;

[0079] The message accumulation processing module handles message accumulation: refers to the message accumulation ratio to obtain the accumulation situation of the message queue at this time. When the message queue is about to be filled, the number of tokens generated per unit time for the message receiving token bucket of the type that is in the accumulation state is continuously reduced; the unforwarded messages are retrieved and deduplicated; the accumulated messages are stored on disk, and the number of tokens in the message receiving token bucket of the accumulated messages is cleared, and no messages of this type are received.

[0080] As the information interaction and execution process between the modules of the above-mentioned device are based on the same concept as the embodiment of the method of the present invention, the specific contents can be found in the description of the embodiment of the method of the present invention and will not be repeated here.

[0081] Likewise, the benefits of the device of the present invention are:

[0082] Using the token bucket mode to control the message receiving mechanism can stabilize the frequency of message receiving and prevent the system from crashing due to a sudden increase in the message middle price. When the pressure on the message consumer server is too high, the token bucket mode is used to limit the number of messages forwarded to the message consumer, reduce the pressure on the consumer server, and ensure its normal operation;

[0083] Ability to obtain performance information of all message consumers in real time, so that administrators can understand the message processing capabilities of message consumers;

[0084] Messages in a backlog state are transferred from the message queue to disk storage to reduce the pressure on the message middleware. After the message consumer repairs the consumption logic, the messages on the disk are transferred to the message queue to ensure that the messages are not lost.

[0085] It should be noted that not all steps and modules in the above-mentioned processes and device structures are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The system structure described in the above-mentioned embodiments can be a physical structure or a logical structure, that is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or some components in multiple independent devices may be implemented together.

[0086] The above-described embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or changes made by those skilled in the art based on the present invention are within the protection scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. A method for solving message accumulation, characterized in that include: Step 1: Configure the parameter information in the message processing process, including: message type, message queue length, message priority, message flow control ratio, message accumulation ratio, message maximum survival time and message consumption timeout time. Step 2: Monitor the production speed of different types of messages, the consumption speed of messages, the resource usage of message consumers, and the response status after the messages are consumed. Step 3: Perform message flow control based on the message monitoring situation and limit the frequency of message receiving and sending: Specifically, the token bucket mode is used for message flow control: Create two token buckets for each message type, one for sending a message and the other for receiving a message, and preset their maximum capacities. For message reception, when the percentage of a certain type of message queue used exceeds the message flow control ratio, the reception speed of this type of message is limited: add tokens to the message receiving token bucket at a fixed rate until the number of tokens in the message receiving token bucket reaches the maximum capacity, and then stop adding tokens; sort the messages by priority, and determine whether there are still tokens of high-priority messages in the message receiving token bucket. If so, transfer the high-priority messages to the message queue, otherwise reject the message sending request; For message sending, when the resource utilization rate of message consumers continues to increase, the frequency of message sending is limited: add tokens to the message sending token bucket at a fixed rate until the number of tokens in the message sending token bucket reaches the maximum capacity, and then stop adding tokens; sort the messages by priority, and determine whether there are still tokens of high-priority messages in the message sending token bucket. If yes, send the high-priority messages to the message consumers, otherwise do not send them for the time being; Step 4: Process message accumulation: Refer to the message accumulation ratio to obtain the accumulation status of the message queue at this time. When the message queue is about to be filled, the number of tokens generated per unit time for the message receiving token bucket of the type that is already in the accumulation state is continuously reduced; the unforwarded messages are retrieved and deduplicated; the accumulated messages are stored on disk, and the number of tokens in the message receiving token bucket of the accumulated messages is cleared, and no messages of this type are received.

2. A method for solving message accumulation according to claim 1, characterized in that In step 1, configure the message type to distinguish different types of messages; Configure the message queue length to identify the maximum number of messages of a certain type that can be stored in the system; Configure the message priority to control the order of obtaining tokens from the token bucket during the message flow control phase; Configure the message flow control ratio. The message flow control ratio indicates the percentage of received messages in the message queue to the total number of messages in the queue when performing message receiving flow control restriction. Configure the message accumulation ratio. The message accumulation ratio refers to the ratio of the maximum number of messages stored in the message queue to the total number of messages in the message queue. When the total number of messages of a certain type exceeds the message accumulation ratio, it indicates that the queue of this type of messages is about to be filled. Configure the maximum survival time of a message. The maximum survival time of a message refers to the maximum survival time of a certain type of message in the message queue. When the maximum survival time of a message is exceeded, it means that the message is outdated. Configure the message consumption timeout to indicate the maximum processing time of this type of message that the consumer is expected to set.

3. A method for solving message accumulation according to claim 1, characterized in that In step 2, resource reporting requests from message consumers are also periodically received, and the CPU usage and memory usage of consumers are recorded. Combined with the past resource usage data of message consumers, it is inferred whether the message consumers are capable of processing the messages that need to be consumed. At the same time, normal consumption status submission requests from message consumers are received. After receiving the normal consumption status request from the message consumer, the message type that restores the consumption status is restored according to the request, the message flow control is canceled, and the data stored on the disk for this message type is read, and the message is filled into the message queue of this type to ensure that the message consumer processes the unconsumed messages in a timely manner.

4. A method for solving message accumulation according to claim 1, characterized in that In step 2, the consumer response time of each message is monitored in real time, and the response request is fed back to the message middleware. The messages that have not received consumer responses are polled, and the message consumption timeout of this type of message is obtained according to the message type. If the difference between the current time and the time when the message starts to be consumed exceeds the consumption timeout, it means that the consumer cannot process the message, and the connection with the message consumer will be disconnected, and the message forwarding of this type of message will be stopped.

5. A method for solving message accumulation according to claim 1, characterized in that In step 4, the unforwarded messages are retrieved and deduplicated, including: traversing all existing messages in the message queue, determining whether the timestamp of the message is in the list of consumed message timestamps, if yes, it means that the message has been consumed, deleting the message from the message queue, and detecting the existence time of the message in the message queue at the same time, if the existence time of the message exceeds the maximum survival time of the message, determining that the message is an expired message, and deleting the message from the message queue.

6. A device for solving message accumulation, characterized in that It includes message configuration module, message monitoring module, message flow control module and message accumulation processing module. The message configuration module configures the parameter information in the message processing process, including: message type, message queue length, message priority, message flow control ratio, message accumulation ratio, message maximum survival time and message consumption timeout time. The message monitoring module monitors the production speed of different types of messages, the consumption speed of messages, the resource usage of message consumers, and the response status after the messages are consumed. The message flow control module performs message flow control based on the message monitoring situation and limits the frequency of message receiving and sending: Specifically, the token bucket mode is used for message flow control: Create two token buckets for each message type, one for sending a message and the other for receiving a message, and preset their maximum capacities. For message reception, when the percentage of a certain type of message queue used exceeds the message flow control ratio, the reception speed of this type of message is limited: add tokens to the message receiving token bucket at a fixed rate until the number of tokens in the message receiving token bucket reaches the maximum capacity, and then stop adding tokens; sort the messages by priority, and determine whether there are still tokens of high-priority messages in the message receiving token bucket. If so, transfer the high-priority messages to the message queue, otherwise reject the message sending request; For message sending, when the resource utilization rate of message consumers continues to increase, the frequency of message sending is limited: add tokens to the message sending token bucket at a fixed rate until the number of tokens in the message sending token bucket reaches the maximum capacity, and then stop adding tokens; sort the messages by priority, and determine whether there are still tokens of high-priority messages in the message sending token bucket. If yes, send the high-priority messages to the message consumers, otherwise do not send them for the time being; The message accumulation processing module handles message accumulation: refers to the message accumulation ratio to obtain the accumulation situation of the message queue at this time. When the message queue is about to be filled, the number of tokens generated per unit time for the message receiving token bucket of the type that is in the accumulation state is continuously reduced; the unforwarded messages are retrieved and deduplicated; the accumulated messages are stored on disk, and the number of tokens in the message receiving token bucket of the accumulated messages is cleared, and no messages of this type are received.

7. The device for solving message accumulation according to claim 1, characterized in that The message configuration module configures the message type to distinguish different types of messages; Configure the message queue length to identify the maximum number of messages of a certain type that can be stored in the system; Configure the message priority to control the order of obtaining tokens from the token bucket during the message flow control phase; Configure the message flow control ratio. The message flow control ratio indicates the percentage of received messages in the message queue to the total number of messages in the queue when performing message receiving flow control restriction. Configure the message accumulation ratio. The message accumulation ratio refers to the ratio of the maximum number of messages stored in the message queue to the total number of messages in the message queue. When the total number of messages of a certain type exceeds the message accumulation ratio, it indicates that the queue of this type of messages is about to be filled. Configure the maximum survival time of a message. The maximum survival time of a message refers to the maximum survival time of a certain type of message in the message queue. When the maximum survival time of a message is exceeded, it means that the message is outdated. Configure the message consumption timeout to indicate the maximum processing time of this type of message that the consumer is expected to set.

8. The device for solving message accumulation according to claim 6, characterized in that the message The monitoring module also periodically receives resource reporting requests from message consumers, records the consumer's CPU usage and memory usage, and combines the message consumer's past resource usage data to infer whether the message consumer is capable of processing the messages that need to be consumed; at the same time, it receives the message consumer's normal consumption status submission request. After receiving the message consumer's normal consumption status request, it restores the consumption status according to the message type requested, cancels the message flow control, reads the data stored on the disk for this message type, and fills the message into this type of message queue to ensure that the message consumer processes unconsumed messages in a timely manner.

9. The device for solving message accumulation according to claim 6, characterized in that The message monitoring module monitors the consumer response time of each message in real time, and feeds back the response request to the message middleware, polls the messages that have not received the consumer response, and obtains the message consumption timeout of this type of message according to the message type. If the difference between the current time and the time when the message starts to be consumed exceeds the consumption timeout, it indicates that the consumer cannot process the message, and the connection with the message consumer will be disconnected, and the message forwarding of this type of message will be stopped.

10. The device for solving message accumulation according to claim 6, characterized in that the message The accumulation processing module performs retrieval and deduplication operations on the unforwarded messages, including: traversing all existing messages in the message queue, determining whether the timestamp of the message is in the list of consumed message timestamps, if yes, it means that the message has been consumed, deleting the message from the message queue, and detecting the existence time of the message in the message queue at the same time, if the existence time of the message exceeds the maximum survival time of the message, determining that the message is an expired message, and deleting the message from the message queue.

Images