A message queue distribution management system and method based on data monitoring

Abstract

The present invention discloses a message queue distribution management system and method based on data monitoring, which relates to the field of data processing technology. The system includes: a message queue monitoring module, a message processing mode management module, a message compensation mode management module, a single compensation module, a batch compensation module, a failure alarm module and a message queue security monitoring module. The message processing status is compared with the rules of the message processing mode. When at least one indicator meets the threshold judgment rule, the high-speed processing mode is adopted, otherwise the normal processing mode is entered. When the message queue is in the normal processing mode, the failure rate of message processing in the second unit time period is used to determine whether to resend the messages in the database individually or in batches. The high-throughput message sending system is supervised by a three-level compensation mechanism of automatic retry, timed retry and manual intervention, ensuring the final consistency and high reliability of the messages while maintaining high throughput.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a message queue distribution management system and method based on data monitoring. Background Art

[0002] For business systems with high real-time requirements, whether it is task-based data distribution or real-time push through MQ, these methods have the following problems:

[0003] Low distribution efficiency: The performance of task-based distribution depends on the number of servers and the time it takes to set up the task, which can easily become a performance bottleneck. Especially in high-concurrency scenarios, the distribution efficiency drops significantly.

[0004] High transmission delay: The speed of data synchronization depends on the time interval set for the task. If the time interval is too long, the real-time distribution will be worse. If the time interval is short, frequent query services will consume server performance and it will be difficult to meet the real-time requirements.

[0005] Difficulty adapting to dynamic environments: Existing technologies lack effective mechanisms for handling network topology changes and dynamic node joining / exiting, and distribution bottlenecks cannot be dynamically evaluated, resulting in insufficient system robustness.

[0006] Low security: In existing message distribution, when data is sent through MQ, if MQ is congested, unconsumed messages cannot be processed in a timely manner, causing MQ data to swell until memory overflows, further causing MQ to crash.

[0007] Therefore, there is an urgent need for a data distribution and processing solution that is efficient, reliable and adaptable to dynamic environments. Summary of the Invention

[0008] The purpose of the present invention is to provide a method to solve the problems raised in the prior art.

[0009] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: a message queue distribution management system based on data monitoring;

[0010] The system includes: message queue monitoring module, message processing mode management module, message compensation mode management module, single compensation module, batch compensation module, failure alarm module and message queue security monitoring module;

[0011] The message queue monitoring module is used to aggregate messages sent by several different senders into the message queue and monitor the message processing status of the message queue;

[0012] The message processing mode management module is used to collect the message processing status of the message queue and adjust the processing mode of the message queue;

[0013] The message compensation mode management module is used to obtain the failure rate of message processing failures, issue a single compensation instruction when the failure rate is less than a threshold, and issue a batch compensation instruction when the failure rate is greater than the threshold;

[0014] A single compensation module resends a single message that fails to be sent;

[0015] The batch compensation module batch resends the messages that failed to be sent;

[0016] The failure alarm module counts the number of failed message processing attempts. When the number of failed message processing attempts exceeds the threshold, it alerts the relevant management personnel to check.

[0017] The message queue security monitoring module calculates the safety factor of the message queue and issues an alarm when the safety factor is lower than the threshold.

[0018] Furthermore, the message queue monitoring module includes: a backlog management unit, a message processing rate management unit, and a first failure rate management unit;

[0019] The backlog management unit is used to obtain the backlog of messages in the message queue in a time period between two moments;

[0020] The message processing rate management unit is used to manage the rate change of the message processing rate between any two moments;

[0021] The first failure rate management unit is used to calculate the ratio of messages that fail to be sent in the first unit time period to the total number of messages in the first unit time period.

[0022] In the embodiment, the backlog is measured by detecting the total amount of messages in the message queue at two moments, and the difference between the two total amounts of messages is the message backlog of the message queue.

[0023] Furthermore, the message processing mode management module includes: a message total amount threshold management unit, a recommended speed threshold management unit, a threshold management unit and a rule management unit;

[0024] The message total amount threshold management unit calculates the message total amount threshold by collecting performance parameters of the computer that processes the message queue and the average data size of the messages in the message queue, wherein the computer performance parameters include computer memory. The message total amount threshold L is calculated by L=(R×k1) / q0, where R represents the computer memory, k1 represents the first evaluation coefficient of the computer, and the condition 0<k1<1 is satisfied, and q0 represents the average data size of the messages in the message queue;

[0025] The recommended speed threshold management unit calculates the recommended speed threshold by collecting performance parameters of the computer, wherein the performance parameters of the computer include the number of cores of the processor and the average speed of each core in processing the number of messages per unit time, wherein calculating the recommended speed threshold P includes: P=code×k2×csp, where code represents the number of cores of the processor, k2 represents the second evaluation coefficient of the computer, k2 satisfies the condition that is an integer greater than 1, and csp represents the average speed of each core in processing the number of messages per unit time;

[0026] The threshold management unit is used to manage the judgment thresholds of the message processing mode. The judgment thresholds include the total message volume threshold, the recommended speed threshold, the backlog threshold, the rate change threshold and the processing failure rate threshold;

[0027] The rule management unit is used to manage the judgment rules of the message processing mode, and adjust the processing mode of the message queue to the normal processing mode and the high-speed processing mode. The normal processing mode is to push the newly received messages into the message queue in sequence, and the high-speed processing mode is to store the newly received messages in the database first. When at least one indicator meets the threshold judgment rule, the high-speed processing mode is adopted.

[0028] Setting the message total threshold too low will result in frequent dumping, increasing database pressure; setting the threshold too high may cause MQ downtime;

[0029] Setting the recommended speed threshold too high may cause target service overload; setting it too low may cause message backlogs;

[0030] If the failure rate threshold is too low, frequent rollbacks will be triggered, increasing database pressure.

[0031] Furthermore, the judgment rules include whether the number of messages in the message queue is greater than the total message threshold, whether the backlog of the message queue is greater than the backlog threshold, whether the rate change is less than the rate change threshold, and whether the failure rate of message processing is greater than the processing failure rate threshold.

[0032] Furthermore, the message compensation mode management module includes: a failure message management unit, a compensation mode management unit and a compensation instruction management unit;

[0033] The failed message management unit is used to store messages that cannot be pushed correctly by the push end of the message queue into the database;

[0034] The compensation mode management unit is used to manage a failure rate of message processing failures in a second unit time period, where the failure rate includes the ratio of the number of messages stored in the database in the second time period to the total number of messages received in the second time period, where the total number of received messages includes the number of messages received in the database and the number of messages received in the message queue;

[0035] The compensation instruction management unit is used to push instructions for storing messages in the database to the message queue when the processing mode of the message queue is the normal processing mode. When the failure rate of message processing failures in the second unit time period is less than a threshold, a single compensation instruction is issued; when the failure rate of message processing failures in the second unit time period is greater than a threshold, a batch compensation instruction is issued.

[0036] Furthermore, the single compensation module includes a failure count management unit and a sending interval management unit;

[0037] The failure count management unit is used to obtain the number of failures in processing any message.

[0038] The sending interval management unit is used to obtain the basic time interval and number of processing failures of push messages, and calculate the sending interval of the same message, where the sending interval is Tp, Tp=t0×2 n , where t0 represents the basic time interval and n represents the number of failures of the same message.

[0039] Furthermore, the batch compensation module includes a message priority calculation unit and a compensation queue management unit;

[0040] The message priority calculation unit is used to obtain the number of message processing failures, the importance coefficient of the message, and the network delay between the message receiver and the message queue, and calculate the priority coefficient of each message, where the priority coefficient is f, f = (n × imp) / t2, where imp represents the importance coefficient of the message, and t2 represents the network delay between the message receiver and the message queue. For different senders, corresponding importance coefficients are preset, the sender of the message is obtained, and the sender's importance coefficient is used as the importance coefficient of the message sent by the sender;

[0041] When the number of failures is greater and the network delay between the receiving end and the message queue is higher, the message will be pushed earlier so that the receiving end can receive the message in time and reduce the retention time of the message in the system;

[0042] The compensation queue management unit is used to obtain the rate at which the computer processes push messages and the compensation frequency to calculate the number of single push messages, where the number of single push messages is E, E=TP / fr, where TP represents the rate at which the computer processes push messages, and fr represents the compensation frequency.

[0043] Furthermore, the message queue security monitoring module includes: a rate evaluation value unit, a message volume evaluation unit, a second failure rate management unit, a safety factor calculation unit, and an alarm feedback unit;

[0044] The rate evaluation value unit is used to calculate the message processing speed evaluation value G1, G1=p0 / P, where p0 represents the processing speed of messages in the current message queue;

[0045] The message volume evaluation unit is used to calculate the evaluation value G2 of the message volume in the message queue, G2=L / l0, where l0 represents the number of messages remaining in the current message queue;

[0046] The second failure rate management unit is used to obtain the failure rate of the current message processing from the first failure rate management unit, and record the failure rate as φ;

[0047] The safety factor calculation unit is used to calculate the safety factor W of the current message processing, W = α × G1 + β × G2 + γ × (1-φ), where the conditions are met, α + β + γ = 1, α > β, α > γ;

[0048] The alarm feedback unit is used to compare the safety factor W with the alarm threshold ω, and when W < ω, an alarm prompt is given to relevant staff, where 0 < ω < 1.

[0049] In a high-speed message processing system, when the message processing speed is lower than the speed threshold, G1 is less than 1. When the number of messages remaining in the message queue is greater than the threshold, G2 is less than 1. Therefore, when the message queue processing efficiency is lower, G1 and G2 are smaller. At the same time, when the message processing success rate is lower, the system operation status is poor.

[0050] Therefore, the closer the W value is to 0, the greater the operating risk of the system.

[0051] In order to better apply the above system, a message queue distribution management method based on data monitoring is also proposed, which includes:

[0052] Step S1: Obtain the message processing status of the message queue of the message queue;

[0053] Step S2: Compare the message processing status with the message processing mode rules. If at least one indicator meets the threshold judgment rule, the high-speed processing mode is adopted; otherwise, the normal processing mode is entered.

[0054] Step S3: establishing a database to cache unprocessed messages, where the unprocessed messages include messages that failed to be processed and messages cached in high-speed processing mode;

[0055] Step S4: When the message queue is in normal processing mode, determine whether to resend the messages in the database individually or in batches according to the failure rate of message processing in the second unit time period;

[0056] Step S5: Monitor the number of message processing failures. When the number of message processing failures exceeds a threshold, request manual intervention.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] 1. The present invention monitors asynchronous message push by monitoring data services, realizes rapid release of communication with MQ message queues, and enhances message consumption rate;

[0059] 2. The present invention monitors the MQ service in real time through distribution status monitoring, and performs service degradation based on comprehensive indicators. After degradation due to abnormal data monitoring service consumption or MQ service abnormality, the data is saved to the database to ensure data security. At the same time, the distribution status monitoring serves as a compensation service to push the unsent data back to the message queue for the data monitoring service to monitor and push;

[0060] 3. The present invention uses a mathematical model to adjust the processing strategy in real time, and supervises the high-throughput message sending system through a three-level compensation mechanism of automatic retry, timed retry and manual intervention, ensuring the eventual consistency and high reliability of messages while maintaining high throughput. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 This is a structural diagram of a message queue distribution management system based on data monitoring according to the present invention;

[0062] Figure 2 A flow chart of a message queue distribution management method based on data monitoring according to the present invention;

[0063] Figure 3 The figure is a schematic diagram of an embodiment of a message queue distribution management system based on data monitoring according to the present invention. DETAILED DESCRIPTION

[0064] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0065] Example: Figure 1-Figure 3 As shown, the present invention provides a technical solution, a message queue distribution management system and method based on data monitoring;

[0066] The system includes: message queue monitoring module, message processing mode management module, message compensation mode management module, single compensation module, batch compensation module, failure alarm module and message queue security monitoring module;

[0067] The message queue monitoring module is used to aggregate messages sent by several different senders into the message queue and monitor the message processing status of the message queue.

[0068] The message queue monitoring module includes: a backlog management unit, a message processing rate management unit and a first failure rate management unit;

[0069] The backlog management unit is used to obtain the backlog of messages in the message queue in a time period between two moments;

[0070] The message processing rate management unit is used to manage the rate change of the message processing rate between any two moments;

[0071] The first failure rate management unit is used to calculate the ratio of messages that fail to be sent in the first unit time period to the total number of messages in the first unit time period.

[0072] In the embodiment, the backlog is measured by detecting the total amount of messages in the message queue at two times, and the difference between the two total amounts of messages is the message backlog of the message queue;

[0073] For example, at the first sampling moment, the total number of messages in the message queue is 90. 5 seconds later, the second sampling is performed. At the second sampling moment, the total number of messages in the message queue is 120. Then the message backlog is 120-90=30.

[0074] In the embodiment, when obtaining the instantaneous message processing rate of the message queue at the sampling moment, the change of the message processing rate between two consecutive sampling moments is obtained to obtain the rate change;

[0075] For example, at the first sampling moment, the message queue processes 50 messages per second. 5 seconds later, the second sampling is performed. At the second sampling moment, the message queue processes 60 messages per second. The rate change is (60-50) / 5=2 messages per second. 2 .

[0076] The message processing mode management module is used to collect the message processing status of the message queue and adjust the processing mode of the message queue.

[0077] The message processing mode management module includes: a message total amount threshold management unit, a recommended speed threshold management unit, a threshold management unit and a rule management unit;

[0078] The message total amount threshold management unit calculates the message total amount threshold by collecting performance parameters of the computer that processes the message queue and the average data size of the messages in the message queue, wherein the computer performance parameters include computer memory. The message total amount threshold L is calculated by L=(R×k1) / q0, where R represents the computer memory, k1 represents the first evaluation coefficient of the computer, and the condition 0<k1<1 is satisfied, and q0 represents the average data size of the messages in the message queue;

[0079] Preferably, in the embodiment, the value of k1 is 0.8;

[0080] The recommended speed threshold management unit calculates the recommended speed threshold by collecting performance parameters of the computer, wherein the performance parameters of the computer include the number of cores of the processor and the average speed of each core in processing the number of messages per unit time, wherein calculating the recommended speed threshold P includes: P=code×k2×csp, where code represents the number of cores of the processor, k2 represents the second evaluation coefficient of the computer, k2 satisfies the condition that is an integer greater than 1, and csp represents the average speed of each core in processing the number of messages per unit time;

[0081] Preferably, in the embodiment, the value of k2 is 100;

[0082] The threshold management unit is used to manage the judgment thresholds of the message processing mode. The judgment thresholds include the total message volume threshold, the recommended speed threshold, the backlog threshold, the rate change threshold and the processing failure rate threshold;

[0083] The rule management unit is used to manage the judgment rules of the message processing mode, and adjust the processing mode of the message queue to the normal processing mode and the high-speed processing mode. The normal processing mode is to push the newly received messages into the message queue in sequence, and the high-speed processing mode is to store the newly received messages in the database first. When at least one indicator meets the threshold judgment rule, the high-speed processing mode is adopted.

[0084] The judgment rules include whether the number of messages in the message queue is greater than the total message threshold, whether the backlog of the message queue is greater than the backlog threshold, whether the rate change is less than the rate change threshold, and whether the failure rate of message processing is greater than the processing failure rate threshold.

[0085] In this embodiment, the first unit time period is based on the current time. In the past 5 hours, the total number of message processing is 500, of which 20 messages failed to be processed. The failure rate of message processing failure is 20 / 500=4%;

[0086] Set the total message threshold in the message queue to 150, the backlog threshold to 50, and the rate change threshold to 1.5 messages per second. 2 , the processing failure rate threshold is 5%;

[0087] If there are 120 messages in the current message queue, 120 < 150, the backlog is 30 < 50, and the rate change is 2 messages / second 2 >1.5 pieces / second 2 , the failure rate is 4%<5%. Since all indicators have reached the threshold of the rules, the normal mode is used to process the message queue.

[0088] The message compensation mode management module is used to obtain the failure rate of message processing failures, issue a single compensation instruction when the failure rate is less than a threshold, and issue a batch compensation instruction when the failure rate is greater than the threshold.

[0089] In the embodiment, the method for obtaining the value of the second unit time period obtains the message that has been stored in the database the longest, takes the time when this message is generated as the starting time of the second unit time period, and records the time period between the current time and the starting time as the second unit time period.

[0090] The message compensation mode management module includes: a failure message management unit, a compensation mode management unit and a compensation instruction management unit;

[0091] The failed message management unit is used to store messages that cannot be pushed correctly by the push end of the message queue into the database;

[0092] The compensation mode management unit is used to manage a failure rate of message processing failures in a second unit time period, where the failure rate includes the ratio of the number of messages stored in the database in the second time period to the total number of messages received in the second time period, where the total number of received messages includes the number of messages received in the database and the number of messages received in the message queue;

[0093] The compensation instruction management unit is used to push instructions for storing messages in the database to the message queue when the processing mode of the message queue is the normal processing mode. When the failure rate of message processing failures in the second unit time period is less than a threshold, a single compensation instruction is issued; when the failure rate of message processing failures in the second unit time period is greater than a threshold, a batch compensation instruction is issued.

[0094] The single compensation module performs single retransmission on the message that fails to be sent. The single compensation module includes a failure number management unit and a sending interval management unit.

[0095] The failure count management unit is used to obtain the number of failures in processing any message.

[0096] The sending interval management unit is used to obtain the basic time interval and number of processing failures of push messages, and calculate the sending interval of the same message, where the sending interval is Tp, Tp=t0×2 n , where t0 represents the basic time interval and n represents the number of failures of the same message.

[0097] In the embodiment, the situations where message processing fails include: due to network fluctuations, the message in the message queue fails to be successfully pushed to the message receiving end, the user at the message receiving end is disconnected and fails to successfully receive the message;

[0098] In the embodiment, the basic interval is set to 5 minutes, and a total of 10 messages that failed to be pushed for the first time are collected, and the 10 messages are pushed to the message processing queue after 5 minutes;

[0099] After the second push, 8 messages were successfully processed and 2 messages were not successfully processed. The failure count of these two messages is increased by 1, and the failure count is recorded as 2;

[0100] Calculate the push interval Tp2 for the second push, Tp2=5×2 2 = 20 minutes, and push these two messages again after 20 minutes.

[0101] The batch compensation module batch resends the messages that failed to be sent. The batch compensation module includes a message priority calculation unit and a compensation queue management unit.

[0102] The message priority calculation unit is used to obtain the number of message processing failures, the importance coefficient of the message, and the network delay between the message receiver and the message queue, and calculate the priority coefficient of each message, where the priority coefficient is f, f = (n × imp) / t2, where imp represents the importance coefficient of the message, and t2 represents the network delay between the message receiver and the message queue. For different senders, corresponding importance coefficients are preset, the sender of the message is obtained, and the sender's importance coefficient is used as the importance coefficient of the message sent by the sender;

[0103] The compensation queue management unit is used to obtain the rate at which the computer processes push messages and the compensation frequency to calculate the number of single push messages, where the number of single push messages is E, E=TP / fr, where TP represents the rate at which the computer processes push messages, and fr represents the compensation frequency.

[0104] In the embodiment, for example, the computer can process 1000 push messages per minute, and the compensation frequency is 10 times per hour. Therefore, E = 1000 messages per minute ÷ 10 times per hour = 6000 messages per time. That is, within one hour, a push queue of 6000 messages is created every 6 minutes.

[0105] Get the priority coefficient of each message in the database, and push messages to the queue according to the priority coefficient from large to small.

[0106] The failure alarm module counts the number of failed processing of messages that failed to be sent. When the number of failed processing of any message exceeds the threshold, the relevant management personnel are reminded to check.

[0107] The message queue security monitoring module calculates the safety factor of the message queue and issues an alarm when the safety factor is lower than the threshold.

[0108] The message queue security monitoring module includes: a rate evaluation value unit, a message volume evaluation unit, a second failure rate management unit, a safety factor calculation unit and an alarm feedback unit;

[0109] The rate evaluation value unit is used to calculate the message processing speed evaluation value G1, G1=p0 / P, where p0 represents the processing speed of messages in the current message queue;

[0110] The message volume evaluation unit is used to calculate the evaluation value G2 of the message volume in the message queue, G2=L / l0, where l0 represents the number of messages remaining in the current message queue;

[0111] The second failure rate management unit is used to obtain the failure rate of the current message processing from the first failure rate management unit, and record the failure rate as φ;

[0112] The safety factor calculation unit is used to calculate the safety factor W of the current message processing, W = α × G1 + β × G2 + γ × (1-φ), where the conditions are met, α + β + γ = 1, α > β, α > γ;

[0113] The alarm feedback unit is used to compare the safety factor W with the alarm threshold ω, and when W < ω, an alarm prompt is given to relevant staff, where 0 < ω < 1.

[0114] More preferably, in the embodiment, the value of α is 0.4, the value of β is 0.3, and the value of γ is 0.3.

[0115] Methods include:

[0116] Step S1: Obtain the message processing status of the message queue of the message queue;

[0117] Step S2: Compare the message processing status with the message processing mode rules. If at least one indicator meets the threshold judgment rule, the high-speed processing mode is adopted; otherwise, the normal processing mode is entered.

[0118] Step S3: establishing a database to cache unprocessed messages, where the unprocessed messages include messages that failed to be processed and messages cached in high-speed processing mode;

[0119] Step S4: When the message queue is in normal processing mode, determine whether to resend the messages in the database individually or in batches according to the failure rate of message processing in the second unit time period;

[0120] Step S5: Monitor the number of message processing failures. When the number of message processing failures exceeds a threshold, request manual intervention.

[0121] The implementation process of another embodiment of the present invention includes:

[0122] 1. Send the data to be distributed to the MQ message queue through the source data service.

[0123] 2. The data monitoring cluster service monitors data. The monitoring service needs to be configured with the same MQ address as the source data service. The data monitoring service itself is one of the consumer groups. After monitoring data, it queries the data push rules and target service:

[0124] 2.1 First, check the rules to see if the current message meets the push conditions. When the check passes, the listening service will immediately send an asynchronous call to call the target service to ensure that the message is quickly released and communicated with MQ.

[0125] 2.2 If the target service fails to process the message after pushing it, the data monitoring service will save the unprocessed message to the database and record a pending message.

[0126] 2.3 If the message does not match the rules, it will not be distributed and will be discarded.

[0127] 3. Distribution status monitoring, as part of the monitoring service, monitors the consumption of MQ message queues in real time. When the data monitoring cluster service receives data from the source service, it first uses the distribution status monitoring to determine whether the warning value has been reached. The distribution status monitoring determines whether the MQ memory, consumption failure rate, and consumption delay meet the user-preset rules.

[0128] 3.1 Based on the monitoring indicators, it is judged whether the comprehensive indicators have reached the predetermined indicators. The total indicators provided by this system include the total number of messages, consumption delay, failure rate and other indicators. If the predetermined indicators are reached, the distribution monitoring service will immediately start the high-speed consumption mode: that is, immediately consume the messages in the MQ message queue, save the messages to the database, record N pending messages, and ensure that the MQ service will not crash due to a large number of messages.

[0129] 3.2 If the mq message queue indicator monitored by the monitoring service does not reach the predetermined indicator, the high-speed consumption mode will not be enabled.

[0130] 3.3 Regularly poll the messages to be pushed and push them back to the MQ message queue, and then enter the distribution logic.

[0131] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

Claims

1. A message queue distribution management system based on data monitoring, characterized by: The system includes: Message queue monitoring module, message processing mode management module, message compensation mode management module, single compensation module, batch compensation module, failure alarm module and message queue security monitoring module; The message queue monitoring module is used to collect messages sent by several different senders into the message queue and monitor the message processing status of the message queue; The message processing mode management module is used to collect the message processing status of the message queue and adjust the processing mode of the message queue; The message compensation mode management module is used to obtain the failure rate of message processing failures, issue a single compensation instruction when the failure rate is less than a threshold, and issue a batch compensation instruction when the failure rate is greater than the threshold; The single compensation module performs a single retransmission on the message that failed to be sent; The batch compensation module batch resends the messages that failed to be sent; The failure alarm module counts the number of failed processing of messages that failed to be sent, and when the number of failed processing of any message exceeds a threshold, it reminds relevant management personnel to check; The message queue security monitoring module calculates the safety factor of the message queue and issues an alarm when the safety factor is lower than a threshold; The system's management approach includes: Get the message processing status of the message queue; Compare the message processing status with the message processing mode rules. When at least one indicator meets the threshold judgment rule, the high-speed processing mode is adopted; otherwise, the normal processing mode is entered. The threshold determination rules include whether the number of messages in the message queue is greater than the total message threshold, whether the backlog of the message queue is greater than the backlog threshold, whether the rate change is less than the rate change threshold, and whether the failure rate of message processing is greater than the processing failure rate threshold; The normal processing mode is to push newly received messages into the message queue in order; The high-speed processing mode is to store newly received messages in the database first; Establish a database to cache unprocessed messages, where unprocessed messages include messages that failed to be processed and messages cached in high-speed processing mode; When the message queue is in normal processing mode, determining whether to resend the messages in the database individually or in batches according to the failure rate of message processing in the second unit time period; Monitor the number of message processing failures. When the number of message processing failures exceeds a threshold, request manual intervention.

2. The message queue distribution management system based on data monitoring according to claim 1, characterized in that: The message queue monitoring module includes: a backlog management unit, a message processing rate management unit and a first failure rate management unit; The backlog management unit is used to obtain the backlog of messages in the message queue in a time period between two moments; The message processing rate management unit is used to manage the rate change of the message processing rate at any two moments; The first failure rate management unit is used to calculate the ratio of messages that fail to be sent in the first unit time period to the total number of messages in the first unit time period.

3. The message queue distribution management system based on data monitoring according to claim 2, characterized in that: The message processing mode management module includes: a message total amount threshold management unit, a recommended speed threshold management unit, a threshold management unit and a rule management unit; The message total amount threshold management unit calculates the message total amount threshold by collecting performance parameters of the computer that processes the message queue and the average data size of the messages in the message queue, wherein the computer performance parameters include computer memory. The message total amount threshold L is calculated by L=(R×k1) / q0, where R represents the computer memory, k1 represents the first evaluation coefficient of the computer, and the condition 0<k1<1 is satisfied, and q0 represents the average data size of the messages in the message queue; The recommended speed threshold management unit calculates the recommended speed threshold by collecting performance parameters of the computer, wherein the performance parameters of the computer include the number of cores of the processor and the average speed of each core in processing the number of messages per unit time, wherein calculating the recommended speed threshold P includes: P=code×k2×csp, where code represents the number of cores of the processor, k2 represents the second evaluation coefficient of the computer, k2 satisfies the condition that is an integer greater than 1, and csp represents the average speed of each core in processing the number of messages per unit time; The threshold management unit is used to manage the judgment thresholds of the message processing mode. The judgment thresholds include the total message volume threshold, the recommended speed threshold, the backlog threshold, the rate change threshold and the processing failure rate threshold; The rule management unit is used to manage the judgment rules of the message processing mode, and adjust the processing mode of the message queue to the normal processing mode and the high-speed processing mode. The normal processing mode is to push the newly received messages into the message queue in sequence, and the high-speed processing mode is to store the newly received messages in the database first. When at least one indicator meets the threshold judgment rule, the high-speed processing mode is adopted.

4. The message queue distribution management system based on data monitoring according to claim 3, characterized in that: The message compensation mode management module includes: a failure message management unit, a compensation mode management unit and a compensation instruction management unit; The failed message management unit is used to store messages that cannot be pushed correctly by the push end of the message queue into the database; The compensation mode management unit is used to manage a failure rate of message processing failures in a second unit time period, wherein the failure rate includes the ratio of the number of messages stored in the database in the second time period to the total number of messages received in the second time period, wherein the total number of received messages includes the number of messages received in the database and the number of messages received in the message queue; The compensation instruction management unit is used to push instructions for storing messages in the database to the message queue when the processing mode of the message queue is the normal processing mode. When the failure rate of message processing failures in the second unit time period is less than a threshold, a single compensation instruction is issued; when the failure rate of message processing failures in the second unit time period is greater than a threshold, a batch compensation instruction is issued.

5. The message queue distribution management system based on data monitoring according to claim 4, characterized in that: A single compensation module includes a failure count management unit and a sending interval management unit; The failure count management unit is used to obtain the number of failures in processing any message. The sending interval management unit is used to obtain the basic time interval and number of processing failures of push messages, and calculate the sending interval of the same message, where the sending interval is Tp, Tp=t0×2n, where t0 represents the basic time interval and n represents the number of failures of the same message.

6. The message queue distribution management system based on data monitoring according to claim 5, characterized in that: The batch compensation module includes a message priority calculation unit and a compensation queue management unit; The message priority calculation unit is used to obtain the number of message processing failures, the importance coefficient of the message, and the network delay between the message receiver and the message queue, and calculate the priority coefficient of each message, where the priority coefficient is f, f = (n × imp) / t2, where imp represents the importance coefficient of the message, and t2 represents the network delay between the message receiver and the message queue. For different senders, corresponding importance coefficients are preset, the sender of the message is obtained, and the sender's importance coefficient is used as the importance coefficient of the message sent by the sender; The compensation queue management unit is used to obtain the rate at which the computer processes push messages and the compensation frequency to calculate the number of single push messages, where the number of single push messages is E, E=TP / fr, where TP represents the rate at which the computer processes push messages, and fr represents the compensation frequency.

7. The message queue distribution management system based on data monitoring according to claim 6, characterized in that: The message queue security monitoring module includes: a rate evaluation value unit, a message volume evaluation unit, a second failure rate management unit, a safety factor calculation unit and an alarm feedback unit; The rate evaluation value unit is used to calculate the message processing speed evaluation value G1, G1=p0 / P, where p0 represents the processing speed of messages in the current message queue; The message volume evaluation unit is used to calculate the evaluation value G2 of the message volume in the message queue, G2=L / l0, where l0 represents the number of messages remaining in the current message queue; The second failure rate management unit is used to obtain the failure rate of the current message processing from the first failure rate management unit, and record the failure rate as φ; The safety factor calculation unit is used to calculate the safety factor W of the current message processing, W = α × G1 + β × G2 + γ × (1-φ), where the conditions are met, α + β + γ = 1, α > β, α > γ; The alarm feedback unit is used to compare the safety factor W with the alarm threshold ω, and when W < ω, an alarm prompt is given to relevant staff, where 0 < ω < 1.

Images