Message queue distribution management system and method based on data monitoring

Abstract

The invention discloses a message queue distribution management system and method based on data monitoring, and relates to the technical field of data processing, and the system comprises a message queue monitoring module, a message processing mode management module, a message compensation mode management module, a single compensation module, a batch compensation module, a failure alarm module and a message queue safety monitoring module. And comparing the message processing state with a rule of a message processing mode, when at least one index meets a threshold judgment rule, adopting a high-speed processing mode, otherwise, entering a common processing mode, and when the message queue is in the common processing mode, according to the failure rate of message processing in a second unit time period, performing message processing on the message queue according to the failure rate of message processing in the second unit time period. According to the method, whether the messages in the database are subjected to single retransmission or batch retransmission is judged, a high-throughput message sending system is supervised through a three-level compensation mechanism of automatic retry, timed retry and manual intervention, and high throughput is kept while final consistency and high reliability of the messages are ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a message queue distribution management system and method based on data monitoring. Background Art

[0002] For business systems with high real-time requirements, whether it is task-based data distribution or real-time push through MQ, these methods have the following problems: Low distribution efficiency: The performance of task-based distribution depends on the number of servers and the time of task setting, which can easily become a performance bottleneck. Especially in high-concurrency scenarios, the distribution efficiency drops significantly.

[0003] High transmission delay: The speed of data synchronization depends on the time interval of the task setting. If the time interval is too long, the real-time distribution will be worse. If the time interval is short, frequent query services will consume server performance and it will be difficult to meet the real-time requirements.

[0004] Difficult to adapt to dynamic environments: Existing technologies lack effective mechanisms for handling network topology changes and dynamic node joining / exiting, and distribution bottlenecks cannot be dynamically evaluated, resulting in insufficient system robustness.

[0005] Low security: In the existing message distribution, when data is sent by MQ, if MQ is congested, the unconsumed messages cannot be processed in time, causing the MQ data to expand until the memory overflows, further causing the MQ to crash.

[0006] Therefore, there is an urgent need for a data distribution and processing solution that is efficient, reliable and adaptable to dynamic environments. Summary of the invention

[0007] The purpose of the present invention is to provide a method to solve the problems raised in the prior art.

[0008] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: a message queue distribution management system based on data monitoring; The system includes: a message queue monitoring module, a message processing mode management module, a message compensation mode management module, a single compensation module, a batch compensation module, a failure alarm module and a message queue security monitoring module; The message queue monitoring module is used to collect messages sent by several different senders into the message queue and monitor the message processing status of the message queue; The message processing mode management module is used to collect the message processing status of the message queue and adjust the processing mode of the message queue; The message compensation mode management module is used to obtain the failure rate of message processing failures, issue a single compensation instruction when the failure rate is less than a threshold, and issue a batch compensation instruction when the failure rate is greater than the threshold; A single compensation module performs a single retransmission of a message that fails to be sent; The batch compensation module batch resends the messages that failed to be sent; The failure alarm module counts the number of failed processing of messages that failed to be sent. When the number of failed processing of any message exceeds the threshold, the relevant management personnel are reminded to check; The message queue security monitoring module calculates the security factor of the message queue and issues an alarm when the security factor is lower than the threshold.

[0009] Further, the message queue monitoring module includes: a backlog management unit, a message processing rate management unit and a failure rate management unit; The backlog management unit is used to obtain the backlog of messages in the message queue in a time period between two moments; The message processing rate management unit is used to manage the rate change of the message processing rate between any two moments; The failure rate management unit is used to calculate the ratio of messages that fail to be sent in the first unit time period to the total number of messages in the first unit time period.

[0010] In the embodiment, the backlog is measured by detecting the total amount of messages in the message queue at two times, and the difference between the two total amounts of messages is the message backlog of the message queue.

[0011] Further, the message processing mode management module includes: a message total amount threshold management unit, a recommended speed threshold management unit, a threshold management unit and a rule management unit; The message total amount threshold management unit calculates the message total amount threshold by collecting the performance parameters of the computer that processes the message queue and the average data size of the messages in the message queue, wherein the performance parameters of the computer include the computer memory, and the calculation of the message total amount threshold L includes, L=(R×k1) / q0, R represents the computer memory, k1 represents the first evaluation coefficient of the computer, and satisfies the condition 0<k1<1, q0 represents the average data size of the messages in the message queue; The recommended speed threshold management unit calculates the recommended speed threshold by collecting the performance parameters of the computer, wherein the performance parameters of the computer include the number of cores of the processor and the average speed of each core processing the number of messages per unit time, wherein the calculation of the recommended speed threshold P includes, P=code×k2×csp, code represents the number of cores of the processor, k2 represents the second evaluation coefficient of the computer, k2 satisfies the condition that it is an integer greater than 1, and csp represents the average speed of each core processing the number of messages per unit time; The threshold management unit is used to manage the judgment threshold of the message processing mode, and the judgment threshold includes the total message threshold, the recommended speed threshold, the backlog threshold, the rate change threshold and the processing failure rate threshold; The rule management unit is used to manage the judgment rules of the message processing mode, and adjust the processing mode of the message queue to the normal processing mode and the high-speed processing mode. The normal processing mode is to push the newly received messages into the message queue in sequence, and the high-speed processing model is to store the newly received messages into the database first. When at least one indicator meets the threshold judgment rule, the high-speed processing mode is adopted.

[0012] If the threshold for total message volume is set too low, it will cause frequent triggering of dumping and increase database pressure; if the threshold is set too high, it may cause MQ downtime; Setting the recommended speed threshold too high may cause target service overload; setting it too low may cause message backlog; If the failure rate threshold is too low, it will cause frequent triggering of dumping, increasing the database pressure; Furthermore, the judgment rules include whether the number of messages in the message queue is greater than the total message threshold, whether the backlog of the message queue is greater than the backlog threshold, whether the rate change is less than the rate change threshold, and whether the failure rate of processing messages is greater than the processing failure rate threshold.

[0013] Further, the message compensation mode management module includes: a failure message management unit, a compensation mode management unit and a compensation instruction management unit; The failed message management unit is used to store the messages that the push end of the message queue cannot push correctly into the database; The compensation mode management unit is used to manage the failure rate of message processing failures in the second unit time period, the failure rate includes the number of messages stored in the database in the second time period to the total number of messages received in the second time period, the total number of received messages includes the number of messages received by the database and the number of messages received by the message queue; The compensation instruction management unit is used to push instructions for storing messages in a database to the message queue when the processing mode of the message queue is the normal processing mode. When the failure rate of message processing failures in the second unit time period is less than a threshold, a single compensation instruction is issued; and when the failure rate of message processing failures in the second unit time period is greater than a threshold, a batch compensation instruction is issued.

[0014] Further, the single compensation module includes a failure number management unit and a sending interval management unit; The failure count management unit is used to obtain the number of failures in processing any message. The sending interval management unit is used to obtain the basic time interval and number of processing failures of push messages, and calculate the sending interval of the same message, where the sending interval is Tp, Tp=t0×2 n , where t0 represents the basic time interval and n represents the number of failures of the same message.

[0015] Further, the batch compensation module includes a message priority calculation unit and a compensation queue management unit; The message priority calculation unit is used to obtain the number of message processing failures, the importance coefficient of the message and the network delay between the message receiver and the message queue, and calculate the priority coefficient of each message, where the priority coefficient is f, f=(n×imp) / t2, where imp represents the importance coefficient of the message, and t2 represents the network delay between the message receiver and the message queue. For different senders, corresponding importance coefficients are preset, the sender of the message is obtained, and the sender's importance coefficient is used as the importance coefficient of the message sent by the sender; When the number of failures is greater and the network delay between the receiving end and the message queue is higher, the message will be pushed earlier so that the receiving end can receive the message in time and reduce the retention time of the message in the system; The compensation queue management unit is used to obtain the rate at which the computer processes push messages and the compensation frequency to calculate the number of single push messages, where the number of single push messages is E, E=TP / fr, where TP represents the rate at which the computer processes push messages, and fr represents the compensation frequency.

[0016] Further, the message queue security monitoring module includes: a rate evaluation value unit, a message volume evaluation unit, a failure rate management unit, a safety factor calculation unit and an alarm feedback unit; The rate evaluation value unit is used to calculate the message processing speed evaluation value G1, G1=p0 / P, where p0 represents the processing speed of the messages in the current message queue; The message volume evaluation unit is used to calculate the evaluation value G2 of the message volume in the message queue, G2=L / l0, where l0 represents the number of messages remaining in the current message queue; The failure rate management unit is used to obtain the failure rate of the current message processing from the failure rate management unit, and the failure rate is recorded as φ; The safety factor calculation unit is used to calculate the safety factor W of the current message processing, W=α×G1+β×G2+γ×(1-φ), where the conditions are met, α+β+γ=1, α>β, α>γ; The alarm feedback unit is used to compare the safety factor W with the alarm threshold ω, and when W<ω, an alarm prompt is given to relevant staff, where 0<ω<1.

[0017] In a high-speed message processing system, when the message processing speed is lower than the speed threshold, G1 < 1, and when the remaining messages in the message queue are greater than the threshold, G2 < 1. Therefore, when the message queue processing efficiency is lower, G1 and G2 are smaller. At the same time, when the success rate of message processing is lower, the system operation status is poor. Therefore, the closer the W value is to 0, the greater the operating risk of the system.

[0018] In order to better apply the above system, a message queue distribution management method based on data monitoring is also proposed, and the method includes: Step S1: Obtain the message processing status of the message queue of the message queue; Step S2: Compare the message processing status with the message processing mode rules. When at least one indicator meets the threshold judgment rule, the high-speed processing mode is adopted, otherwise the normal processing mode is entered; Step S3: Establish a database to cache unprocessed messages, where the unprocessed messages include messages that failed to be processed and messages cached in high-speed mode; Step S4: when the message queue is in the normal processing mode, judging whether to resend the messages in the database individually or in batches according to the failure rate of message processing in the second unit time period; Step S5: monitor the number of message processing failures, and when the number of message processing failures exceeds a threshold, request manual intervention.

[0019] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention monitors asynchronous message push by monitoring data services, realizes rapid release of communication with the mq message queue, and enhances the message consumption rate; 2. The present invention monitors the MQ service in real time through distribution status monitoring, and performs service downgrade through comprehensive indicators. After downgrade due to abnormal data monitoring service consumption or abnormal MQ service, the data is saved to the database to ensure data security. At the same time, the distribution status monitoring serves as a compensation service to push the unsent data back to the message queue for the data monitoring service to monitor and push; 3. The present invention uses a mathematical model to adjust the processing strategy in real time, and supervises the high-throughput message sending system through a three-level compensation mechanism of automatic retry, timed retry and manual intervention, ensuring the final consistency and high reliability of the message while maintaining high throughput. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a structural diagram of a message queue distribution management system based on data monitoring according to the present invention; Figure 2 A flow chart of a message queue distribution management method based on data monitoring according to the present invention; Figure 3 The present invention is a schematic diagram of an embodiment of a message queue distribution management system based on data monitoring. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] Example: Figure 1-Figure 3 As shown, the present invention provides a technical solution, a message queue distribution management system and method based on data monitoring; The system includes: a message queue monitoring module, a message processing mode management module, a message compensation mode management module, a single compensation module, a batch compensation module, a failure alarm module and a message queue security monitoring module; The message queue monitoring module is used to collect messages sent by several different senders into the message queue and monitor the message processing status of the message queue.

[0023] The message queue monitoring module includes: a backlog management unit, a message processing rate management unit and a failure rate management unit; The backlog management unit is used to obtain the backlog of messages in the message queue in a time period between two moments; The message processing rate management unit is used to manage the rate change of the message processing rate between any two moments; The failure rate management unit is used to calculate the ratio of messages that fail to be sent in the first unit time period to the total number of messages in the first unit time period.

[0024] In the embodiment, the backlog is detected by detecting the total amount of messages in the message queue at two times, and the difference between the two total amounts of messages is the message backlog of the message queue; For example, at the first sampling time, the total number of messages in the message queue is 90. The second sampling is performed 5 seconds later. At the second sampling time, the total number of messages in the message queue is 120. Then the message backlog is 120-90=30. In the embodiment, when obtaining the sampling time, the instantaneous message processing rate of the message queue is obtained, and the change of the message processing rate between two consecutive sampling times is obtained to obtain the rate change amount; For example, at the first sampling time, the message queue processes 50 messages per second. Five seconds later, the second sampling is performed. At the second sampling time, the message queue processes 60 messages per second. The rate change = (60-50) / 5 = 2 messages per second. 2 .

[0025] The message processing mode management module is used to collect the message processing status of the message queue and adjust the processing mode of the message queue.

[0026] The message processing mode management module includes: a message total amount threshold management unit, a recommended speed threshold management unit, a threshold management unit and a rule management unit; The message total amount threshold management unit calculates the message total amount threshold by collecting the performance parameters of the computer that processes the message queue and the average data size of the messages in the message queue, wherein the performance parameters of the computer include the computer memory, and the calculation of the message total amount threshold L includes, L=(R×k1) / q0, R represents the computer memory, k1 represents the first evaluation coefficient of the computer, and satisfies the condition 0<k1<1, q0 represents the average data size of the messages in the message queue; Preferably, in the embodiment, the value of k1 is 0.8; The recommended speed threshold management unit calculates the recommended speed threshold by collecting the performance parameters of the computer, wherein the performance parameters of the computer include the number of cores of the processor and the average speed of each core processing the number of messages per unit time, wherein the calculation of the recommended speed threshold P includes, P=code×k2×csp, code represents the number of cores of the processor, k2 represents the second evaluation coefficient of the computer, k2 satisfies the condition that it is an integer greater than 1, and csp represents the average speed of each core processing the number of messages per unit time; Preferably, in the embodiment, the value of k2 is 100; The threshold management unit is used to manage the judgment threshold of the message processing mode, and the judgment threshold includes the total message threshold, the recommended speed threshold, the backlog threshold, the rate change threshold and the processing failure rate threshold; The rule management unit is used to manage the judgment rules of the message processing mode, and adjust the processing mode of the message queue to the normal processing mode and the high-speed processing mode. The normal processing mode is to push the newly received messages into the message queue in sequence, and the high-speed processing model is to store the newly received messages into the database first. When at least one indicator meets the threshold judgment rule, the high-speed processing mode is adopted.

[0027] The judgment rules include whether the number of messages in the message queue is greater than the total message threshold, whether the backlog of the message queue is greater than the backlog threshold, whether the rate change is less than the rate change threshold, and whether the failure rate of message processing is greater than the processing failure rate threshold.

[0028] In the embodiment, the first unit time period is based on the current time. In the past 5 hours, the total number of message processing is 500, of which 20 messages failed to be processed, and the failure rate of message processing failure is 20 / 500=4%; Set the total message threshold in the message queue to 150, the backlog threshold to 50, and the rate change threshold to 1.5 messages / second 2 , the processing failure rate threshold is 5%; If there are 120 messages in the current message queue, 120 < 150, the backlog is 30 < 50, and the rate change is 2 messages / second 2 >1.5 pieces / second 2 , the failure rate is 4%<5%. Since all indicators have reached the threshold of the rules, the normal mode is used to process the message queue.

[0029] The message compensation mode management module is used to obtain the failure rate of message processing failures, issue a single compensation instruction when the failure rate is less than a threshold, and issue a batch compensation instruction when the failure rate is greater than the threshold.

[0030] In the embodiment, the method for obtaining the value of the second unit time period obtains the message that has been stored in the database the longest, takes the generation time of this message as the starting time of the second unit time, and records the time period between the current time and the starting time as the second unit time period.

[0031] The message compensation mode management module includes: a failure message management unit, a compensation mode management unit and a compensation instruction management unit; The failed message management unit is used to store the messages that the push end of the message queue cannot push correctly into the database; The compensation mode management unit is used to manage the failure rate of message processing failures in the second unit time period, the failure rate includes the number of messages stored in the database in the second time period to the total number of messages received in the second time period, the total number of received messages includes the number of messages received by the database and the number of messages received by the message queue; The compensation instruction management unit is used to push instructions for storing messages in a database to the message queue when the processing mode of the message queue is the normal processing mode. When the failure rate of message processing failures in the second unit time period is less than a threshold, a single compensation instruction is issued; and when the failure rate of message processing failures in the second unit time period is greater than a threshold, a batch compensation instruction is issued.

[0032] The single compensation module performs a single retransmission on the message that fails to be sent, and the single compensation module includes a failure number management unit and a sending interval management unit.

[0033] The failure count management unit is used to obtain the number of failures in processing any message. The sending interval management unit is used to obtain the basic time interval and number of processing failures of push messages, and calculate the sending interval of the same message, where the sending interval is Tp, Tp=t0×2 n , where t0 represents the basic time interval and n represents the number of failures of the same message.

[0034] In the embodiments, the situations where message processing fails include: due to network fluctuations, the message in the message queue fails to be successfully pushed to the message receiving end, the user at the message receiving end is disconnected and fails to successfully receive the message; In the embodiment, the basic interval is set to 5 minutes, and a total of 10 messages that failed to be pushed for the first time are collected, and the 10 messages are pushed to the message processing queue after 5 minutes; After the second push, 8 messages were successfully processed and 2 messages were not successfully processed. The number of failures of these two messages is increased by 1, and the number of failures is recorded as 2; Calculate the push interval Tp2 of the second push, Tp2=5×2 2 = 20 minutes, and push these two messages again after 20 minutes.

[0035] The batch compensation module batch resends the messages that failed to be sent. The batch compensation module includes a message priority calculation unit and a compensation queue management unit.

[0036] The message priority calculation unit is used to obtain the number of message processing failures, the importance coefficient of the message and the network delay between the message receiver and the message queue, and calculate the priority coefficient of each message, where the priority coefficient is f, f=(n×imp) / t2, where imp represents the importance coefficient of the message, and t2 represents the network delay between the message receiver and the message queue. For different senders, corresponding importance coefficients are preset, the sender of the message is obtained, and the sender's importance coefficient is used as the importance coefficient of the message sent by the sender; The compensation queue management unit is used to obtain the rate at which the computer processes push messages and the compensation frequency to calculate the number of single push messages, where the number of single push messages is E, E=TP / fr, where TP represents the rate at which the computer processes push messages, and fr represents the compensation frequency.

[0037] In the embodiment, for example, the number of push messages that can be processed by the computer is 1000 per minute, and the compensation frequency is 10 times per hour, so E = 1000 per minute ÷ 10 times per hour = 6000 per time, that is, within one hour, a push queue of 6000 messages is created every 6 minutes; Get the priority coefficient of each message in the database, and fill the message into the push queue according to the priority coefficient from large to small.

[0038] The failure alarm module counts the number of failed processing of messages that failed to be sent. When the number of failed processing of any message exceeds the threshold, the relevant management personnel are reminded to check.

[0039] The message queue security monitoring module calculates the security factor of the message queue and issues an alarm when the security factor is lower than the threshold.

[0040] The message queue security monitoring module includes: a rate evaluation value unit, a message volume evaluation unit, a failure rate management unit, a safety factor calculation unit and an alarm feedback unit; The rate evaluation value unit is used to calculate the message processing speed evaluation value G1, G1=p0 / P, where p0 represents the processing speed of the messages in the current message queue; The message volume evaluation unit is used to calculate the evaluation value G2 of the message volume in the message queue, G2=L / l0, where l0 represents the number of messages remaining in the current message queue; The failure rate management unit is used to obtain the failure rate of the current message processing from the failure rate management unit, and the failure rate is recorded as φ; The safety factor calculation unit is used to calculate the safety factor W of the current message processing, W=α×G1+β×G2+γ×(1-φ), where the conditions are met, α+β+γ=1, α>β, α>γ; The alarm feedback unit is used to compare the safety factor W with the alarm threshold ω, and when W<ω, an alarm prompt is given to relevant staff, where 0<ω<1.

[0041] Preferably, in the embodiment, the value of α is 0.4, the value of β is 0.3, and the value of γ is 0.3.

[0042] Methods include: Step S1: Obtain the message processing status of the message queue of the message queue; Step S2: Compare the message processing status with the message processing mode rules. When at least one indicator meets the threshold judgment rule, the high-speed processing mode is adopted, otherwise the normal processing mode is entered; Step S3: Establish a database to cache unprocessed messages, where the unprocessed messages include messages that failed to be processed and messages cached in high-speed mode; Step S4: when the message queue is in the normal processing mode, judging whether to resend the messages in the database individually or in batches according to the failure rate of message processing in the second unit time period; Step S5: monitor the number of message processing failures, and when the number of message processing failures exceeds a threshold, request manual intervention.

[0043] The implementation process of another embodiment of the present invention includes: 1. Send the data to be distributed to the MQ message queue through the source data service.

[0044] 2. The data monitoring cluster service monitors the data. The monitoring service needs to be configured with the same MQ address as the source data service. The data monitoring service itself is one of the consumer groups. After monitoring the data, it queries the data push rules and target service: 2.1 First, check the rules to see if the current message meets the push conditions. When the check passes, the listening service will immediately send an asynchronous call to call the target service to ensure that the message is quickly released and communicated with MQ.

[0045] 2.2 If the target service is pushed and the target service fails to process, the data monitoring service will save the unprocessed message to the database to record a pending message.

[0046] 2.3 If the message does not pass the rule match, it will not be distributed and will be discarded.

[0047] 3. As part of the monitoring service, the distribution status monitoring will monitor the consumption of the mq message queue in real time. When the data monitoring cluster service monitors the data from the source service, it will first use the distribution status monitoring to determine whether the warning value is reached. The distribution status monitoring will determine whether the MQ memory, consumption failure rate and consumption delay meet the user's preset rules.

[0048] 3.1 According to the monitoring indicators, it is judged whether the comprehensive indicators have reached the predetermined indicators. The total indicators provided by this system include the total number of messages, consumption delay, failure rate and other indicators. If the predetermined indicators are reached, the distribution monitoring service will immediately start the high-speed consumption mode: that is, immediately consume the messages in the mq message queue, save the messages to the database, record N pending messages, and ensure that the mq service will not crash due to a large number of messages.

[0049] 3.2 If the mq message queue indicator monitored by the monitoring service does not reach the predetermined indicator, the high-speed consumption mode will not be enabled.

[0050] 3.3 Regularly poll the messages to be pushed and push them back to the mq message queue, and then enter the distribution logic.

[0051] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A message queue distribution management system based on data monitoring, characterized in that: The system includes: Message queue monitoring module, message processing mode management module, message compensation mode management module, single compensation module, batch compensation module, failure alarm module and message queue security monitoring module; The message queue monitoring module is used to collect messages sent by several different senders into the message queue and monitor the message processing status of the message queue; The message processing mode management module is used to collect the message processing status of the message queue and adjust the processing mode of the message queue; The message compensation mode management module is used to obtain the failure rate of message processing failures, issue a single compensation instruction when the failure rate is less than a threshold, and issue a batch compensation instruction when the failure rate is greater than the threshold; The single compensation module performs a single retransmission on the message that failed to be sent; The batch compensation module batch resends the messages that failed to be sent; The failure alarm module counts the number of failed processing of messages that failed to be sent, and when the number of failed processing of any message exceeds a threshold, it reminds relevant management personnel to check; The message queue security monitoring module calculates the security factor of the message queue and issues an alarm when the security factor is lower than a threshold.

2. According to the data monitoring-based message queue distribution management system of claim 1, it is characterized by: The message queue monitoring module includes: a backlog management unit, a message processing rate management unit and a failure rate management unit; The backlog management unit is used to obtain the backlog of messages in the message queue in a time period between two moments; The message processing rate management unit is used to manage the rate change of the message processing rate at any two moments; The failure rate management unit is used to calculate the ratio of messages that fail to be sent in the first unit time period to the total number of messages in the first unit time period.

3. The message queue distribution management system based on data monitoring according to claim 2 is characterized in that: The message processing mode management module includes: a message total amount threshold management unit, a recommended speed threshold management unit, a threshold management unit and a rule management unit; The message total amount threshold management unit calculates the message total amount threshold by collecting the performance parameters of the computer that processes the message queue and the average data size of the messages in the message queue, wherein the performance parameters of the computer include the computer memory, and the calculation of the message total amount threshold L includes, L=(R×k1) / q0, R represents the computer memory, k1 represents the first evaluation coefficient of the computer, and satisfies the condition 0<k1<1, q0 represents the average data size of the messages in the message queue; The recommended speed threshold management unit calculates the recommended speed threshold by collecting the performance parameters of the computer, wherein the performance parameters of the computer include the number of cores of the processor and the average speed of each core processing the number of messages per unit time, wherein the calculation of the recommended speed threshold P includes, P=code×k2×csp, code represents the number of cores of the processor, k2 represents the second evaluation coefficient of the computer, k2 satisfies the condition that it is an integer greater than 1, and csp represents the average speed of each core processing the number of messages per unit time; The threshold management unit is used to manage the judgment threshold of the message processing mode, and the judgment threshold includes the total message threshold, the recommended speed threshold, the backlog threshold, the rate change threshold and the processing failure rate threshold; The rule management unit is used to manage the judgment rules of the message processing mode, and adjust the processing mode of the message queue to the normal processing mode and the high-speed processing mode. The normal processing mode is to push the newly received messages into the message queue in sequence, and the high-speed processing model is to store the newly received messages into the database first. When at least one indicator meets the threshold judgment rule, the high-speed processing mode is adopted.

4. The message queue distribution management system based on data monitoring according to claim 3 is characterized in that: The judgment rules include whether the number of messages in the message queue is greater than the total message threshold, whether the backlog of the message queue is greater than the backlog threshold, whether the rate change is less than the rate change threshold, and whether the failure rate of message processing is greater than the processing failure rate threshold.

5. The message queue distribution management system based on data monitoring according to claim 3 is characterized in that: The message compensation mode management module includes: a failure message management unit, a compensation mode management unit and a compensation instruction management unit; The failed message management unit is used to store the messages that the push end of the message queue cannot push correctly into the database; The compensation mode management unit is used to manage the failure rate of message processing failures in the second unit time period, wherein the failure rate includes the number of messages stored in the database in the second time period to the total number of messages received in the second time period, wherein the total number of received messages includes the number of messages received by the database and the number of messages received by the message queue; The compensation instruction management unit is used to push instructions for storing messages in a database to the message queue when the processing mode of the message queue is the normal processing mode. When the failure rate of message processing failures in the second unit time period is less than a threshold, a single compensation instruction is issued; and when the failure rate of message processing failures in the second unit time period is greater than a threshold, a batch compensation instruction is issued.

6. A message queue distribution management system based on data monitoring according to claim 5, characterized in that: The single compensation module includes a failure number management unit and a sending interval management unit; The failure count management unit is used to obtain the number of failures in processing any message. The sending interval management unit is used to obtain the basic time interval and the number of processing failures of the push message, and calculate the sending interval of the same message, where the sending interval is Tp, Tp=t0×2n, where t0 represents the basic time interval and n represents the number of failures of the same message.

7. The message queue distribution management system based on data monitoring according to claim 6 is characterized in that: The batch compensation module includes a message priority calculation unit and a compensation queue management unit; The message priority calculation unit is used to obtain the number of message processing failures, the importance coefficient of the message and the network delay between the message receiver and the message queue, and calculate the priority coefficient of each message, where the priority coefficient is f, f=(n×imp) / t2, where imp represents the importance coefficient of the message, and t2 represents the network delay between the message receiver and the message queue. For different senders, corresponding importance coefficients are preset, the sender of the message is obtained, and the sender's importance coefficient is used as the importance coefficient of the message sent by the sender; The compensation queue management unit is used to obtain the rate at which the computer processes push messages and the compensation frequency to calculate the number of single push messages, where the number of single push messages is E, E=TP / fr, where TP represents the rate at which the computer processes push messages, and fr represents the compensation frequency.

8. The message queue distribution management system based on data monitoring according to claim 7 is characterized in that: The message queue security monitoring module includes: a rate evaluation value unit, a message volume evaluation unit, a failure rate management unit, a safety factor calculation unit and an alarm feedback unit; The rate evaluation value unit is used to calculate the message processing speed evaluation value G1, G1=p0 / P, where p0 represents the processing speed of the messages in the current message queue; The message volume evaluation unit is used to calculate the evaluation value G2 of the message volume in the message queue, G2=L / l0, where l0 represents the number of messages remaining in the current message queue; The failure rate management unit is used to obtain the failure rate of the current message processing from the failure rate management unit, and record the failure rate as φ; The safety factor calculation unit is used to calculate the safety factor W of the current message processing, W=α×G1+β×G2+γ×(1-φ), where the conditions are met, α+β+γ=1, α>β, α>γ; The alarm feedback unit is used to compare the safety factor W with the alarm threshold ω, and when W<ω, an alarm prompt is given to relevant staff, where 0<ω<1.

9. A message queue distribution management method based on data monitoring, used in a message queue distribution management system based on data monitoring as claimed in any one of claims 1 to 8, characterized in that: The method comprises the steps of: Step S1: Obtain the message processing status of the message queue of the message queue; Step S2: Compare the message processing status with the message processing mode rules. When at least one indicator meets the threshold judgment rule, the high-speed processing mode is adopted, otherwise the normal processing mode is entered; Step S3: Establish a database to cache unprocessed messages, where the unprocessed messages include messages that failed to be processed and messages cached in high-speed mode; Step S4: when the message queue is in the normal processing mode, judging whether to resend the messages in the database individually or in batches according to the failure rate of message processing in the second unit time period; Step S5: monitor the number of message processing failures, and when the number of message processing failures exceeds a threshold, request manual intervention.

Images