Queue system message accumulation processing method and device and electronic equipment

Abstract

The invention discloses a queue system message accumulation processing method and device and electronic equipment, and relates to the technical field of distribution.The method comprises the steps that operation data of a queue system are collected based on a preset frequency adjusting mechanism, and a first index, a second index and a third index of the queue system are determined based on the operation data, the first index is a depth deviation degree of a message queue determined through logarithm compression operation, the second index is consumer saturation of the queue system, the third index is a message growth rate of the queue system calculated based on a dual time window, and an index entropy value of the queue system is determined based on the first index, the second index and the third index. And performing decision processing on the queue system based on the index entropy. According to the method and the device, the technical problem of high false alarm rate of early warning for message accumulation in the prior art in the process of monitoring the queue system is solved.

Description

Technical Field

[0001] This application relates to the field of distributed technology, and more specifically, to a method, apparatus, and electronic device for processing message backlog in a queue system. Background Technology

[0002] In distributed systems, the queue mechanism, as the core infrastructure for asynchronous communication, can decouple strong dependencies between services, handle traffic peak shaping, and stagger traffic flow. This mechanism has become one of the key technical pillars in high-concurrency scenarios. Its application scenarios not only cover real-time data processing, event-driven architecture, and distributed transaction coordination, but also play an increasingly important role in ensuring system elasticity, improving fault tolerance, and optimizing resource utilization.

[0003] Message backlog in a queue system is broadly termed "Q backlog," encompassing three typical forms: deep backlog in message queues, buffer blocking in Send-Q / Recv-Q (Send Queue / Receive Queue) networks, and backlog in Java thread pool task queues (i.e., the program's own memory queue). In distributed systems, transient queue backlog under normal conditions causes brief system delays but does not affect the continuity of system services. However, "Q backlog" under abnormal conditions, such as a queue consumer seemingly freezing, poses a risk of progressively damaging system services, ranging from service degradation to complete outage, leading to a loss of system controllability.

[0004] In existing technologies, traditional monitoring solutions use a "Q-accumulation" alarm mechanism based on a single static threshold. For example, when the MQ depth exceeds a preset threshold, an alarm is triggered. This monitoring method cannot distinguish the essential nature of queue backlog, resulting in a high false alarm rate.

[0005] For example, when a sudden surge in traffic causes a surge in messages and triggers an alarm, but the consumer processing capacity (Transactions Per Second, TPS) has not yet reached its bottleneck, the system is fully capable of self-regulation and can return to normal without the intervention of operations and maintenance personnel. In this scenario, traditional monitoring solutions will still trigger alarms, causing operations and maintenance personnel to frequently make ineffective responses.

[0006] There is currently no effective solution to the above problems. Summary of the Invention

[0007] This application provides a method, apparatus, and electronic device for processing message accumulation in a queue system, in order to at least solve the technical problem of high false alarm rate in the prior art for message accumulation warning during the monitoring of a queue system.

[0008] According to one aspect of this application, a method for handling message backlog in a queue system is provided, comprising: collecting operational data of the queue system based on a preset frequency adjustment mechanism, wherein the preset frequency adjustment mechanism is used to update the data collection cycle of the queue system's system status; determining a first indicator, a second indicator, and a third indicator of the queue system based on the operational data, wherein the first indicator is the depth deviation of the message queue determined by logarithmic compression operation, the second indicator is the consumer saturation of the queue system, and the third indicator is the message growth rate of the queue system calculated based on a dual time window; determining the indicator entropy value of the queue system based on the first indicator, the second indicator, and the third indicator, wherein the indicator entropy value is used to characterize the probability of message backlog risk in the queue system; and performing decision processing on the queue system based on the indicator entropy value.

[0009] Optionally, the steps for collecting operational data of the queue system include: collecting message queue data through a preset component, wherein the message queue data includes at least the queue depth, producer input rate, and consumer acknowledgment rate of the message queue in the queue system, and the preset component is deployed on the host where the message queue is located; collecting buffer data through a preset program, wherein the buffer data is used to characterize at least the depth change information of the receive queue / send queue included in the buffer of the queue system and the packet transmission delay information, and the preset program is used to run directly in the kernel of the queue system; collecting thread pool data through a preset tool, wherein the thread pool data includes at least the queue depth, number of active threads, and task submission frequency of the task queue in the queue system, and the preset tool is used to insert data collection code in the form of bytecode injection; collecting consumer process data through a preset plugin, wherein the consumer process data includes at least the processor utilization, memory usage, process status, and process processing speed corresponding to the consumer process in the task queue; and using the message queue data, buffer data, thread pool data, and consumer process data as the operational data of the queue system.

[0010] Optionally, the step of determining the first indicator of the queue system based on operational data includes: obtaining the baseline queue depth value corresponding to the data collection period of the operational data, wherein the baseline queue depth value is the queue depth boundary value determined based on P groups of historical operational data of the queue system, and P is a positive integer; determining the current queue depth value of the queue system based on the operational data; calculating the ratio of the current queue depth value to the baseline queue depth value; and performing logarithmic compression on the comparison value based on a preset logarithmic compression algorithm to obtain the first indicator.

[0011] Optionally, the step of determining the second indicator of the queue system based on operational data includes: obtaining the historical message processing efficiency corresponding to the data collection period of the operational data, wherein the historical message processing efficiency is the maximum message processing efficiency of the queue system within a preset historical time period; determining the current message processing efficiency of the queue system based on the operational data, wherein the current message processing efficiency is used to characterize the message processing capability of the queue system's consumers during the current data collection period; and determining the second indicator based on the ratio of the current message processing efficiency to the historical message processing efficiency.

[0012] Optionally, the step of determining the third indicator of the queue system based on operational data includes: determining the queue depth change sequence of the queue system based on operational data, wherein the queue depth change sequence is used to characterize the change information of the total length of all messages included in the message queue; and performing linear regression fitting on the depth change sequence based on a short window and a long window in a dual time window to obtain the third indicator, wherein the short window is used to determine the instantaneous growth rate of queue messages in the queue system, and the long window is used to determine the changing trend of queue messages in the queue system.

[0013] Optionally, before determining the first indicator of the queue system based on the operational data, the method for handling message backlog in the queue system further includes: performing time-series decomposition on P groups of historical operational data to obtain seasonal terms, trend terms, and residual terms corresponding to the P groups of historical operational data. The seasonal term characterizes the periodic variation pattern of the message queue depth in the queue system, the trend term characterizes the periodic variation trend of the message queue depth in the queue system, and the residual term characterizes the random disturbance variation amplitude of the message queue depth in the queue system. Based on the seasonal term and trend term, the first depth value corresponding to the data collection period of the operational data is determined. Based on the residual term and the first depth value, the baseline queue depth value corresponding to the data collection period of the operational data is determined.

[0014] Optionally, the steps for making decisions about the queue system based on the indicator entropy value include: determining that the queue system is in a first system state when the indicator entropy value is greater than a first preset entropy value; transmitting the alarm information corresponding to the first system state to the operation and maintenance personnel and triggering the circuit breaker mechanism, wherein the circuit breaker mechanism is used to directly perform system service isolation / traffic degradation operations based on preset policies.

[0015] Optionally, the steps for making decisions about the queue system based on the index entropy value include: determining that the queue system is in a second system state when the index entropy value is less than or equal to a first preset entropy value and greater than a second preset entropy value, wherein the second preset threshold is less than the first preset threshold; transmitting the alarm information corresponding to the second system state to the operation and maintenance personnel of the queue system, and performing correlation analysis on the queue system, wherein the correlation analysis includes at least one of the following: a first correlation analysis, used to analyze the correlation between the backlog status of the thread pool and the pause time of the garbage collection mechanism; a second correlation analysis, used to analyze the correlation between the backlog status of the sending queue in the buffer and the bandwidth utilization rate; a third correlation analysis, used to analyze the phase matching relationship between the depth growth curve of the message queue and the processing capacity curve of the consumer; performing fault location based on the analysis results corresponding to the correlation analysis, and obtaining a fault diagnosis report; creating a work order based on the fault diagnosis report, wherein the work order is used to prompt the operation and maintenance personnel to repair the queue system fault.

[0016] Optionally, the steps for making decisions about the queue system based on the index entropy value include: determining that the queue system is in a third system state when the index entropy value is less than or equal to a second preset entropy value; archiving the operating data when the queue system is in the third system state; updating the baseline queue depth value of the queue system based on the archived operating data, and generating a system state assessment report corresponding to the queue system.

[0017] According to another aspect of this application, a processing apparatus for message backlog in a queue system is also provided, comprising: a data acquisition unit, configured to acquire operational data of the queue system based on a preset frequency adjustment mechanism, wherein the preset frequency adjustment mechanism is used to update the data acquisition cycle based on the system status of the queue system; an indicator determination unit, configured to determine a first indicator, a second indicator, and a third indicator of the queue system based on the operational data, wherein the first indicator is the depth deviation of the message queue determined by logarithmic compression operation, the second indicator is the consumer saturation of the queue system, and the third indicator is the message growth rate of the queue system calculated based on a dual time window; an entropy value determination unit, configured to determine the indicator entropy value of the queue system based on the first indicator, the second indicator, and the third indicator, wherein the indicator entropy value is used to characterize the probability of message backlog risk occurring in the queue system; and a decision-making unit, configured to perform decision processing on the queue system based on the indicator entropy value.

[0018] According to another aspect of this application, a computer program product is also provided, which stores a computer program, wherein, when the computer program is running, the computer program product is controlled to execute the queue system message accumulation processing method of any of the above.

[0019] According to another aspect of this application, an electronic device is also provided, wherein the electronic device includes one or more processors and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the message accumulation processing method of any of the above-described queue system.

[0020] In this application, the operation data of the queue system is first collected based on a preset frequency adjustment mechanism. The preset frequency adjustment mechanism is used to update the data collection cycle of the queue system's system status. Then, based on the operation data, this application determines a first indicator, a second indicator, and a third indicator of the queue system. The first indicator is the depth deviation of the message queue determined by logarithmic compression operation, the second indicator is the consumer saturation of the queue system, and the third indicator is the message growth rate of the queue system calculated based on a dual time window. Then, based on the first, second, and third indicators, this application determines the indicator entropy value of the queue system. The indicator entropy value is used to characterize the probability of message backlog risk in the queue system. Finally, this application performs decision processing on the queue system based on the indicator entropy value.

[0021] As described above, this application first uses an adaptive preset frequency adjustment mechanism to dynamically adjust the frequency of queue operation data collection based on the real-time system status of the queue system, thereby balancing the timeliness of the collected operation data with the utilization rate of data collection resources. Then, this application deeply analyzes the nature and extent of message backlog in the message queue, obtaining three-dimensional key performance indicators: message queue depth deviation, consumer saturation, and message growth rate. Finally, this application calculates the weighted entropy value of the above three-dimensional key performance indicators to obtain an entropy value that accurately characterizes the probability of message backlog risk. This provides a quantitative basis for intelligent decision-making in the queue system while achieving the technical effect of improving the accuracy of message backlog risk warning in the queue system.

[0022] Therefore, this application adopts a multi-dimensional and dynamic evaluation method of the queue system's operating data. Through an adaptive preset frequency adjustment mechanism and entropy calculation, it achieves the goal of accurately assessing the risk of message accumulation in the queue system, thereby reducing the false alarm rate of message accumulation warning. This solves the technical problem of high false alarm rate in existing technologies for message accumulation warning during queue system monitoring. Attached Figure Description

[0023] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1This is a flowchart of an optional message backlog processing method for a queue system according to an embodiment of this application; Figure 2 This is a schematic diagram of an optional three-dimensional decision engine according to an embodiment of this application; Figure 3 This is an architecture diagram of an optional queue system message backlog processing system according to an embodiment of this application; Figure 4 This is a flowchart of an optional queue system message backlog processing system according to an embodiment of this application; Figure 5 This is a schematic diagram of an optional message backlog processing apparatus for a queue system according to an embodiment of this application; Figure 6 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] It should also be noted that all relevant information (including but not limited to information related to user flight segments) and data (including but not limited to operational data of the queuing system used for collection) involved in this application are information and data authorized by the user or fully authorized by all parties. For example, if there is an interface between this system and the relevant user or organization, before obtaining the relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain the relevant information after receiving the consent information from the aforementioned user or organization.

[0027] Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of relevant information and data involved in this application all comply with the relevant laws, regulations, and standards of the relevant regions, and necessary confidentiality measures have been taken. This application does not violate public order and good morals. In addition, this application provides a corresponding operation entry point for users to choose to agree to or refuse authorization. If the user chooses to refuse authorization, the corresponding expert decision-making process will be initiated.

[0028] In one alternative embodiment, a method for monitoring a queue system based on a "Q-accumulation" alarm mechanism with a single static threshold is provided. However, since this monitoring method cannot distinguish the essential nature of queue backlog, this embodiment also suffers from the technical problem of low efficiency in locating queue faults.

[0029] Optionally, as a critical resource hub, the anomalies of the queue system are often triggered by chain failures in upstream and downstream systems. For example, a backlog of tasks in the thread pool can lead to database connection exhaustion, which in turn can cause the MQ consumer to stop, ultimately creating a cascading failure. Because each queue type is monitored by independent monitoring tools, the overall alarm system is fragmented and lacks cross-resource correlation analysis capabilities, resulting in technical problems such as long average fault location time and low fault location efficiency in the queue system.

[0030] This application proposes the first unified and accurate alarm framework for generalized "product Q", replacing the static threshold early warning system with a dynamic multi-dimensional evaluation model. The core technological breakthrough of this application lies in: (1) Unified abstraction layer: Define the core monitoring tuples (queue depth, consumer saturation, time-series trend) across queue types.

[0031] (2) Dynamic decision engine: Based on time series analysis, identify the nature of backlog (instantaneous surge / continuous deterioration) and conduct comprehensive analysis in combination with resource saturation.

[0032] The present invention will now be described in detail with reference to various embodiments.

[0033] Example 1 According to an embodiment of this application, an embodiment of a method for processing message backlog in a queue system is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0034] This application provides a message backlog processing system (hereinafter referred to as the processing system) for executing the message backlog processing method of the queue system in this application. Figure 1This is a flowchart of an optional message backlog processing method for a queue system according to an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps: Step S101: Based on a preset frequency adjustment mechanism, collect the operation data of the queue system, wherein the preset frequency adjustment mechanism is used to update the data collection cycle of the system status of the queue system.

[0035] Optionally, during the indicator collection phase, the implementation of the preset frequency adjustment mechanism relies on the perception of the computing layer and the decision-making layer, and the preset frequency adjustment module continuously monitors the changes in entropy and the fluctuation range of the indicators.

[0036] For example, during non-transactional periods (such as 2-4 AM), when the system's message queue is in a stable phase, the state awareness module detects a risk score (entropy value) ≤ 0.7 and no abnormal rules set by the decision-making layer are triggered, thus triggering a low-frequency collection once per minute. When the risk score (entropy value) is between 0.7 and 0.85, the state awareness module immediately increases the collection frequency to once every 10 seconds, at which point the processing system will frequently capture changes in partition offsets. When the risk score (entropy value) > 0.85 or an abnormal rule set by the decision-making layer is triggered, the system automatically switches to high-frequency monitoring once per second, directly connecting to the queue node through a customized collection script to obtain the number of accumulated messages in real time. At the same time, it uses a caching mechanism (such as Redis (REmote DIctionary Server, an open-source key-value store database)) to temporarily store instantaneous data, avoiding the impact of high-frequency collection on the database.

[0037] Optionally, the preset frequency adjustment mechanism refers to a mechanism that can dynamically adjust the data acquisition frequency according to the current system state of the queue system (such as the first system state, the second system state, and the third system state).

[0038] In step S101, the processing system can effectively balance the real-time nature of monitoring data and the efficiency of system resource utilization by dynamically adjusting the data acquisition frequency. Reducing the acquisition frequency during low load periods or when the system is running smoothly helps to reduce the impact of the monitoring system on the production environment, while increasing the acquisition frequency during high load periods or when the system is running abnormally ensures that enough information is obtained to quickly respond to and analyze problems, thereby achieving accurate monitoring of the queue system status.

[0039] Step S102: Determine the first, second, and third indicators of the queue system based on the operating data. The first indicator is the depth deviation of the message queue determined by logarithmic compression operation, the second indicator is the consumer saturation of the queue system, and the third indicator is the message growth rate of the queue system calculated based on the dual time window.

[0040] Optionally, when calculating the first indicator, the processing system performs a logarithmic compression operation to compare the current queue depth with a dynamic baseline generated based on historical data, and calculates the degree of deviation between the two in logarithmic form. This effectively handles depth fluctuations of queues of different sizes, avoids false alarms of instantaneous fluctuations in small queues, and prevents depth changes in large queues from being ignored, thereby improving the accuracy of the first indicator calculation.

[0041] Optionally, the second indicator reflects the ratio of the current processing capacity of consumers in the queuing system to the historical peak processing capacity. The higher the consumer saturation, the closer the consumer's processing capacity is to its limit, and the greater the risk of message backlog in the system.

[0042] Optionally, the third indicator is calculated based on a dual time window, which can help the processing system distinguish between two backlog modes: instantaneous surge in messages and continuous deterioration of messages. This provides a basis for the system status detection of the queue system and improves the efficiency of subsequent fault location of the queue system.

[0043] In step S102, the determination of the above three-dimensional indicators is obtained by the processing system based on the deep analysis of the queue system's operating data. By comprehensively considering the depth deviation, consumer saturation, and message growth rate, the processing system can more comprehensively and accurately identify the risk of message backlog in the queue system, thereby avoiding misjudgment and omission that may be caused by a single indicator, and realizing cross-scenario message backlog risk assessment.

[0044] Step S103: Determine the index entropy value of the queue system based on the first index, the second index, and the third index, wherein the index entropy value is used to characterize the probability of message backlog risk in the queue system.

[0045] Optionally, the processing system obtains the indicator entropy value by weighted summation of the first, second, and third indicators. The indicator entropy value integrates the uncertainties of three dimensions: depth deviation, consumer saturation, and message growth rate. This realizes the transformation from the traditional threshold alarm mechanism to a risk assessment mechanism based on information entropy. The early warning mechanism provided by this application can more accurately quantify backlog risks, distinguish between instantaneous anomalies and persistent faults, thereby providing more targeted warning and response strategies, avoiding false alarms and missed alarms, and thus improving the accuracy and efficiency of system fault detection.

[0046] Step S104: Make decisions about the queue system based on the index entropy value.

[0047] Optionally, the processing system determines the system state of the queue system based on the magnitude of the index entropy value. Then, it performs different decision processing operations based on the system state, thereby realizing the automation and intelligence of system decision-making. The dynamic evaluation based on the index entropy value enables rapid response and intelligent decision-making. This mechanism ensures that when backlog risks occur, the processing system can take the most appropriate action in a timely manner, reduce the mean time between failures, and improve the stability and availability of the queue system.

[0048] As described above, this application first uses an adaptive preset frequency adjustment mechanism to dynamically adjust the frequency of queue operation data collection based on the real-time system status of the queue system, thereby balancing the timeliness of the collected operation data with the utilization rate of data collection resources. Then, this application deeply analyzes the nature and extent of message backlog in the message queue, obtaining three-dimensional key performance indicators: message queue depth deviation, consumer saturation, and message growth rate. Finally, this application calculates the weighted entropy value of the above three-dimensional key performance indicators to obtain an entropy value that accurately characterizes the probability of message backlog risk. This provides a quantitative basis for intelligent decision-making in the queue system while achieving the technical effect of improving the accuracy of message backlog risk warning in the queue system.

[0049] Therefore, this application adopts a multi-dimensional and dynamic evaluation method of the queue system's operating data. Through an adaptive preset frequency adjustment mechanism and entropy calculation, it achieves the goal of accurately assessing the risk of message accumulation in the queue system, thereby reducing the false alarm rate of message accumulation warning. This solves the technical problem of high false alarm rate in existing technologies for message accumulation warning during queue system monitoring.

[0050] In one alternative embodiment, Figure 2 This is a schematic diagram of an optional three-dimensional decision engine according to an embodiment of this application. The processing system achieves accurate diagnosis of queue backlog in multiple scenarios by constructing a three-dimensional decision engine, a hierarchical system architecture, and an intelligent processing flow. The processing system takes an entropy calculation engine as its core and combines dynamic baseline analysis and resource coupling evaluation to form a complete closed-loop system from data acquisition to decision execution.

[0051] Optionally, such as Figure 2 As shown, the processing system connects three key performance indicators, namely queue depth deviation, consumer protection degree, and message growth rate, through an entropy calculation engine. Then, the data of each dimension are fused and calculated by the entropy calculation engine, and decision processing is carried out based on the calculated entropy value.

[0052] Specifically, the processing system breaks through the limitations of traditional static threshold alarms and establishes a three-dimensional decision engine. For the queue depth deviation dimension, it adopts a dynamic baseline comparison mechanism and logarithmic compression algorithm to effectively solve the problems of false alarms in small queues and missed alarms in large queues. In particular, the processing system ensures that normal fluctuations in a small range will not trigger false alarms by enabling logarithmic compression processing technology, and then accurately identifies abnormal states through deviation calculation.

[0053] For the consumer saturation dimension, the processing system introduces a historical peak reference system to calculate the processing capacity utilization rate in real time; for the message growth rate dimension, the processing system realizes growth pattern recognition through sliding window time series analysis. Its core innovation lies in the design of a dual threshold judgment mechanism, which calculates the instantaneous growth rate through a short window and tracks trend changes through a long window.

[0054] Finally, the entropy calculation engine performs weighted entropy calculation and fusion processing on the three-dimensional data, and outputs a risk score value in the range of 0-1, thereby realizing the intelligent transformation from simple Boolean judgment to the quantification of system imbalance and root cause analysis.

[0055] In one optional embodiment, for a heterogeneous environment of the queue system, during the data collection process, the processing system collects message queue data through a preset component. The message queue data includes at least the queue depth, producer input rate, and consumer acknowledgment rate of the message queues in the queue system. The preset component is deployed on the host where the message queues reside. The processing system collects buffer data through a preset program. The buffer data is used to characterize at least the depth changes of the receive / send queues and packet transmission delay information included in the buffers of the queue system. The preset program is used to run directly in the kernel of the queue system. The processing system collects thread pool data through a preset tool. The thread pool data includes at least the queue depth, number of active threads, and task submission frequency of the task queues in the queue system. The preset tool is used to insert data collection code through bytecode injection. The processing system collects consumer process data through a preset plugin. The consumer process data includes at least the processor utilization, memory usage, process status, and process processing speed corresponding to the consumer processes in the task queue.

[0056] Finally, the processing system uses the message queue data, buffer data, thread pool data, and consumer process data collected above as the running data of the queue system.

[0057] Optionally, the default component refers to the Exporter component of Prometheus (an open-source system monitoring and alerting tool), which can be deployed on the host where the message queue resides to capture the queue system's runtime data in real time.

[0058] Optionally, by deploying pre-defined components, the processing system can capture changes in the operational status of the message queue in a timely manner. This direct integration reduces additional system latency and improves the real-time performance and accuracy of data acquisition. In particular, for highly dynamic queue systems, it can ensure the timeliness and integrity of monitoring data, providing a high-quality data foundation for subsequent analysis and decision-making.

[0059] Optionally, the default program can be set to an eBPF (Extended Berkeley Packet Filter, a lightweight, secure, dynamically loadable program that can run in the Linux kernel) program. It can run directly in the Linux (a programming language) kernel without modifying the kernel code or loading additional modules, and can achieve real-time monitoring of information such as Send-Q / Recv-Q buffer depth changes and packet transmission delays in network communication.

[0060] Optionally, kernel-level data acquisition programs feature extremely low performance overhead and high-precision data capture, avoiding common performance bottlenecks and data latency issues in user-space data acquisition methods. This ensures that network communication data acquisition is not interfered with at the operating system level, providing the most direct and accurate data stream information. This feature is particularly important for high-concurrency network application scenarios, helping the system to promptly detect anomalies in network communication and prevent packet backlog or delays.

[0061] Optionally, the default tool can be set to Byte Buddy (a bytecode injection tool) to dynamically insert additional data collection code during the execution of the task queue processing logic code in the queue system, collecting information such as task queue depth, number of active threads, and task submission frequency, without recompiling or deploying the application.

[0062] Optionally, bytecode injection technology can be used to collect thread pool data. This allows for the collection of internal runtime status data of the thread pool, such as task queue length and thread activity, without affecting the application's operation. This method not only avoids the tedious process of modifying the application's source code but also ensures the comprehensiveness and real-time nature of the collected data. This tool is crucial for analyzing the task processing capabilities of the thread pool and optimizing resource allocation. Especially in distributed systems, it can help operations and maintenance personnel quickly identify the bottlenecks in the thread pool, make timely resource allocation or optimization strategy adjustments, and ensure stable system operation.

[0063] Optionally, the default plugin can be set to Telegraf (a data acquisition plugin) to collect runtime data related to consumer processes in the queue system, such as processor utilization, memory usage, process status, and processing speed. This data reflects the actual performance and health status of the consumer processes when processing queue messages.

[0064] Optionally, by collecting consumer process data through preset plugins, it is possible to gain a deeper understanding of the consumer process's operational status, including CPU utilization, memory consumption, and task processing efficiency. This is crucial for accurately identifying the causes of queue backlog, such as whether the consumer process is unable to process more messages due to resource exhaustion or whether the processing speed is significantly reduced due to some abnormal state. Based on this data, the system can monitor the health status of the consumer process and detect anomalies, and take timely measures to prevent the consumer process from becoming a bottleneck for queue backlog, ensuring the continuity and efficiency of message processing.

[0065] In summary, by integrating operational data from multiple sources, the processing system provides rich information for subsequent multi-dimensional analysis, enabling comprehensive monitoring of the queue system's operational status. This data integration strategy allows the system to analyze problems from a broader perspective, not limited to the state of a single component, but able to gain insight into the overall changes in the heterogeneous environment of the queue system, thereby more accurately identifying potential fault risks and optimization opportunities.

[0066] In one optional embodiment, during the calculation of the first index, the processing system first obtains the baseline queue depth value corresponding to the data collection period of the running data, wherein the baseline queue depth value is the queue depth boundary value determined based on P groups of historical running data of the queue system, where P is a positive integer. Then, the processing system determines the current queue depth value of the queue system based on the running data. Next, the processing system calculates the ratio of the current queue depth value to the baseline queue depth value. Finally, the processing system performs logarithmic compression based on the comparison value using a preset logarithmic compression algorithm to obtain the first index.

[0067] Optionally, the baseline queue depth value is calculated based on the historical operating data of the queue system through time series analysis and statistical methods. It represents the expected range of queue depth under normal operating conditions. The evaluation process of the baseline queue depth value comprehensively considers the seasonal fluctuations, long-term trends and random disturbances of the queue system.

[0068] Optionally, compared with a single static threshold detection method, the baseline queue depth value in this application can more accurately reflect the average queue depth of the queue system over a historical period, avoiding false alarms or missed alarms caused by static thresholds.

[0069] Optionally, the processing system performs logarithmic compression based on the comparison value of a preset logarithmic compression algorithm. The application of the logarithmic compression algorithm can effectively process ratio data with large magnitude variations, avoid ignoring small queue anomalies, and prevent excessive alarms caused by large deviations.

[0070] In one optional embodiment, during the calculation of the second indicator, the processing system first obtains the historical message processing efficiency corresponding to the data collection period of the running data, wherein the historical message processing efficiency is the maximum message processing efficiency of the queue system within a preset historical time period. Then, the processing system determines the current message processing efficiency of the queue system based on the running data, wherein the current message processing efficiency is used to characterize the message processing capability of the queue system's consumers during the current data collection period. Finally, the processing system determines the second indicator based on the ratio of the current message processing efficiency to the historical message processing efficiency.

[0071] Optionally, by determining the current message processing efficiency in real time, the processing system can ensure that the monitoring system can quickly capture the instantaneous status of the consumer's processing capacity. Especially under high concurrency, any fluctuation in the consumer's processing efficiency may quickly lead to message backlog, thereby affecting business continuity. Real-time acquisition of the current message processing efficiency can help the system intervene before message backlog occurs, avoid service degradation, and ensure stable business operation.

[0072] Optionally, by determining the historical message processing efficiency, the processing system can provide a baseline for comparing the message processing efficiency of the current period. The historical message processing efficiency reflects the highest performance level that the queue system can achieve under normal operating conditions. Therefore, by comparing it with the current message processing efficiency, it is possible to quickly identify whether the current consumer processing capacity is close to saturation. This helps to discover potential consumer saturation or consumer processing bottlenecks in the early stages, thereby taking preventive measures before the problem escalates and enhancing the system's self-adjustment and optimization capabilities.

[0073] In summary, this embodiment can accurately assess the saturation state of consumer processing capacity by calculating the ratio of the current message processing efficiency to the historical peak. The quantification of the second indicator (consumer saturation) provides an intuitive way to measure whether consumers are approaching their processing limits. Even if the queue depth is within the normal range, if the consumer processing capacity drops to near the saturation point, the processing system can immediately identify and take measures to avoid service degradation or interruption due to insufficient processing efficiency. The introduction of the second indicator enhances the early warning capability and fault prevention effect of the alarm system, and helps to improve the overall stability and efficiency of the distributed system.

[0074] In an optional embodiment, during the calculation of the third index, the processing system first determines the queue depth change sequence of the queue system based on the running data. The queue depth change sequence is used to characterize the change information of the total length of all messages included in the message queue. Then, the processing system performs linear regression fitting on the depth change sequence based on the short window and the long window in the dual time window to obtain the third index. The short window is used to determine the instantaneous growth rate of the queue messages in the queue system, and the long window is used to determine the changing trend of the queue messages in the queue system.

[0075] Optionally, a queue depth change sequence refers to a series of time points in time where the total length of the message queue changes, generated based on the queue system's operational data. This information is collected in real-time or periodically to form a curve or data stream that reflects the queue depth over time. For example, for an order queue used during peak trading hours, the queue depth change sequence can record the queue depth every minute, thus reflecting the changing trend of queue load.

[0076] Optionally, determining the queue depth change sequence provides a time series data basis for subsequent analysis, enabling the processing system to gain a more comprehensive understanding of the dynamic changes in queue depth based on historical and current data. This helps the processing system distinguish between regular fluctuations and potential faults, thereby providing timely warnings before queue backlog occurs and avoiding service interruptions.

[0077] Optionally, the processing system can quickly calculate the instantaneous message growth rate corresponding to the short window and the average message growth trend corresponding to the long window by performing linear regression fitting on the depth change sequence based on long and short windows.

[0078] Optionally, the combination of setting a dual time window and linear regression analysis can distinguish between two different types of queue depth changes: instantaneous surges and continuous deterioration. That is, by using a short window to identify instantaneous changes in queue depth, a rapid response can be made to sudden traffic events; at the same time, by using a long window to analyze the long-term trend of queue depth changes, the system can identify and warn of potential persistent backlog risks. This two-level analysis not only improves the timeliness of alarms but also enhances the system's ability to predict queue status changes. As a result, when anomalies occur, the system can respond more quickly and accurately, avoid escalation of faults, and ensure the stable operation of the system.

[0079] In summary, this embodiment achieves refined monitoring of queue depth changes by determining the queue depth change sequence and combining it with linear regression analysis using dual time windows. This mechanism not only improves the system's sensitivity to queue backlog risk perception but also distinguishes different types of depth changes, providing more accurate data support for subsequent decision-making and effectively enhancing the monitoring efficiency and stability of distributed systems and microservice architectures.

[0080] In one optional embodiment, before determining the first indicator of the queue system based on the operational data, the processing system first performs time-series decomposition on P groups of historical operational data to obtain the seasonal term, trend term, and residual term corresponding to the P groups of historical operational data. The seasonal term is used to characterize the periodic variation pattern of the message queue depth in the queue system, the trend term is used to characterize the periodic variation trend of the message queue depth in the queue system, and the residual term is used to characterize the random disturbance variation amplitude of the message queue depth in the queue system. Then, based on the seasonal term and the trend term, the processing system determines the first depth value corresponding to the data collection period of the operational data. Finally, based on the residual term and the first depth value, the processing system determines the baseline queue depth value corresponding to the data collection period of the operational data.

[0081] Optionally, the processing system employs STL (Seasonal-Trend decomposition using Loess) time series decomposition technology to decompose historical queue depth data into three parts: seasonal, trend, and residual components. Through the time series decomposition process, the processing system comprehensively analyzes the periodic patterns, long-term trends, and random disturbances of queue depth. Therefore, the processing system can more accurately understand the normal depth range of the queue under different times and business environments, thereby improving the adaptability and accuracy of the determined baseline queue depth value.

[0082] Optionally, the determination of the first depth value is based on the seasonal patterns and long-term trends of the queue system's operating data. This predictive method can provide the system with a dynamically adjusted normal depth expectation. Even during peak or off-peak periods, it can accurately reflect the normal range of the queue depth for the current period, thereby helping to reduce the false alarm rate and ensuring that the alarm system can accurately identify abnormal backlogs under different business cycles, thus improving the accuracy and timeliness of alarms.

[0083] Optionally, this embodiment determines the baseline queue depth value corresponding to the data collection period of the running data based on the residual term and the first depth value. This can effectively filter out random disturbances in the changes in message queue depth, ensuring that the baseline value is more stable and reliable. Combined with the first depth value, the baseline value generated by this method not only considers the periodic and trend changes in queue depth, but also the impact of random fluctuations, making the monitoring threshold of the system closer to the actual operating conditions. This not only reduces false alarms caused by random system jitter, but also identifies depth changes that are truly outside the normal range earlier and more accurately, thereby providing early warning and intervention in the early stages of a problem and avoiding systemic failures.

[0084] In summary, this embodiment, through time-series decomposition, prediction of the first depth value, and dynamic determination of the baseline queue depth value, can achieve precise monitoring of changes in the queue system depth. The above steps not only improve the accuracy and reliability of the alarm system, but also adapt to the dynamic changes of the queue system, providing strong technical support for maintaining the stability and efficient operation of distributed systems and microservice architectures.

[0085] In one optional embodiment, when the index entropy value is greater than a first preset entropy value, the processing system determines that the queue system is in a first system state. Then, the processing system transmits the alarm information corresponding to the first system state to the operation and maintenance personnel and triggers the circuit breaker mechanism. The circuit breaker mechanism is used to directly perform system service isolation / traffic degradation operations based on preset policies.

[0086] Optionally, a higher index entropy value indicates a greater risk of imbalance or failure in the queue system.

[0087] Optionally, the first system state refers to an "emergency state". When the index entropy value exceeds the first preset entropy value, it means that there is a serious message backlog or other emergency anomalies in the queue system, and the operation and maintenance personnel need to intervene immediately.

[0088] Optionally, a circuit breaker mechanism refers to a predefined security measure whereby once the queuing system enters the first system state, the processing system skips the subsequent fault location process and directly and automatically executes preset strategies, such as service isolation or traffic degradation, to prevent the fault from spreading.

[0089] Optionally, this embodiment combines the real-time transmission of alarm information with the automatic triggering of the circuit breaker mechanism to form a mechanism for real-time response and proactive defense. When the queue system faces severe backlog or other high-risk problems, the operation and maintenance personnel can receive immediate notification. At the same time, the processing system automatically takes measures to limit the scope of the fault, avoiding the delay and uncertainty of manual intervention, thereby effectively controlling the spread of the fault, enhancing the system's self-repair and recovery capabilities, and providing a solid technical guarantee for the high availability and efficient operation of the distributed system.

[0090] In one optional embodiment, when the index entropy value is less than or equal to a first preset entropy value and greater than a second preset entropy value, the processing system determines that the queue system is in a second system state, wherein the second preset threshold is less than the first preset threshold. Then, the processing system transmits the alarm information corresponding to the second system state to the operation and maintenance personnel of the queue system and performs correlation analysis on the queue system. Then, the processing system locates the fault based on the analysis results corresponding to the correlation analysis and obtains a fault diagnosis report. Finally, the processing system creates a work order based on the fault diagnosis report, wherein the work order is used to prompt the operation and maintenance personnel to repair the fault in the queue system.

[0091] In the above embodiments, the association analysis includes at least one of the following: The first correlation analysis is used to analyze the correlation between the backlog state of the thread pool and the pause time of the garbage collection mechanism; the second correlation analysis is used to analyze the correlation between the backlog state of the send queue in the buffer and the bandwidth utilization; the third correlation analysis is used to analyze the phase matching relationship between the depth growth curve of the message queue and the processing capacity curve of the consumer.

[0092] Optionally, the second system state refers to the "early warning state". In this embodiment, by setting a first preset entropy value and a second preset entropy value, the system state is determined hierarchically. This can accurately distinguish different levels of system risks, making it easier to take appropriate response measures. Compared with a single threshold alarm, this mechanism can identify potential problems earlier and improve the system's early warning capability and fault prevention effect.

[0093] Optionally, the introduction of correlation analysis makes fault location in the queue system more accurate and facilitates the rapid identification of the source of the problem. Compared with traditional fault diagnosis, this early warning and accurate location method can significantly reduce fault location time, speed up the problem-solving process, and effectively reduce the negative impact of faults on business.

[0094] Optionally, a fault diagnosis report refers to a detailed report generated based on the fault location results. It includes information such as the root cause of the fault, the scope of impact, and suggested response strategies, providing guidance for the next steps. Through a detailed fault diagnosis report, maintenance personnel can quickly connect the nature and scope of the queue fault, thereby taking targeted remedial measures. In this embodiment, this data analysis-based fault location method can find the problem faster and more accurately than subjective judgment or step-by-step troubleshooting, reducing the blindness and time consumption in the fault location process, and improving maintenance efficiency and the overall recovery speed of the system.

[0095] Optionally, a work order refers to a task order established for the fault repair process. Based on the content of the fault diagnosis report, the work order records in detail the specific situation of the fault, the expected solution, the completion deadline, the responsible person, and other information. It is a formal record and work guide for the fault repair process.

[0096] In summary, this embodiment distinguishes different system states by setting two thresholds, and performs correlation analysis, fault location, and work order creation in the second system state. This enables early warning and accurate repair of potential problems in the queue system. The execution of the above steps not only speeds up fault response and reduces fault location time, but also improves the efficiency and quality of system recovery through standardized fault repair processes, greatly enhancing the stability and availability of distributed systems and microservice architectures.

[0097] In one optional embodiment, when the index entropy value is less than or equal to a second preset entropy value, the processing system determines that the queue system is in a third system state. Then, when the queue system is in the third system state, the processing system archives the running data. Based on the archived running data, the processing system updates the baseline queue depth value of the queue system and generates a system state assessment report corresponding to the queue system.

[0098] Optionally, archiving refers to saving currently running data to a historical data warehouse for subsequent analysis, baseline updates, and performance optimization.

[0099] Optionally, the system status assessment report, which is a document that comprehensively describes the health status of the queue system, includes the current queue depth, processing capacity usage, system performance trends, and optimization suggestions. The generation of the system status assessment report provides operations and maintenance personnel with a comprehensive view of the system, which facilitates the regular review and planning of the queue system and helps to identify potential problems in the queue system in advance, thereby enabling preventive measures to avoid future system failures.

[0100] In summary, the above steps together constitute a complete system stability and performance optimization mechanism. This embodiment ensures the efficient operation of the queue system through precise state determination, proper data preservation, and baseline updates based on historical data analysis. At the same time, it improves the system's self-detection and repair capabilities, reduces unnecessary intervention, and thus enhances the stability of the entire business system and the user experience.

[0101] As described above, this application first uses an adaptive preset frequency adjustment mechanism to dynamically adjust the frequency of queue operation data collection based on the real-time system status of the queue system, thereby balancing the timeliness of the collected operation data with the utilization rate of data collection resources. Then, this application deeply analyzes the nature and extent of message backlog in the message queue, obtaining three-dimensional key performance indicators: message queue depth deviation, consumer saturation, and message growth rate. Finally, this application calculates the weighted entropy value of the above three-dimensional key performance indicators to obtain an entropy value that accurately characterizes the probability of message backlog risk. This provides a quantitative basis for intelligent decision-making in the queue system while achieving the technical effect of improving the accuracy of message backlog risk warning in the queue system.

[0102] Therefore, this application adopts a multi-dimensional and dynamic evaluation method of the queue system's operating data. Through an adaptive preset frequency adjustment mechanism and entropy calculation, it achieves the goal of accurately assessing the risk of message accumulation in the queue system, thereby reducing the false alarm rate of message accumulation warning. This solves the technical problem of high false alarm rate in existing technologies for message accumulation warning during queue system monitoring.

[0103] In one alternative embodiment, Figure 3 This is an architecture diagram of an optional queue system message backlog processing system according to an embodiment of this application. Figure 4 This is a flowchart of an optional queue system message backlog processing system according to an embodiment of this application.

[0104] Optionally, such as Figure 3 As shown, the processing system is divided into four layers from bottom to top: data acquisition layer, computing layer, decision layer, and execution layer. The overall system adopts a highly fault-tolerant architecture design, with each layer deployed independently and having fault isolation capabilities.

[0105] (1) The data acquisition layer, as the basic sensing unit of the system, is responsible for the unified collection and processing of various product-Q indicators in heterogeneous environments: For message queue scenarios, the processing system installs the corresponding Exporter on the host where the message queue server resides and configures Prometheus's fetching rules, allowing Prometheus to periodically pull data from the Exporter, including core metrics such as queue depth, producer input rate, and consumer acknowledgment rate. For example, when new messages are generated in a Kafka (an open-source stream processing platform) cluster, the Kafka Exporter monitors the changes in the number of messages in real time and then passes data such as the producer input rate to Prometheus.

[0106] For network communication scenarios, the processing system employs eBPF kernel-level monitoring technology. This technology allows programs to run directly within the Linux kernel without modifying the kernel source code or loading modules, enabling the acquisition of depth changes in the Socket buffer's Send-Q / Recv-Q capabilities and packet transmission latency. For example, the processing system writes an eBPF program, compiles and loads it using the bcc toolkit, and then deploys it on the server requiring monitoring. When data transmission occurs on the network, the eBPF program can track the entry and exit of data packets in the Socket buffer in real time.

[0107] For Java thread pool scenarios, the system uses the Byte Buddy bytecode injection tool. This tool dynamically modifies the bytecode of thread pool-related classes during application startup, inserting monitoring code to dynamically collect key data such as task queue length, active thread count, and task submission frequency. For example, by including the Byte Buddy dependency packages in the application's classpath and configuring the thread pool classes and methods to be monitored, the injected monitoring logic will automatically execute and collect data as the thread pool processes tasks during application runtime.

[0108] For monitoring consumer processes, the processing system uses the Telegraf pluggable data collector. This collector supports multiple input pluggins to synchronously collect information such as CPU utilization, memory usage, process status, and processing speed. During deployment, the processing system deploys Telegraf to the host where the consumer process resides, configures the procstat (a process status statistics tool) pluggin, specifies the consumer process ID to be monitored, and then Telegraf collects relevant data at the set time intervals.

[0109] Finally, all the collected raw runtime data is sent to Logstash (a data pipeline). Logstash performs real-time cleaning and filtering through configured filters. Then, the processing system stores the processed runtime data in Elasticsearch (a distributed search engine and storage system) according to the specified index structure for subsequent time-series analysis and processing in the computing layer.

[0110] (2) The computation layer is used to build a multi-stage data processing pipeline to achieve accurate transformation from raw data to decision indicators: Optionally, the depth deviation Dev = log10(Q_current / Q_baseline), where Q_current refers to the current queue depth value and Q_baseline refers to the baseline queue depth value.

[0111] Optionally, the consumer saturation Sat = min(1, TPS_current / TPS_peak), where TPS_current refers to the real-time TPS and TPS_peak refers to the peak TPS.

[0112] Optionally, in the message growth rate ΔQ / Δt, ΔQ refers to the number of messages growing, and Δt refers to the size of the time window.

[0113] Optionally, the processing system outputs a risk score using the weighted entropy formula E=0.4 * max(1,Dev) + 0.3 * (1 - Sat) +0.3 * Sigmoid(ΔQ / Δt), thereby providing a quantitative basis for decision-making.

[0114] (3) The decision-making layer, as the intelligent analysis center of the system, undertakes the core function of transforming quantitative indicators into executable strategies: In an emergency (entropy value > 0.85), the system automatically triggers a voice call alarm and SMS to the operations and maintenance manager, and performs protective actions such as service isolation or traffic degradation according to preset policies.

[0115] In the early warning state (entropy value 0.7-0.85), the system sends alarm SMS and email to the operations and maintenance manager, generates a structured diagnostic report and automatically creates an operations and maintenance work order, starts the automated root cause localization tool for in-depth detection, and reserves a backup resource pool in a hot standby state to prepare for potential upgrades.

[0116] Under normal conditions (entropy ≤ 0.7), the system continuously archives time-series data for trend modeling and analysis, dynamically updates baseline calculation model parameters, and periodically generates system health assessment reports for optimization reference.

[0117] All critical operations are logged and stored in the database for evidence preservation. The complete record includes the operation time, execution account, and resource change details, forming an audit log. Ultimately, a complete operation and maintenance closed loop of "real-time monitoring - intelligent decision-making - precise execution - effect verification" is built, which greatly improves the system's autonomy and emergency response reliability.

[0118] (4) The execution layer constructs a closed-loop response ecosystem from diagnosis to handling, and achieves seamless connection of operation and maintenance through a three-level collaboration mechanism. The processing system handles the emergency state, early warning state, and normal state of the decision-making layer respectively: In an emergency (entropy value > 0.85), the system automatically triggers a voice call alarm and SMS to the operations and maintenance manager, and performs protective actions such as service isolation or traffic degradation according to preset policies.

[0119] In the early warning state (entropy value 0.7-0.85), the system sends alarm SMS and email to the operations and maintenance manager, generates a structured diagnostic report and automatically creates an operations and maintenance work order, starts the automated root cause localization tool for in-depth detection, and reserves a backup resource pool in a hot standby state to prepare for potential upgrades.

[0120] Under normal conditions (entropy ≤ 0.7), the system continuously archives time-series data for trend modeling and analysis, dynamically updates baseline calculation model parameters, and periodically generates system health assessment reports for optimization reference.

[0121] All critical operations are logged and stored in the database for evidence preservation. The complete record includes the operation time, execution account, and resource change details, forming an audit log. Ultimately, a complete operation and maintenance closed loop of "real-time monitoring - intelligent decision-making - precise execution - effect verification" is built, which greatly improves the system's autonomy and emergency response reliability.

[0122] In summary, this embodiment, through the synergistic innovation of a multi-dimensional dynamic evaluation model and an intelligent decision engine, completely solves the problems of high false alarm rate and low diagnostic accuracy in traditional Q-accumulation monitoring. Compared with existing technologies, this solution achieves three major breakthroughs: First, it innovatively integrates four-dimensional indicators—queue depth deviation, consumer saturation, message growth rate, and resource coupling—combined with a dynamic baseline algorithm and entropy decision mechanism, achieving high-accuracy identification and significantly reducing the false alarm rate. Second, it constructs a unified monitoring framework, supporting rapid access to heterogeneous scenarios such as message queues, network communication, and thread pools through parameterized adapters, and is easily scalable. Third, it achieves closed-loop management from monitoring to handling, automatically triggering precise scaling strategies. This solution transforms traditional passive response into proactive defense, providing a new paradigm for Q-accumulation monitoring in distributed systems that combines accuracy, universality, and economy.

[0123] Example 2 This application embodiment can also provide a message accumulation processing device for a queue system. It should be noted that the message accumulation processing device for a queue system in this application embodiment can be used to execute the message accumulation processing method for a queue system provided in this application embodiment. The message accumulation processing device for a queue system provided in this application embodiment will be described below.

[0124] According to an embodiment of this application, an apparatus for implementing the above-described message backlog processing method in a queue system is also provided. Figure 5 This is a schematic diagram of an optional message backlog processing apparatus for a queuing system according to an embodiment of this application, as shown below. Figure 5 As shown, the device includes: a data acquisition unit 501, an index determination unit 502, an entropy determination unit 503, and a decision-making unit 504.

[0125] Optionally, the data acquisition unit 501 is used to acquire the operation data of the queue system based on a preset frequency adjustment mechanism, wherein the preset frequency adjustment mechanism is used to update the data acquisition cycle based on the system status of the queue system; the index determination unit 502 is used to determine the first index, the second index, and the third index of the queue system based on the operation data, wherein the first index is the depth deviation of the message queue determined by logarithmic compression operation, the second index is the consumer saturation of the queue system, and the third index is the message growth rate of the queue system calculated based on a dual time window; the entropy value determination unit 503 is used to determine the index entropy value of the queue system based on the first index, the second index, and the third index, wherein the index entropy value is used to characterize the probability of message backlog risk in the queue system; and the decision unit 504 is used to perform decision processing on the queue system based on the index entropy value.

[0126] In one optional embodiment, the data acquisition unit 501 includes: a first acquisition subunit, a second acquisition subunit, a third acquisition subunit, a fourth acquisition subunit, and a running data determination subunit.

[0127] Optionally, the first acquisition subunit is used to acquire message queue data through a preset component, wherein the message queue data includes at least the queue depth, producer input rate, and consumer acknowledgment rate of the message queue in the queue system, and the preset component is deployed on the host where the message queue is located; the second acquisition subunit is used to acquire buffer data through a preset program, wherein the buffer data is used to characterize at least the depth change information of the receive queue / send queue included in the buffer of the queue system and packet transmission delay information, and the preset program is used to run directly in the kernel of the queue system; the third acquisition subunit is used to acquire thread pool data through a preset tool, wherein the thread pool data includes at least the queue depth, number of active threads, and task submission frequency of the task queue in the queue system, and the preset tool is used to insert data acquisition code in the form of bytecode injection; the fourth acquisition subunit is used to acquire consumer process data through a preset plugin, wherein the consumer process data includes at least the processor utilization, memory usage, process status, and process processing speed corresponding to the consumer process in the task queue; and the running data determination subunit is used to use the message queue data, buffer data, thread pool data, and consumer process data as the running data of the queue system.

[0128] In one optional embodiment, the index determination unit 502 includes: a baseline depth value acquisition subunit, a current depth value determination subunit, a ratio calculation subunit, and a first index determination subunit.

[0129] Optionally, the baseline depth value acquisition subunit is used to acquire the baseline queue depth value corresponding to the data acquisition period of the running data, wherein the baseline queue depth value is the queue depth boundary value determined based on P groups of historical running data of the queue system, and P is a positive integer; the current depth value determination subunit is used to determine the current queue depth value of the queue system based on the running data; the ratio calculation subunit is used to calculate the ratio of the current queue depth value to the baseline queue depth value; and the first index determination subunit is used to perform logarithmic compression based on the comparison value of the preset logarithmic compression algorithm to obtain the first index.

[0130] In an optional embodiment, the indicator determination unit 502 further includes: a historical efficiency acquisition subunit, a current efficiency determination subunit, and a second indicator determination subunit.

[0131] Optionally, the historical efficiency acquisition subunit is used to acquire the historical message processing efficiency corresponding to the data collection period of the running data, wherein the historical message processing efficiency is the maximum message processing efficiency of the queue system within a preset historical time period; the current efficiency determination subunit is used to determine the current message processing efficiency of the queue system based on the running data, wherein the current message processing efficiency is used to characterize the message processing capability of the queue system's consumers during the current data collection period; and the second indicator determination subunit is used to determine the second indicator based on the ratio of the current message processing efficiency to the historical message processing efficiency.

[0132] In an optional embodiment, the index determination unit 503 further includes a depth change sequence determination subunit and a third index determination subunit.

[0133] Optionally, a depth change sequence determination subunit is used to determine the queue depth change sequence of the queue system based on operational data, wherein the queue depth change sequence is used to characterize the change information of the total length of all messages included in the message queue; a third index determination subunit is used to perform linear regression fitting on the depth change sequence based on a short window and a long window in a dual time window to obtain a third index, wherein the short window is used to determine the instantaneous growth rate of queue messages in the queue system, and the long window is used to determine the changing trend of queue messages in the queue system.

[0134] In an optional embodiment, the message accumulation processing device for the queue system further includes: a timing decomposition unit, a first baseline depth determination unit, and a second baseline depth determination unit.

[0135] Optionally, the time-series decomposition unit is used to perform time-series decomposition on P groups of historical operating data to obtain the seasonal term, trend term, and residual term corresponding to the P groups of historical operating data. The seasonal term is used to characterize the periodic variation pattern of the message queue depth in the queue system, the trend term is used to characterize the periodic variation trend of the message queue depth in the queue system, and the residual term is used to characterize the random disturbance variation amplitude of the message queue depth in the queue system. The first baseline depth determination unit is used to determine the first depth value corresponding to the data acquisition period of the operating data based on the seasonal term and the trend term. The second baseline depth determination unit is used to determine the baseline queue depth value corresponding to the data acquisition period of the operating data based on the residual term and the first depth value.

[0136] In an optional embodiment, the decision unit 504 includes: a first system state determination subunit and a circuit breaker triggering subunit.

[0137] Optionally, the first system state determination subunit is used to determine that the queue system is in the first system state when the index entropy value is greater than the first preset entropy value; the circuit breaker triggering subunit is used to transmit the alarm information corresponding to the first system state to the operation and maintenance personnel and trigger the circuit breaker mechanism, wherein the circuit breaker mechanism is used to directly perform system service isolation / traffic degradation operations based on the preset policy.

[0138] In one optional embodiment, the decision unit 504 includes: a second system status determination subunit, a correlation analysis subunit, a fault location subunit, and a work order creation subunit.

[0139] Optionally, the second system state determination subunit is used to determine that the queue system is in a second system state when the index entropy value is less than or equal to a first preset entropy value and greater than a second preset entropy value, wherein the second preset threshold is less than the first preset threshold; the correlation analysis subunit is used to transmit the alarm information corresponding to the second system state to the operation and maintenance personnel of the queue system, and to perform correlation analysis on the queue system, wherein the correlation analysis includes at least one of the following: a first correlation analysis, used to analyze the correlation between the backlog status of the thread pool and the pause time of the garbage collection mechanism; a second correlation analysis, used to analyze the correlation between the backlog status of the sending queue in the buffer and the bandwidth utilization rate; a third correlation analysis, used to analyze the phase matching relationship between the depth growth curve of the message queue and the processing capacity curve of the consumer; the fault location subunit is used to locate the fault based on the analysis results corresponding to the correlation analysis and obtain a fault diagnosis report; the work order creation subunit is used to create a work order based on the fault diagnosis report, wherein the work order is used to prompt the operation and maintenance personnel to repair the fault in the queue system.

[0140] In an optional embodiment, the decision unit 504 includes: a third system state determination subunit, an archiving subunit, and a depth value update subunit.

[0141] Optionally, the third system state determination subunit is used to determine that the queue system is in the third system state when the index entropy value is less than or equal to the second preset entropy value; the archiving subunit is used to archive the running data when the queue system is in the third system state; and the depth value update subunit is used to update the baseline queue depth value of the queue system based on the archived running data and generate a system state evaluation report corresponding to the queue system.

[0142] As described above, this device first uses an adaptive preset frequency adjustment mechanism to dynamically adjust the frequency of queue operation data collection based on the real-time system status of the queue system, thereby balancing the timeliness of the collected operation data with the utilization rate of data collection resources. Then, the device deeply analyzes the nature and extent of message backlog in the message queue, obtaining three-dimensional key performance indicators: message queue depth deviation, consumer saturation, and message growth rate. Finally, by calculating the weighted entropy value of the above three-dimensional key performance indicators, the device obtains an entropy value that accurately characterizes the probability of message backlog risk. This provides a quantitative basis for the intelligent decision-making of the queue system and achieves the technical effect of improving the accuracy of message backlog risk warning in the queue system.

[0143] Therefore, this device adopts a multi-dimensional and dynamic evaluation method of the queue system's operating data. Through an adaptive preset frequency adjustment mechanism and entropy calculation, it achieves the goal of accurately assessing the risk of message accumulation in the queue system, thereby reducing the false alarm rate of message accumulation warning. This solves the technical problem of high false alarm rate in existing technologies for message accumulation warning during queue system monitoring.

[0144] It should be noted that the data acquisition unit 501, the indicator determination unit 502, the entropy value determination unit 503, and the decision-making unit 504 mentioned above correspond to steps S101 to S104 in the method embodiment. The instances and application scenarios implemented by the above units and the corresponding steps are the same, but are not limited to the content disclosed in the above embodiment.

[0145] Example 3 Embodiments of this application can also provide an electronic device. Figure 6 This is a structural block diagram of an electronic device according to an embodiment of this application, such as... Figure 6 As shown, the electronic device includes: one or more ( Figure 6 (Only one is shown) Processor 602, memory 604, memory controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0146] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and devices in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, to realize the above-mentioned message accumulation processing method of the queue system.

[0147] The memory may include high-speed random access memory (RAM), and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, which can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks (LANs), mobile communication networks, and combinations thereof.

[0148] The processor can access information and applications stored in memory via a transmission device to execute the following steps: Based on a preset frequency adjustment mechanism, collect operational data of the queue system, wherein the preset frequency adjustment mechanism is used to update the data collection cycle based on the system status of the queue system; determine a first, second, and third indicator of the queue system based on the operational data, wherein the first indicator is the message queue depth deviation determined by logarithmic compression, the second indicator is the consumer saturation of the queue system, and the third indicator is the message growth rate of the queue system calculated based on a dual time window; determine the indicator entropy value of the queue system based on the first, second, and third indicators, wherein the indicator entropy value is used to characterize the probability of message backlog risk in the queue system; and perform decision processing on the queue system based on the indicator entropy value.

[0149] This application provides a solution for handling message backlog in a queue system. As described above, this application first uses an adaptive preset frequency adjustment mechanism to dynamically adjust the frequency of queue operation data collection based on the real-time system status, thereby balancing the timeliness of the collected operation data with the utilization rate of data collection resources. Then, this application deeply analyzes the nature and degree of message backlog in the message queue, obtaining three-dimensional key performance indicators: message queue depth deviation, consumer saturation, and message growth rate. Finally, this application calculates the weighted entropy value of the above three-dimensional key performance indicators to obtain an entropy value that accurately characterizes the probability of message backlog risk. This provides a quantitative basis for intelligent decision-making in the queue system and improves the accuracy of message backlog risk warnings in the queue system.

[0150] Therefore, this application adopts a multi-dimensional and dynamic evaluation method of the queue system's operating data. Through an adaptive preset frequency adjustment mechanism and entropy calculation, it achieves the goal of accurately assessing the risk of message accumulation in the queue system, thereby reducing the false alarm rate of message accumulation warning. This solves the technical problem of high false alarm rate in existing technologies for message accumulation warning during queue system monitoring.

[0151] Those skilled in the art will understand that Figure 6The structure shown is for illustrative purposes only. Electronic devices can also be smartphones, tablets, PDAs, mobile internet devices, PADs, and other terminal devices. Figure 6 This does not limit the structure of the aforementioned electronic device. For example, electronic devices may also include components that are more... Figure 6 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 6 The different configurations shown.

[0152] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0153] Example 4 Embodiments of this application may also provide a storage medium.

[0154] Optionally, in this embodiment of the application, the storage medium can be used to store the program code executed by the message backlog processing method of the queue system provided in the above method embodiment.

[0155] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0156] This application also provides a computer program product, which, when executed on a data processing device, is adapted to perform the steps of a method for processing message accumulation in a queue system.

[0157] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0158] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0159] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0160] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0161] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0162] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0163] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for handling message backlog in a queue system, characterized in that, include: Based on a preset frequency adjustment mechanism, the operation data of the queue system is collected, wherein the preset frequency adjustment mechanism is used to update the data collection cycle based on the system status of the queue system. Based on the operational data, a first indicator, a second indicator, and a third indicator of the queue system are determined. The first indicator is the depth deviation of the message queue determined by logarithmic compression operation, the second indicator is the consumer saturation of the queue system, and the third indicator is the message growth rate of the queue system calculated based on a dual time window. The index entropy value of the queue system is determined based on the first index, the second index, and the third index, wherein the index entropy value is used to characterize the probability of message backlog risk occurring in the queue system; The queue system makes decisions based on the entropy value of the aforementioned indicator.

2. The method for handling message backlog in a queue system according to claim 1, characterized in that, Collect operational data from the queue system, including: Message queue data is collected through a preset component, wherein the message queue data includes at least the queue depth, producer input rate, and consumer confirmation rate of the message queue in the queue system, and the preset component is deployed on the host where the message queue is located. Buffer data is collected through a preset program, wherein the buffer data is used to characterize at least the depth change information of the receive queue / send queue included in the buffer of the queue system and the packet transmission delay information, and the preset program is used to run directly in the kernel of the queue system; Data from the thread pool is collected using a preset tool. The thread pool data includes at least the queue depth, number of active threads, and task submission frequency of the task queue in the queue system. The preset tool is used to insert data collection code through bytecode injection. Consumer process data is collected through a preset plugin, wherein the consumer process data includes at least the processor utilization, memory usage, process status and process processing speed of the consumer process in the task queue. The message queue data, the buffer data, the thread pool data, and the consumer process data are used as the running data of the queue system.

3. The method for handling message backlog in a queue system according to claim 1, characterized in that, Based on the operational data, a first indicator of the queue system is determined, including: Obtain the baseline queue depth value corresponding to the data collection period of the running data, wherein the baseline queue depth value is a queue depth boundary value determined based on P groups of historical running data of the queue system, and P is a positive integer; The current queue depth value of the queue system is determined based on the operational data; Calculate the ratio of the current queue depth value to the baseline queue depth value; The ratio is logarithmically compressed using a preset logarithmic compression algorithm to obtain the first index.

4. The method for handling message backlog in a queue system according to claim 1, characterized in that, Based on the operational data, a second indicator of the queue system is determined, including: The historical message processing efficiency corresponding to the data collection period of the running data is obtained, wherein the historical message processing efficiency is the maximum message processing efficiency of the queue system within a preset historical time period; The current message processing efficiency of the queue system is determined based on the operational data, wherein the current message processing efficiency is used to characterize the message processing capability of the consumers of the queue system during the current data collection period; The second indicator is determined based on the ratio of the current message processing efficiency to the historical message processing efficiency.

5. The method for handling message backlog in a queue system according to claim 1, characterized in that, The third metric of the queue system is determined based on the operational data, including: Based on the operational data, a queue depth change sequence of the queue system is determined, wherein the queue depth change sequence is used to characterize the change information of the total length of all messages included in the message queue; Based on the short window and long window in the dual time window, a linear regression fit is performed on the deep change sequence to obtain the third index, wherein the short window is used to determine the instantaneous growth rate of queue messages in the queue system, and the long window is used to determine the changing trend of queue messages in the queue system.

6. The method for handling message backlog in a queue system according to claim 3, characterized in that, Before determining the first metric of the queue system based on the operational data, the method for handling message backlog in the queue system further includes: The P groups of historical operating data are decomposed into time series to obtain the seasonal term, trend term and residual term corresponding to the P groups of historical operating data. The seasonal term is used to characterize the periodic change pattern of the message queue depth in the queue system, the trend term is used to characterize the periodic change trend of the message queue depth in the queue system, and the residual term is used to characterize the random disturbance change amplitude of the message queue depth in the queue system. Based on the seasonality and the trend, determine the first depth value corresponding to the data collection period of the operational data; Based on the residual term and the first depth value, the baseline queue depth value corresponding to the data acquisition period of the running data is determined.

7. The method for handling message backlog in a queue system according to claim 1, characterized in that, Decision processing for the queue system based on the aforementioned index entropy value includes: If the entropy value of the indicator is greater than the first preset entropy value, the queue system is determined to be in the first system state. The alarm information corresponding to the first system status is transmitted to the operation and maintenance personnel, and the circuit breaker mechanism is triggered. The circuit breaker mechanism is used to directly perform system service isolation / traffic degradation operations based on preset policies.

8. The method for handling message backlog in a queue system according to claim 1, characterized in that, Decision processing for the queue system based on the aforementioned index entropy value includes: If the index entropy value is less than or equal to a first preset entropy value and greater than a second preset entropy value, the queue system is determined to be in a second system state, wherein the second preset threshold is less than the first preset threshold. The alarm information corresponding to the second system status is transmitted to the operation and maintenance personnel of the queue system, and a correlation analysis is performed on the queue system, wherein the correlation analysis includes at least one of the following: The first correlation analysis is used to analyze the correlation between the backlog state of the thread pool and the pause time of the garbage collection mechanism; The second correlation analysis is used to analyze the correlation between the backlog status of the send queue in the buffer and the bandwidth utilization. The third correlation analysis is used to analyze the phase matching relationship between the depth growth curve of the message queue and the processing capacity curve of the consumer; Based on the analysis results corresponding to the correlation analysis, the fault is located and a fault diagnosis report is obtained. A work order is created based on the fault diagnosis report, wherein the work order is used to prompt maintenance personnel to repair the fault in the queue system.

9. The method for handling message backlog in a queue system according to claim 1, characterized in that, Decision processing for the queue system based on the aforementioned index entropy value includes: If the entropy value of the indicator is less than or equal to the second preset entropy value, the queue system is determined to be in the third system state. When the queue system is in the third system state, the operational data is archived; Based on the archived operational data, the baseline queue depth value of the queue system is updated, and a system status assessment report corresponding to the queue system is generated.

10. A processing device for message backlog in a queue system, characterized in that, include: A data acquisition unit is used to acquire the operation data of the queue system based on a preset frequency adjustment mechanism, wherein the preset frequency adjustment mechanism is used to update the data acquisition cycle based on the system status of the queue system. The indicator determination unit is used to determine a first indicator, a second indicator, and a third indicator of the queue system based on the operating data. The first indicator is the depth deviation of the message queue determined by logarithmic compression operation, the second indicator is the consumer saturation of the queue system, and the third indicator is the message growth rate of the queue system calculated based on a dual time window. An entropy value determination unit is used to determine the index entropy value of the queue system based on the first index, the second index, and the third index, wherein the index entropy value is used to characterize the probability of message backlog risk occurring in the queue system; The decision-making unit is used to make decisions about the queue system based on the index entropy value.

11. A computer program product, characterized in that, The computer program product includes a computer program, wherein, when the computer program is executed, it controls the computer program product to perform the message backlog processing method of any one of claims 1 to 9.

12. An electronic device, characterized in that, The system includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the message backlog processing method of any one of claims 1 to 9.

Images