Prometheus remote adaptive flow limiting and backfilling method and device and storage medium

Abstract

The invention discloses a Prometheus remote adaptive current limiting and backfilling method, a Prometheus remote adaptive current limiting and backfilling device and a storage medium, and relates to the field of computer monitoring, a lightweight bypass control module is introduced at the front end of a remote data path of Prometheus, a pressure state can be automatically identified during a system exception period, a current limiting strategy can be dynamically started, high-priority index uploading is preferentially guaranteed, and the backfilling efficiency is improved. And the temporarily stored low-priority indexes are safely backfilled after the network or service is recovered, so that the robustness of the monitoring system and the reliability of key data are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of computer monitoring, specifically to a Prometheus remote_write adaptive rate limiting and backfilling method, device, and storage medium. Background Technology

[0002] Prometheus, a widely adopted monitoring system, asynchronously pushes metric data to a remote server for storage or analysis via the remote_write protocol. This protocol writes metric data to multiple push queues, offering dynamic queue scaling and static rate limiting solutions to address metric explosions (such as a surge in high-based metrics). However, it struggles with situations where network jitter causes a sharp decrease in effective bandwidth, or when the remote push server experiences performance bottlenecks. This can lead to core business metrics failing to be uploaded to the central server in a timely manner, severely impacting system observability and fault location capabilities. Summary of the Invention

[0003] To overcome the shortcomings of the above technologies, this invention provides a method, apparatus, and storage medium for ensuring the reliability of core monitoring indicators in the remote_write protocol scenario of the Prometheus monitoring system, through a state machine and priority filtering mechanism, under network anomalies, server overload, or data surges.

[0004] The technical solution adopted by this invention to overcome its technical problems is: A Prometheus remote_write adaptive rate limiting and backfilling method includes: S1. When Prometheus starts, it loads the rate limiting module and performs initialization configuration, which initializes the state machine controller and adds it to the listening state. S2. In listening mode, the health evaluator samples once per second and updates the sliding window; S3. When severe network fluctuations occur, the state controller drives the system to switch from the listening state to the working state; S4. In the working state, when Prometheus submits a sample to the remote_write memory queue, the WAL filter writer processes it according to the configuration rules; S5. When the backfilling trigger condition is met, the state controller drives the system to switch from the working state to the backfilling state; S6. After the backfill scheduler confirms that all rate-limited WAL file data has been processed, it notifies the state machine controller, which then drives the system to switch from the backfill state to the listening state.

[0005] Furthermore, the method for initializing the state machine controller in step S1 is as follows: after Prometheus starts, it reads the sliding window time span T_window, the number of consecutive sampling cycles t required to trigger the abnormal sampling point, the backlog abnormal threshold b, the success rate abnormal threshold p, the number of consecutive abnormal sampling cycles N to trigger the working state, the backlog lower limit threshold c, and the continuous compliance time t1 from the working state to the backfilling state.

[0006] Furthermore, step S2 includes the following steps: S2-1. The health evaluator samples once per second. The sampled data includes: the real-time backlog of the current remote_write memory queue and the success rate of the most recent batch transmission within the current sampling period. S2-2. Set up two sliding windows. The fixed length of the circular buffer of the two sliding windows is equal to the time span T_window of the sliding window. The first sliding window is used as the backlog detection sliding window, and the second sliding window is used as the success rate detection sliding window. S2-3. When the queue backlog of the number of consecutive sampling periods t required to trigger the backlog detection sliding window is greater than or equal to the backlog anomaly threshold b, it is recorded as 1 backlog anomaly sampling point, and t is less than or equal to T_window; S2-4. When the transmission success rate of the number of consecutive sampling periods t required to trigger the past abnormal sampling points in the success rate detection sliding window is less than the success rate abnormal threshold p, it is recorded as 1 success rate abnormal sampling point.

[0007] The judgment logic for the two abnormal sampling points is parallel and independent, with no priority distinction. Both types of abnormal sampling points can be triggered simultaneously within a single sampling period.

[0008] Furthermore, in step S3, if the success rate of remote_write is less than the success rate abnormal threshold p in the success rate detection sliding window for N consecutive sampling periods, it is determined that the network has serious fluctuations. The health evaluator sends a flow limiting trigger signal to the state controller, and the state controller drives the system to switch from the listening state to the working state.

[0009] Furthermore, the configuration rule in step S4 is as follows: the original metrics stored by Prometheus are used as low-priority metrics, and the metrics generated by the record rules stored by Prometheus are used as high-priority metrics. When Prometheus submits a sample to the remote_write memory queue, if the sample is a high-priority metric, it is written to the remote_write memory queue normally. If the sample is a low-priority metric, it is serialized and appended to a separate local rate-limiting WAL file.

[0010] Furthermore, step S5 includes the following steps: S5-1. When the backlog in the remote_write memory queue is less than the backlog lower limit threshold c or the success rate is greater than or equal to the success rate exception threshold p, and the continuous compliance time from working state to backfilling state is t1, the state controller drives the system to switch from working state to backfilling state. S5-2. The backfill scheduler reads the rate-limited WAL file and resubmits the data to the standard remote_write memory queue. The Thanos server inserts the backfill data into the correct position based on the timestamp and time series identifier.

[0011] Furthermore, in step S6, after the system switches from the backfill state back to the listening state, it deletes the empty rate-limiting WAL file.

[0012] This invention also relates to a Prometheus remote_write adaptive rate limiting and backfilling device, comprising a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus; wherein: The memory is used to store computer programs; The processor is configured to execute by running programs stored in the memory: When Prometheus starts, it loads the rate limiting module and performs initialization configuration, which initializes the state machine controller and adds it to the listening state. In listening mode, the health evaluator samples once per second and updates the sliding window; When severe network fluctuations occur, the state controller drives the system to switch from listening state to working state. In the working state, when Prometheus submits samples to the remote_write memory queue, the WAL filter writer processes them according to the configuration rules; When the backfilling conditions are met, the state controller drives the system to switch from the working state to the backfilling state. After the backfill scheduler confirms that all rate-limited WAL file data has been processed, it notifies the state machine controller, which then drives the system to switch from the backfill state to the listening state.

[0013] The present invention also relates to a computer-readable storage medium having a computer program stored thereon, the computer program being implemented when executed by a processor: When Prometheus starts, it loads the rate limiting module and performs initialization configuration, which initializes the state machine controller and adds it to the listening state. In listening mode, the health evaluator samples once per second and updates the sliding window; When severe network fluctuations occur, the state controller drives the system to switch from listening state to working state. In the working state, when Prometheus submits samples to the remote_write memory queue, the WAL filter writer processes them according to the configuration rules; When the backfilling conditions are met, the state controller drives the system to switch from the working state to the backfilling state. After the backfill scheduler confirms that all rate-limited WAL file data has been processed, it notifies the state machine controller, which then drives the system to switch from the backfill state to the listening state.

[0014] The beneficial effects of this invention are as follows: The technical solution of this invention can ensure that critical business indicators are prioritized during network jitter, server overload, or high cardinality storms, ensuring that maintenance personnel can always obtain the most critical system status information, greatly improving diagnostic capabilities during faults. It effectively avoids the impact of high cardinality indicators on remote storage, preventing the entire monitoring system from collapsing; under normal circumstances, it only listens without additional performance loss; non-core indicators are persisted through WAL and automatically backfilled after system recovery, ensuring the eventual consistency and historical traceability of all data. It is fully compatible with the existing Prometheus remote_write protocol and ecosystem, requiring no modification to the remote storage end. This invention can be integrated as a plugin or built-in module, resulting in low deployment costs. Attached Figure Description

[0015] Figure 1 This is a system structure diagram of the present invention; Figure 2 This is a flowchart of the method of the present invention. Detailed Implementation

[0016] The following is in conjunction with the appendix Figure 1 Appendix Figure 2 The present invention will be further described below.

[0017] Example 1 A Prometheus remote_write adaptive rate limiting and backfilling method includes: When S1.Prometheus starts, it loads the rate limiting module and performs initialization configuration, which in turn initializes the state machine controller and adds it to the listening state. S2. In listening mode, the health evaluator samples once per second and updates the sliding window.

[0018] S3. When severe network fluctuations occur, the state controller drives the system to switch from the listening state to the working state.

[0019] S4. In the working state, when Prometheus submits samples to the remote_write memory queue, the WAL filter writer processes them according to the configuration rules.

[0020] S5. When the backfilling condition is met, the state controller drives the system to switch from the working state to the backfilling state.

[0021] S6. After the backfill scheduler confirms that all rate-limited WAL file data has been processed, it notifies the state machine controller, which then drives the system to switch from the backfill state to the listening state.

[0022] It can ensure the reliable uploading of core monitoring indicators and the observability of core links during network anomalies, server overload, or data surges, preventing the entire system from overloading and crashing, and under normal circumstances, it has no impact on the performance of the main process.

[0023] In one embodiment of the present invention, the method for initializing the state machine controller in step S1 is as follows: after Prometheus starts, it reads the sliding window time span T_window (unit: seconds), the number of consecutive sampling cycles t required to trigger the abnormal sampling point, the backlog abnormal threshold b, the success rate abnormal threshold p, the number of consecutive abnormal sampling cycles N to trigger the working state, the backlog lower limit threshold c (c is less than b), and the continuous compliance time t1 (unit: seconds) from the working state to the backfilling state.

[0024] In one embodiment of the present invention, step S2 includes the following steps: S2-1. The health evaluator samples once per second. The sampled data includes: the real-time backlog of the current remote_write memory queue and the success rate of the most recent batch transmission within the current sampling period.

[0025] S2-2. Set up two sliding windows. The fixed length of the circular buffer of each sliding window is equal to the time span T_window of the sliding window. The first sliding window is used for backlog detection, and the second sliding window is used for success rate detection. The sliding window uses a pre-configured circular queue, where new data overwrites the oldest data to achieve window sliding.

[0026] S2-3. When the queue backlog of the number of consecutive sampling periods t required to trigger the backlog detection sliding window is greater than or equal to the backlog anomaly threshold b, it is recorded as 1 backlog anomaly sampling point, and t is less than or equal to T_window.

[0027] S2-4. When the transmission success rate of the number of consecutive sampling periods t required to trigger the past abnormal sampling points in the success rate detection sliding window is less than the success rate abnormal threshold p, it is recorded as 1 success rate abnormal sampling point.

[0028] In one embodiment of the present invention, in step S3, if the success rate of remote_write is less than the success rate abnormal threshold p in the success rate detection sliding window for N consecutive sampling periods, it is determined that the network has serious fluctuations. The health evaluator sends a flow limiting trigger signal to the state controller, and the state controller drives the system to switch from the listening state to the working state.

[0029] In one embodiment of the present invention, the configuration rule in step S4 is as follows: the metrics stored in Prometheus can be divided into raw metrics and metrics generated by Recording Rules. Recording Rules are a feature of Prometheus that allows users to use raw metrics to pre-compute important or costly expressions and save the results as new time series to improve query speed. In Prometheus, the general form of metrics generated by Recording Rules is level:metric:operations (where level represents the aggregation level, metric is the name of the raw metric, and operations represent the list of operations, separated by colons). This invention uses the raw metrics stored in Prometheus as low-priority metrics and the metrics generated by Recording Rules stored in Prometheus as high-priority metrics. When Prometheus submits a sample to the remote_write memory queue, if the sample is a high-priority metric, it is written normally to the remote_write memory queue; if the sample is a low-priority metric, it is serialized and appended to a separate local rate-limited WAL file.

[0030] In one embodiment of the present invention, step S5 includes the following steps: S5-1. When the backlog in the remote_write memory queue is less than the backlog lower limit threshold c or the success rate is greater than or equal to the success rate exception threshold p, and the continuous compliance time t1 from the working state to the backfilling state is reached, the state controller drives the system to switch from the working state to the backfilling state.

[0031] S5-2. The backfill scheduler reads the rate-limited WAL file and resubmits the data to the standard remote_write memory queue. The Thanos server inserts the backfill data into the correct position based on the timestamp and time series identifier.

[0032] In one embodiment of the present invention, after the system switches from the backfill state to the monitoring state in step S6, the empty rate-limiting WAL file is deleted, and the system is restored to the lightweight monitoring mode as in step S2 of S102, in preparation for dealing with the next abnormal event.

[0033] Example 2 A Prometheus remote_write adaptive rate limiting and backfilling device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus; wherein: The memory is used to store computer programs; The processor is configured to execute by running programs stored in the memory: When Prometheus starts, it loads the rate limiting module and performs initialization configuration, which in turn initializes the state machine controller and adds it to the listening state.

[0034] In listening mode, the health evaluator samples once per second and updates the sliding window.

[0035] When severe network fluctuations occur, the state controller drives the system to switch from listening state to working state.

[0036] In the working state, when Prometheus submits samples to the remote_write memory queue, the WAL filter writer processes them according to the configured rules.

[0037] When the backfilling conditions are met, the state controller drives the system to switch from the working state to the backfilling state.

[0038] After the backfill scheduler confirms that all rate-limited WAL file data has been processed, it notifies the state machine controller, which then drives the system to switch from the backfill state to the listening state.

[0039] Example 3 A computer-readable storage medium having a computer program stored thereon, the computer program being implemented when executed by a processor: When Prometheus starts, it loads the rate limiting module and performs initialization configuration, which in turn initializes the state machine controller and adds it to the listening state.

[0040] In listening mode, the health evaluator samples once per second and updates the sliding window.

[0041] When severe network fluctuations occur, the state controller drives the system to switch from listening state to working state.

[0042] In the working state, when Prometheus submits samples to the remote_write memory queue, the WAL filter writer processes them according to the configured rules.

[0043] When the backfilling conditions are met, the state controller drives the system to switch from the working state to the backfilling state.

[0044] After the backfill scheduler confirms that all rate-limited WAL file data has been processed, it notifies the state machine controller, which then drives the system to switch from the backfill state to the listening state.

[0045] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A Prometheus remote_write adaptive rate limiting and backfilling method, characterized in that, include: S1. When Prometheus starts, it loads the rate limiting module and performs initialization configuration, which initializes the state machine controller and adds it to the listening state. S2. In listening mode, the health evaluator samples once per second and updates the sliding window; S3. When severe network fluctuations occur, the state controller drives the system to switch from the listening state to the working state; S4. In the working state, when Prometheus submits a sample to the remote_write memory queue, the WAL filter writer processes it according to the configuration rules; S5. When the backfilling trigger condition is met, the state controller drives the system to switch from the working state to the backfilling state; S6. After the backfill scheduler confirms that all rate-limited WAL file data has been processed, it notifies the state machine controller, which then drives the system to switch from the backfill state to the listening state.

2. The Prometheus remote_write adaptive rate limiting and backfilling method according to claim 1, characterized in that, The method for initializing the state machine controller in step S1 is as follows: After Prometheus starts, it reads the sliding window time span T_window, the number of consecutive sampling cycles t required to trigger the abnormal sampling point, the backlog abnormal threshold b, the success rate abnormal threshold p, the number of consecutive abnormal sampling cycles N to trigger the working state, the backlog lower limit threshold c, and the continuous compliance time t1 from the working state to the backfilling state.

3. The Prometheus remote_write adaptive rate limiting and backfilling method according to claim 2, characterized in that, Step S2 includes the following steps: S2-1. The health evaluator samples once per second. The sampled data includes: the real-time backlog of the current remote_write memory queue and the success rate of the most recent batch transmission within the current sampling period. S2-2. Set up two sliding windows. The fixed length of the circular buffer of the two sliding windows is equal to the time span T_window of the sliding window. The first sliding window is used as the backlog detection sliding window, and the second sliding window is used as the success rate detection sliding window. S2-3. When the queue backlog of the number of consecutive sampling periods t required to trigger the backlog detection sliding window is greater than or equal to the backlog anomaly threshold b, it is recorded as 1 backlog anomaly sampling point, and t is less than or equal to T_window; S2-4. When the transmission success rate of the number of consecutive sampling periods t required to trigger the past abnormal sampling points in the success rate detection sliding window is less than the success rate abnormal threshold p, it is recorded as 1 success rate abnormal sampling point.

4. The Prometheus remote_write adaptive rate limiting and backfilling method according to claim 3, characterized in that: In step S3, if the success rate of remote_write is less than the success rate abnormal threshold p in the success rate detection sliding window for N consecutive sampling periods, it is determined that the network has serious fluctuations. The health evaluator sends a rate limiting trigger signal to the state controller, and the state controller drives the system to switch from the listening state to the working state.

5. The Prometheus remote_write adaptive rate limiting and backfilling method according to claim 1, characterized in that: The configuration rule in step S4 is as follows: the raw metrics stored by Prometheus are used as low-priority metrics, and the metrics generated by the record rules stored by Prometheus are used as high-priority metrics. When Prometheus submits a sample to the remote_write memory queue, if the sample is a high-priority metric, it is written to the remote_write memory queue normally. If the sample is a low-priority metric, it is serialized and appended to a separate local rate-limiting WAL file.

6. The Prometheus remote_write adaptive rate limiting and backfilling method according to claim 2, characterized in that, Step S5 includes the following steps: S5-1. When the backlog in the remote_write memory queue is less than the backlog lower limit threshold c or the success rate is greater than or equal to the success rate exception threshold p, and the continuous compliance time from working state to backfilling state is t1, the state controller drives the system to switch from working state to backfilling state. S5-2. The backfill scheduler reads the rate-limited WAL file and resubmits the data to the standard remote_write memory queue. The Thanos server inserts the backfill data into the correct position based on the timestamp and time series identifier.

7. The Prometheus remote_write adaptive rate limiting and backfilling method according to claim 1, characterized in that: In step S6, after the system switches from the backfill state back to the listening state, it deletes the empty rate-limiting WAL file.

8. A Prometheus remote_write adaptive rate limiting and backfilling device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; wherein: The memory is used to store computer programs; The processor is configured to execute by running programs stored in the memory: When Prometheus starts, it loads the rate limiting module and performs initialization configuration, which initializes the state machine controller and adds it to the listening state. In listening mode, the health evaluator samples once per second and updates the sliding window; When severe network fluctuations occur, the state controller drives the system to switch from listening state to working state. In the working state, when Prometheus submits samples to the remote_write memory queue, the WAL filter writer processes them according to the configuration rules; When the backfilling conditions are met, the state controller drives the system to switch from the working state to the backfilling state. After the backfill scheduler confirms that all rate-limited WAL file data has been processed, it notifies the state machine controller, which then drives the system to switch from the backfill state to the listening state.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is implemented when executed by a processor: When Prometheus starts, it loads the rate limiting module and performs initialization configuration, which initializes the state machine controller and adds it to the listening state. In listening mode, the health evaluator samples once per second and updates the sliding window; When severe network fluctuations occur, the state controller drives the system to switch from listening state to working state. In the working state, when Prometheus submits samples to the remote_write memory queue, the WAL filter writer processes them according to the configuration rules; When the backfilling conditions are met, the state controller drives the system to switch from the working state to the backfilling state. After the backfill scheduler confirms that all rate-limited WAL file data has been processed, it notifies the state machine controller, which then drives the system to switch from the backfill state to the listening state.

Images