A monitoring data optimization storage method of a monitoring system server

Abstract

The application relates to the technical field of optimizing storage, in particular to a monitoring data optimization storage method of a monitoring system server, which comprises the following steps: reading a self-monitoring index flow, a task resource occupation flow, an external source request flow and a policy reference flow of the monitoring system server in a task execution time window of the present storage optimization task; the self-monitoring index flow comprises self-monitoring indexes of each link; calculating state offsets of each link at each sampling moment; obtaining internal disturbance values of each link; quantifying a global internal disturbance degree of the present storage optimization task, and adjusting a processing speed of a subsequent same kind of storage optimization task execution according to the global internal disturbance degree; determining policy credible weights of each link at each sampling moment; and generating a policy input value, and judging whether a corresponding policy action is executed according to the policy input value. The application improves the reliability of monitoring data optimization storage scheduling and the decision accuracy of an automatic operation and maintenance engine.

Description

Technical Field

[0001] This application relates to the field of optimized storage technology, specifically to a method for optimizing the storage of monitoring data on a monitoring system server. Background Technology

[0002] With the increasing scale and automation of information systems, monitoring system servers have become a core infrastructure for ensuring business availability. In actual operation, monitoring system servers need to receive, process, and store massive amounts of monitoring data from other business clusters, while also generating monitoring data reflecting their own operational status. To cope with massive amounts of data, the monitoring system must periodically perform storage optimization tasks such as data compression, downsampling, and cold / hot migration. These tasks consume a large amount of the server's own computing and disk resources, causing temporary pauses in its core links. These pauses are accurately collected by the self-monitoring links and saved to disk.

[0003] Most existing storage optimization methods only trigger optimization tasks according to fixed rules, failing to identify and constrain the resource disturbances caused by the optimization tasks themselves. Since the monitoring system is both the processing entity and the monitored object, the brief load spikes caused by the storage optimization tasks mentioned above, once collected by the self-monitoring link, are very likely to participate in subsequent storage policy adjustments, anomaly judgments, or automated scaling decisions. Downstream automated operation and maintenance engines often lack the ability to distinguish between internal and external traffic, and are very likely to misjudge the internal performance disturbances generated by the system itself when executing optimization tasks as a surge in real external business traffic or sudden node failures. This misjudgment will trigger unnecessary node expansion, false alarms, or new resource scheduling, thus forming a reflexive vicious feedback loop of "executing optimization tasks to occupy resources → self-monitoring indicators spike → false triggering of automated operation and maintenance actions → further depletion of system resources," causing the underlying storage optimization scheduling to fail, leading to continuous congestion or even overall collapse of the core link of the monitoring system. Summary of the Invention

[0004] To address the aforementioned technical issues, a method for optimizing the storage of monitoring data on a monitoring system server is provided to resolve existing problems.

[0005] The solution to the technical problem in this application is to provide a method for optimizing the storage of monitoring data on a monitoring system server, comprising the following steps: Within the task execution window of this storage optimization task, the self-monitoring indicator stream, task resource usage stream, external request stream, and policy reference stream of the monitoring system server are read; the self-monitoring indicator stream includes the self-monitoring indicators of each link; Based on the deviation and volatility of self-monitoring indicators, the state offset of each link at each sampling time is calculated; based on the task resource occupancy flow and external request flow, the cumulative contribution of internal resources and external request pressure relative to the state offset is evaluated to obtain the internal disturbance value of each link; the state offset and the internal disturbance value are aggregated to quantify the global internal disturbance of this storage optimization task, and the processing rate of subsequent similar storage optimization tasks is adjusted accordingly to optimize the underlying storage load and limit excessive occupation of system resources; Based on the task execution window and the link recovery period after the task ends, the disturbance impact window is extracted, the state offset of the link within the disturbance impact window and the value of internal disturbance are analyzed, the degree of interference of the self-monitoring index by the internal task is evaluated, and the policy credibility weight of each link at each sampling time is determined. The policy reference stream is parsed to determine the link targeted by the triggered policy action and the time set it references. Based on the policy trust weight, the self-monitoring indicators of the link within the time set are downweighted and corrected, and a policy input value is generated. Based on this, it is determined whether to execute the corresponding policy action to isolate false triggers caused by internal disturbances and achieve optimized storage of monitoring data.

[0006] Preferably, the calculation of the state offset of each link at each sampling time includes: Within the task execution window, the self-monitoring indicators of each link at all sampling times are input into the dynamic baseline algorithm for baseline fitting, and the predicted values ​​at each sampling time are extracted from the fitted baseline as dynamic baseline values. Extract the residuals of the fitted baseline at each sampling time, calculate the fluctuation of the residuals at all sampling times within the task execution window; calculate the difference between the self-monitoring index of each link at each sampling time and the dynamic baseline value, and use the positive mapping result of the ratio of the fluctuation to the state offset of each link at each sampling time.

[0007] Preferably, obtaining the internal disturbance values ​​of each link includes: Based on the task resource usage flow, calculate the task usage at each sampling time; based on the external request flow, calculate the external request volume at each sampling time; define the maximum value of the task usage at each sampling time and the previous sampling time as the task pre-load at each sampling time; define the maximum value of the external request volume at each sampling time and the previous sampling time as the request pre-load at each sampling time. The sum of the products of the state offsets of each link at all sampling times within the task execution window and the task pre-load is taken as the internal disturbance accumulation of each link; the sum of the products of the state offsets of each link at all sampling times within the task execution window and the request pre-load is taken as the external pressure accumulation of each link; the sum of the internal disturbance accumulation and the external pressure accumulation is calculated, and the proportion of the internal disturbance accumulation in the sum is taken as the internal disturbance value of each link.

[0008] Preferably, the calculation process of the global internal disturbance is as follows: calculate the sum of the state offsets of each link at all sampling times within the task execution window, use it as the cumulative offset of each link, normalize it and use it as the weight, and perform a weighted average of the internal disturbance values ​​of all links to obtain the global internal disturbance of the storage optimization task.

[0009] Preferably, adjusting the processing rate when subsequent similar storage optimization tasks are executed includes: ,in, To improve the processing speed when the next similar storage optimization task is triggered; The preset minimum execution rate, The preset base execution rate is set for this type of storage optimization task. The preset adjustment coefficient, The global internal perturbation for this storage optimization task. This represents the function that takes the maximum value.

[0010] Preferably, the process of obtaining the disturbance impact window is as follows: taking the end time of the storage optimization task as the starting point, for all sampling times after the starting point, the first sampling time that meets the preset judgment condition is selected and defined as the recovery termination time. The task execution window of the storage optimization task and the time period contained between the end time and the recovery termination time are merged and defined as the disturbance impact window.

[0011] Preferably, the preset judgment condition is: the state offset of all links at the same sampling time is not greater than 0 and the task occupancy at that sampling time is not greater than the preset background occupancy baseline value.

[0012] Preferably, determining the policy credibility weight of each link at each sampling time includes: for the disturbance impact window, multiplying the state offset of each link at each sampling time by the value of the internal disturbance of that link as the distortion degree of each link at each sampling time, and negatively mapping it as the policy credibility weight.

[0013] Preferably, determining the link targeted by the triggered strategy action and the time set it references includes: parsing the strategy action triggered in the strategy reference stream, extracting the self-monitoring indicator requested by the strategy action when executing the query and the query time range, and determining the link corresponding to the self-monitoring indicator; within the query time range, extracting all sampling moments falling within the disturbance influence time window to form the time set actually referenced by the strategy action for the link.

[0014] Preferably, the calculation process of the policy input value is as follows: based on the policy credibility weight, the original self-monitoring indicators and dynamic baseline values ​​of the link targeted by the policy action at each sampling time in the time set are weighted and summed as the policy substitution value; the mean of all policy substitution values ​​in the time set referenced by the policy action is taken as the policy input value of the policy action for the link.

[0015] This application has at least the following beneficial effects: This application collects four types of data: monitoring indicator stream, task resource consumption stream, external request stream, and policy reference stream. This enables subsequent analysis to jointly observe the system status from four dimensions: link status, internal resource consumption, external pressure, and policy reference. This provides a complete multi-dimensional data foundation for accurately separating internal interference and judging policy dependencies. The application calculates the state offset of each link at each sampling time, achieving precise quantitative extraction of link performance degradation and highlighting real abnormal peaks hidden in the monitoring time series. This provides a highly objective analytical benchmark for accurately assessing whether the state offset is caused by internal resource congestion. The application also obtains the internal disturbance value of each link and objectively quantifies the link within the same time window. The cumulative proportion of cascading failures occurring alongside internal storage tasks and external business requests mathematically decouples the boundary between "real business overload" and "internal task resource contention," achieving accurate quantification of metric fluctuation attribution. By aggregating the state offsets of each link and their internal disturbance values, the dispersed link-level attribution results are synthesized into a unified task-level metric, namely global internal disturbance. This allows the system to comprehensively assess the overall disturbance caused by a storage optimization task to all monitored links. Using this global internal disturbance as feedback, the system automatically adjusts the processing rate of subsequent similar tasks, curbing the excessive consumption of hardware resources by periodic storage optimization tasks at the source, ensuring stable disk writes at the underlying level, and thus improving storage optimization. A dynamic balance is achieved between efficiency and monitoring link stability. By extracting the disturbance impact window, the timing lag of hardware resource release and queue clearing is fully considered, and the analysis interval is proactively extended to the natural recovery period after the task ends, avoiding the omission of residual stuttering data. Simultaneously, the degree of internal interference is transformed into specific policy credibility weights, objectively quantifying the business authenticity of each monitoring indicator, enabling subsequent corrections to differentiate data with varying degrees of distortion. By parsing the policy reference stream, the links and time sets referenced by the triggered policy actions are accurately determined, ensuring that correction operations are only executed on the data points actually involved in policy decision-making, rather than indiscriminately processing all data, significantly reducing [the impact of the system]. Firstly, it reduces computational overhead. Secondly, based on the policy's trustworthy weights, the original self-monitoring indicators are downweighted and corrected. Data with higher distortion levels has a lower proportion in the policy input values, while data with lower distortion levels has a higher proportion. This ensures that the data actually entering the policy decision-making process truly reflects changes in external monitoring workload while preserving the integrity of the original data. At the same time, since the policy input values ​​have eliminated internal task disturbances, policy judgments such as storage optimization, scaling up / down, and alarms executed based on these values ​​will not be falsely triggered by internal disturbances. Logically, this completely blocks the propagation path of reflexive harmful feedback loops, thereby improving the reliability of monitoring data optimization and storage scheduling and the decision-making accuracy of the automated operation and maintenance engine. Attached Figure Description

[0016] The following description, in conjunction with the accompanying drawings, provides a more detailed explanation of a method for optimizing the storage of monitoring data in a monitoring system server according to this application.

[0017] Figure 1 A flowchart illustrating the steps of a method for optimizing the storage of monitoring data on a monitoring system server, as provided in this application embodiment. Detailed Implementation

[0018] The following description, in conjunction with the accompanying drawings and embodiments, provides a more detailed explanation of the monitoring data optimization storage method for a monitoring system server proposed in this application.

[0019] Please see Figure 1 The diagram illustrates a flowchart of a method for optimizing the storage of monitoring data on a monitoring system server according to an embodiment of this application. The method includes the following steps: Step 1: During the task execution window of this storage optimization task, read the self-monitoring indicator stream, task resource usage stream, external request stream, and policy reference stream of the monitoring system server; the self-monitoring indicator stream includes the self-monitoring indicators of each link. Step 2: Based on the deviation and volatility of the self-monitoring indicators, calculate the state offset of each link at each sampling time; based on the task resource occupancy flow and external request flow, evaluate the cumulative contribution of internal resources and external request pressure relative to the state offset to obtain the internal disturbance value of each link; aggregate the state offset and the internal disturbance value to quantify the global internal disturbance of this storage optimization task. Step 3: Based on the task execution time window and the link recovery period after the task ends, extract the disturbance impact time window, analyze the state offset and internal disturbance value of the link within the disturbance impact time window, evaluate the degree of interference of the self-monitoring indicators by the internal tasks, and thus determine the policy credibility weight of each link at each sampling time; parse the policy reference flow to determine the link targeted by the triggered policy action and the time set it references, and perform weight reduction correction on the self-monitoring indicators of the link within the time set based on the policy credibility weight and generate policy input values; Step 4: Adjust the processing rate of subsequent similar storage optimization tasks based on the global internal disturbance to optimize the underlying storage load and limit excessive occupation of system resources; determine whether to execute the corresponding policy action based on the policy input value to achieve optimized storage of monitoring data.

[0020] In step 1, the specific process is as follows: This embodiment takes a cloud-native microservice monitoring center as an example. In this scenario, the monitoring system server continuously receives monitoring metrics generated by container instances, service interfaces, database connection pools, middleware components, and node running status. During business releases, service scaling up or down, or anomaly investigations, the amount of metrics written, query requests, and alarm rule calculations may increase simultaneously. At the same time, the monitoring system server also needs to perform storage optimization tasks such as historical time-series block compression, expired data cleanup, index merging, and cold and hot data migration.

[0021] In this scenario, external business pressure and internal optimization task disturbances will overlap on the same monitoring link. If attribution splitting and distortion reduction are not performed, the system will mistakenly regard the fluctuations caused by internal optimization tasks as real external pressure, thereby triggering new storage optimization, expansion, migration or alarm strategies, forming a reflexive harmful feedback loop.

[0022] Based on the above analysis, data acquisition probes are deployed inside the monitoring system server to obtain the corresponding data streams. Therefore, in step 1, the execution process is as follows: The data acquisition probe calls the internal indicator interface of the monitoring system server to read the self-monitoring indicator stream. The self-monitoring indicator stream includes at least the self-monitoring indicators of each link, specifically: the acquisition latency of the acquisition link, the write queue length and write confirmation time of the write link, the query response time of the query link, the alarm calculation time of the alarm calculation link, and the storage engine I / O wait status of the storage engine processing link. The self-monitoring indicator stream can reflect whether the core capabilities of the monitoring system server itself have declined due to storage optimization tasks. Parse the task execution logs at the system's underlying level and extract the task resource usage stream. The task resource usage stream includes at least the task identifier, task type, task start time, task end time, data block identifier processed by the task, CPU processing time, disk read / write volume, and storage engine background thread usage status. The system monitors inbound network traffic and tracks external request flows, including the number of collection batches, write points, query requests, and alarm rule triggers. The external request flow is used to distinguish whether link fluctuations are caused by external monitoring workloads such as business releases, service scaling, and troubleshooting queries, or by internal storage optimization tasks consuming resources. By intercepting policy query statements within the system, the policy reference stream is extracted, including the name of the referenced self-monitoring metric, the reference time, the purpose of the reference, the triggered policy action, and the target of the policy action. This policy reference stream is used to determine whether the disturbed self-monitoring data will continue to affect subsequent storage optimization policies, alarm policies, or automated operation and maintenance policies. The storage optimization tasks include downsampling, compression, data migration, data cleaning, and index maintenance.

[0023] The task execution window is defined with the start and end times of the storage optimization task as boundaries. Data is captured according to a preset sampling period. The self-monitoring indicator stream, task resource usage stream, external request stream, and policy reference stream whose timestamps fall within the time window are packaged and associated, and stored in the same task analysis unit. The sampling period is set to 15 seconds, that is, the self-monitoring indicator stream, task resource usage stream, external request stream, and policy reference stream are captured every 15 seconds. As for other implementation methods, the implementer can set them according to the actual situation.

[0024] Furthermore, the self-monitoring metric stream will naturally fluctuate with business releases, service scaling, troubleshooting queries, and alarm rule triggers. If the raw data is used directly to judge disturbances, the fluctuations in the normal operation of the monitoring system server will be misjudged as being affected by internal optimization tasks. Therefore, in step 2, the specific execution process is as follows: Step 201: Within the task execution window, input the self-monitoring indicators of each link at all sampling times into the dynamic baseline algorithm for baseline fitting, and extract the predicted values ​​at each sampling time from the fitted baseline as dynamic baseline values. Specifically, a quantile regression model is used for fitting. The self-monitoring indicator stream contains the self-monitoring indicators of each link, which are input into the quantile regression model for fitting and prediction. The quantile regression model performs regression calculations by setting a preset quantile, outputting a prediction curve that shields against abnormal spike interference and reflects the normal operating status of the link. This prediction curve is used as the dynamic baseline for each link, and the predicted value of the dynamic baseline at each sampling time is used as the dynamic baseline value for each link at each sampling time. The preset quantile is set to 50%, but as another implementation method, the implementer can set it according to the actual situation. The quantile regression model is a well-known technology and will not be described in detail here.

[0025] It should be noted that the dynamic baseline value is used to represent a reference level at which the link can be considered to be in a normal operating state.

[0026] Step 202: Extract the residuals of the fitted baseline at each sampling time, calculate the fluctuation of the residuals at all sampling times within the task execution window; calculate the difference between the self-monitoring index of each link at each sampling time and the dynamic baseline value, and use the positive mapping result of the ratio of the fluctuation as the state offset of each link at each sampling time. Specifically, the process of measuring the degree of fluctuation is as follows: obtain the upper quartile and lower quartile of the residuals at all sampling times within the task execution window, and take the absolute value of the difference between the two as the degree of fluctuation; where residuals are a well-known technique, they will not be described in detail here; secondly, the process of positive mapping is as follows: select the maximum value between the value 0 and the ratio as the result of positive mapping. When calculating the ratio, in order to avoid the denominator being 0, a parameter tuning factor is added to the denominator. This parameter tuning factor is set to 0.001. As another implementation method, the implementer can set it according to the actual situation.

[0027] It should be noted that when the difference is less than or equal to 0, it indicates that the corresponding link is not currently experiencing performance degradation or congestion, the actual self-monitoring indicators are not exceeding the normal prediction level, and the system is in a healthy operating state free from interference. Conversely, it indicates that the actual self-monitoring indicators exceed the normal prediction level, reflecting that the link is experiencing performance degradation or congestion, and the system is being negatively impacted by internal optimization task resource constraints or external sudden requests. The smaller the fluctuation, the more stable the self-monitoring indicator is under normal conditions, with fewer inherent random indicator fluctuations. The larger the state offset, the more severe the performance degradation of the link is, exceeding the tolerance range of normal fluctuations, and the system is facing significant external business pressure or internal task constraints.

[0028] Furthermore, in cloud-native microservice monitoring center scenarios, business deployments, service expansion, interface anomaly investigations, and concentrated queries by operations and maintenance personnel can all lead to an increase in external request volume, causing deviations in the data collection, writing, querying, or alarm calculation chain. Failure to differentiate between internal optimization tasks and external request pressure can easily result in control strategies that do not meet actual optimization needs.

[0029] Based on this, the state offset, task resource usage flow, and external request flow are placed within the same task execution window for decomposition of their associated relationships. If a certain state offset mainly occurs with an increase in CPU, disk I / O, and background thread usage of the storage optimization task, then the offset is more likely to be caused by the internal optimization task. If a certain state offset mainly occurs with an increase in the number of collection batches, write points, query requests, or alarm rule triggers, then the offset is more likely to be caused by the actual external workload.

[0030] Step 203: Based on the task resource usage flow, calculate the task usage at each sampling time; based on the external request flow, calculate the external request volume at each sampling time; define the maximum value of the task usage at each sampling time and the previous sampling time as the task pre-load at each sampling time; define the maximum value of the external request volume at each sampling time and the previous sampling time as the request pre-load at each sampling time. The calculation process for task occupancy is as follows: the maximum value normalization method is used to normalize the CPU processing time, disk read / write volume, and storage engine background thread occupancy in the task resource occupancy stream. The average value of the normalized CPU processing time, disk read / write volume, and storage engine background thread occupancy at the same sampling time is taken as the task occupancy. The maximum value normalization method is a well-known technique and will not be elaborated here.

[0031] The calculation process of external request volume is as follows: The maximum value normalization method is used to normalize the number of collection batches, the number of write points, the number of query requests and the number of alarm rule triggers in the external request stream respectively. The average of the normalized number of collection batches, the number of write points, the number of query requests and the number of alarm rule triggers at the same sampling time is taken as the external request volume. It should be noted that when the sampling time is the first moment of the task execution window, its task pre-load and request pre-load are the task occupancy and external request volume of the first moment.

[0032] It should be noted that the task usage reflects the extent to which the storage optimization task consumes the shared resources of the monitoring system server at that sampling moment; the external request volume reflects the pressure exerted on the monitoring system server by external monitoring work at that sampling moment; after the storage optimization task consumes resources, link latency or queue accumulation may only manifest at the next sampling moment; if only the same sampling moment is compared, this short-term transmission will be missed.

[0033] Step 204: The sum of the products of the state offset of each link at all sampling times within the task execution window and the task pre-set amount is taken as the internal disturbance accumulation of each link; the sum of the products of the state offset of each link at all sampling times within the task execution window and the request pre-set amount is taken as the external pressure accumulation of each link; the sum of the internal disturbance accumulation and the external pressure accumulation is calculated, and the proportion of the internal disturbance accumulation in the sum is taken as the internal disturbance value of each link. Specifically, the formula for calculating the internal disturbance value is as follows: in, For link The value affected by internal disturbances, For link The cumulative amount of internal disturbances For link Accumulated external pressure; Specifically, when the sum of the accumulated internal disturbance and the accumulated external pressure is 0, the internal disturbance value of the link is defined as 0.

[0034] It should be noted that a larger accumulation of internal disturbances indicates a high degree of temporal overlap between the link's state offset and the high resource consumption of internal storage tasks, reflecting that internal storage tasks are severely crowding out system resources and are a direct factor causing link congestion; a larger accumulation of external pressure indicates a high degree of temporal overlap between the link's state offset and high-frequency external monitoring requests, reflecting that the performance degradation of the link is directly caused by a sudden increase in actual external business load; when When, it indicates a link The state shifts are more concentrated with increased external request pressure, and therefore the internal disturbance values ​​are correspondingly reduced. This link fluctuation should not be primarily attributed to storage optimization tasks; when When, it indicates a link The state shifts are more concentrated with the increase in internal storage optimization task resource usage, and therefore the internal disturbance value increases accordingly. The larger the value, the more dominant the internal disturbance accumulation is in the total accumulation, reflecting that the state shift of this link is caused by internal disturbances caused by the system's own optimization tasks.

[0035] Step 205: Calculate the sum of the state offsets of each link at all sampling times within the task execution window, and use it as the cumulative offset of each link. After normalization, use it as the weight to perform a weighted average of the internal disturbance values ​​of all links, which is used as the global internal disturbance of the storage optimization task. The normalization process is as follows: calculate the sum of the cumulative offsets of all links, and use the ratio of the cumulative offset of each link to the sum as the normalized result; on this basis, when the sum of the cumulative offsets of all links is 0, skip the normalization process, assign the weights corresponding to all links to 0, and output the global internal perturbation as 0.

[0036] It should be noted that, since the actual offset of different links varies, links with larger actual offsets should have a higher impact on the global internal disturbance. Simply averaging all links would cause links with slight offsets to have the same impact as those with severe offsets, weakening the accuracy of attribution. Therefore, by using the cumulative offset as the weight to calculate the weighted average of the internal disturbance values ​​of all links, we can ensure that links with more significant disturbances dominate the global evaluation, thereby improving the accuracy of the global internal disturbance assessment. Secondly, the larger the global internal disturbance, the more the state offset generated by each link in the system during the execution of the storage optimization task is due to the task occupying hardware resources. This reflects that the performance impact of the storage optimization task on the underlying layer of the entire monitoring system is extremely serious, and its operating intensity has seriously threatened the stability of the core monitoring links. It is necessary to enforce resource throttling and execution degradation in subsequent scheduling.

[0037] During storage optimization tasks, the monitoring system inevitably experiences state shifts such as increased latency or queue congestion due to heavy internal resource consumption. These state shifts are accurately collected and recorded by the monitoring links. If raw monitoring data with abnormal peaks is directly input into downstream automatic scaling or alarm policies without differentiation, the downstream policies will misinterpret it as sudden external business pressure, thus triggering invalid machine scaling or false alarms. Therefore, it is necessary to assess the proportion of data anomalies attributable to internal task interference to calculate the degree of distortion, thereby quantifying and downweighting the monitoring data affected by internal disturbances. While preserving the integrity of the underlying original monitoring records for backtracking, the data output to downstream policies is smoothed and corrected to remove internal influences, thereby fundamentally preventing the false triggering of subsequent automated operation and maintenance actions. In step 3, the specific process is as follows: Step 301: Taking the end time of the storage optimization task as the starting point, for all sampling times after the starting point, select the first sampling time that meets the preset judgment condition and define it as the recovery termination time. Combine the task execution time window of the storage optimization task and the time period between the end time and the recovery termination time and define it as the disturbance impact time window. The preset judgment conditions are: the state offset of all links at the sampling time is not greater than 0 and the task occupancy at the sampling time is not greater than the preset background occupancy baseline value. Specifically, the process of obtaining the background occupancy baseline value is as follows: extract a continuous no-task running time window of a preset duration before the storage optimization task starts; for each sampling moment within this time window, extract the general basic hardware occupancy of the monitoring system server (including general CPU utilization and disk I / O utilization), and perform the same maximum value normalization process to replace the acquisition of task occupancy; then, calculate the average of the above normalized values ​​for all sampling moments within this time window, which is defined as the background occupancy baseline value. The duration of the no-task running time window is set to 2 hours. As another implementation method, the implementer can set it according to the actual situation.

[0038] It should be noted that if there is no sampling time that meets the preset judgment conditions before the start of the next storage optimization task, the start time of the next storage optimization task will be taken as the recovery termination time to avoid the aliasing effect of different tasks.

[0039] Step 302: For the disturbance impact window, the product of the state offset of each link at each sampling time and the internal disturbance value of the link is used as the distortion degree of each link at each sampling time. This product is then negatively mapped and used as the policy confidence weight. The specific process of negative mapping is as follows: the reciprocal of the distortion degree is used as the policy credibility weight. When calculating the reciprocal, in order to avoid the denominator being 0, a parameter tuning factor is added to the denominator. This parameter tuning factor is set to 1. As another implementation method, the implementer can set it according to the actual situation.

[0040] It should be noted that the greater the degree of distortion, the more significant the link state offset at that sampling moment. This offset mainly occurs due to the resource consumption of internal storage optimization tasks rather than being caused by the actual external monitoring workload. This reflects that although the self-monitoring indicators at that sampling moment truthfully record the system state, their cause is not external business pressure, but rather an illusion caused by internal system resource consumption. Direct use of such indicators will induce false alarms and ineffective expansion. The greater the policy credibility weight, the less significant the interference from internal tasks at that sampling moment. This reflects that the self-monitoring indicators have extremely high authenticity and credibility. When inputting them into subsequent alarm engines or expansion policies, their original values ​​should be retained to allow them to trigger downstream actions normally. Conversely, a lower weight indicates that the self-monitoring indicators at that sampling moment are severely interfered with by internal tasks, reflecting a lack of business authenticity. When inputting them into downstream policies, they must be greatly suppressed and downgraded with low weights to force them back to the dynamic baseline level, thereby preventing erroneous automated operation and maintenance actions caused by internal task delays.

[0041] Step 303: parse the policy action triggered in the policy reference stream, extract the self-monitoring indicator and query time range requested by the policy action when executing the query, and determine the link corresponding to the self-monitoring indicator; within the query time range, extract all sampling moments that fall within the disturbance impact time window to form the time set of the actual reference of the policy action for the link. Step 304: Based on the policy credibility weight, the original self-monitoring indicators and dynamic baseline values ​​at each sampling time within the time set are weighted and summed to serve as the policy substitution value; the mean of all policy substitution values ​​within the time set referenced by the policy action is used as the policy input value for the policy action for the link. Specifically: in, For link Within the time set The strategy substitute value at the sampling time. For the first The original self-monitoring indicators at the sampling time, For link In the The dynamic baseline value at the sampling time. For link In the The policy credibility weight at the sampling time; It should be noted that, This indicates the portion of the original self-monitoring metrics that is retained for strategy judgment. This indicates the portion supplemented by the dynamic baseline value, when At that time, the original self-monitoring indicators were used entirely; when As the value decreases, the proportion of the dynamic baseline increases, and the strategy substitution value gradually approaches the dynamic baseline; the higher the degree of distortion, The smaller the value, the smaller the perturbation offset that enters the subsequent policy calculation, thereby reducing the impact of internal perturbation data on policy judgment.

[0042] This embodiment eliminates the negative impact of reflexive internal interference by using two parallel control links, addressing the issues from two dimensions: source resource supply and downstream strategy isolation. The specific analysis basis and implementation logic are as follows: Since the global internal disturbance objectively quantifies the degree of disruption to the normal operation of the link caused by the storage optimization task, the execution rate of the storage optimization task is dynamically adjusted based on the global internal disturbance. The higher the global internal disturbance, the more severe the internal hardware resource crowding is. At this time, by reducing the task execution rate, the amount of data that the task is allowed to process per unit time is forcibly reduced. This action directly deprives the internal destructive task of its resource acquisition ability from the source, forcing it to give up CPU and disk I / O for the core business link, thereby preventing the continuous deterioration of the internal disturbance of the system. At the same time, setting a minimum guaranteed threshold for the rate limiting can ensure that the storage optimization task is degraded but not deadlocked, while taking into account the basic progress of the background storage optimization.

[0043] Secondly, because the original self-monitoring indicators contain a large number of abnormal peaks caused by internal resource congestion within the disturbance impact window, these abnormal peaks are very likely to break through fixed alarm or scaling thresholds, causing the system to experience "false faults, real alarms" or trigger unnecessary machine expansion. By sending the policy input value obtained through point-by-point weight reduction correction to the alarm engine or scaling controller, the final policy input value perfectly avoids the trigger threshold of the downstream engine in terms of numerical form, fundamentally blocking the issuance of invalid scaling and false alarm commands. Therefore, in step 4, the specific process is as follows: Step 401: Based on the global internal perturbation of this storage optimization task, adaptively adjust the processing rate of this type of storage optimization task in the next execution, specifically as follows: in, To improve the processing speed when the next similar storage optimization task is triggered; The preset minimum execution rate, The preset base execution rate is set for this type of storage optimization task. The preset adjustment coefficient, The global internal perturbation for this storage optimization task; This represents the function that takes the maximum value. Specifically, the preset basic execution rate setting process is as follows: determined based on the underlying hardware performance limits of the monitoring system server, it is usually set to 20% to 30% of the maximum disk I / O throughput or the maximum number of processing threads. For example, in a conventional solid-state drive environment, it is set to 50MB / s, representing the normal maximum processing capacity of the task when the system is idle. The preset minimum execution rate is used to prevent storage optimization tasks from being starved due to excessive rate limiting, resulting in historical data accumulation and disk overflow. This value is set to 5% to 10% of the basic execution rate. In this embodiment, it is set to 5MB / s to ensure the most basic progress of the underlying data cleaning or compression tasks. The sensitivity to penalties for internal disturbances is used to control the value range of [value range missing]. In this embodiment, the value is 0.8. In other implementation methods, the implementer can set the value according to the actual situation.

[0044] It should be noted that the greater the global internal disturbance, the lower the actual processing speed. This reflects that when the performance degradation caused by the current storage optimization task is more severe, the system will impose a more severe rate limiting penalty when triggering a similar task in the future. By reducing the amount of monitoring data that can be processed per unit time, the system will forcibly deprive the system of its resource quota, reduce the resource consumption of the monitoring system server by the storage optimization task, and effectively prevent the system from crashing due to repeated high-frequency and high-consumption storage optimization tasks from the source.

[0045] Step 402: Based on the policy input value, determine whether to execute the corresponding policy action to isolate false triggers caused by internal disturbances and optimize the storage of monitoring data. Specifically: For policy actions that rely on a single link, the policy input value of that link is compared with a preset trigger threshold, and the policy action is executed based on the comparison result. For policy actions that rely on multiple links, the policy input values ​​of multiple links are substituted into the preset logical combination rules or weighted calculation formula of the policy action to obtain a comprehensive judgment result. The comprehensive judgment result is then compared with the corresponding multi-index trigger threshold, and a decision is made on whether to execute the policy action based on the comparison result. Specifically, preset trigger thresholds and multi-metric trigger thresholds are pre-defined and configured in the downstream automated operation and maintenance engine by operations and maintenance personnel according to the system's service level agreement (SLA) requirements or hardware capacity limits. Specific setting criteria and engineering experience values ​​are as follows: for the response time of the query link, the preset trigger threshold can be set to 500ms according to the SLA requirements, that is, when the policy input value exceeds 500ms, a delay alarm is triggered; for the queue length of the write link, the preset trigger threshold can be set to 10000 according to the memory buffer capacity, that is, when the number of queues exceeds 10000, horizontal scaling of the database node (scaling strategy) is triggered; implementers can flexibly adjust these thresholds according to the tolerance of the specific monitored business.

[0046] It should be noted that the policy input values ​​sent to the downstream policy actions have been forcibly leveled to the normal dynamic baseline level by proportionally adjusting the severely distorted abnormal peaks through policy credibility weights. This operation ensures that the data used for final judgment effectively avoids the trigger threshold of the downstream engine in terms of numerical form, fundamentally preventing invalid machine expansion and false alarms caused by internal storage optimization task lag.

Claims

1. A method for optimizing the storage of monitoring data on a monitoring system server, characterized in that, The method includes the following steps: Within the task execution window of this storage optimization task, the self-monitoring indicator stream, task resource usage stream, external request stream, and policy reference stream of the monitoring system server are read; the self-monitoring indicator stream includes the self-monitoring indicators of each link; Based on the deviation and volatility of self-monitoring indicators, the state offset of each link at each sampling time is calculated; based on the task resource occupancy flow and external request flow, the cumulative contribution of internal resources and external request pressure relative to the state offset is evaluated to obtain the internal disturbance value of each link; the state offset and the internal disturbance value are aggregated to quantify the global internal disturbance of this storage optimization task, and the processing rate of subsequent similar storage optimization tasks is adjusted accordingly to optimize the underlying storage load and limit excessive occupation of system resources; Based on the task execution window and the link recovery period after the task ends, the disturbance impact window is extracted, the state offset of the link within the disturbance impact window and the value of internal disturbance are analyzed, the degree of interference of the self-monitoring index by the internal task is evaluated, and the policy reliability weight of each link at each sampling time is determined. The policy reference stream is parsed to determine the link targeted by the triggered policy action and the time set it references. Based on the policy trust weight, the self-monitoring indicators of the link within the time set are downweighted and corrected, and a policy input value is generated. Based on this, it is determined whether to execute the corresponding policy action to isolate false triggers caused by internal disturbances and achieve optimized storage of monitoring data.

2. The method for optimizing and storing monitoring data on a monitoring system server as described in claim 1, characterized in that, The calculation of the state offset of each link at each sampling time includes: Within the task execution window, the self-monitoring indicators of each link at all sampling times are input into the dynamic baseline algorithm for baseline fitting, and the predicted values ​​at each sampling time are extracted from the fitted baseline as dynamic baseline values. Extract the residuals of the fitted baseline at each sampling time, calculate the fluctuation of the residuals at all sampling times within the task execution window; calculate the difference between the self-monitoring index of each link at each sampling time and the dynamic baseline value, and use the positive mapping result of the ratio of the fluctuation to the state offset of each link at each sampling time.

3. The method for optimizing the storage of monitoring data on a monitoring system server as described in claim 1, characterized in that, The method of obtaining the internal disturbance values ​​of each link includes: Based on the task resource usage flow, calculate the task usage at each sampling time; based on the external request flow, calculate the external request volume at each sampling time; define the maximum value of the task usage at each sampling time and the previous sampling time as the task pre-load at each sampling time; define the maximum value of the external request volume at each sampling time and the previous sampling time as the request pre-load at each sampling time. The sum of the products of the state offsets of each link at all sampling times within the task execution window and the task pre-load is taken as the internal disturbance accumulation of each link; the sum of the products of the state offsets of each link at all sampling times within the task execution window and the request pre-load is taken as the external pressure accumulation of each link; the sum of the internal disturbance accumulation and the external pressure accumulation is calculated, and the proportion of the internal disturbance accumulation in the sum is taken as the internal disturbance value of each link.

4. The method for optimizing and storing monitoring data on a monitoring system server as described in claim 1, characterized in that, The calculation process of the global internal disturbance is as follows: calculate the sum of the state offsets of each link at all sampling times within the task execution window, use it as the cumulative offset of each link, normalize it and use it as the weight, and then perform a weighted average of the internal disturbance values ​​of all links to obtain the global internal disturbance of the storage optimization task.

5. The method for optimizing and storing monitoring data on a monitoring system server as described in claim 1, characterized in that, The adjustment of the processing rate when subsequent similar storage optimization tasks are executed includes: ,in, To improve the processing speed when the next similar storage optimization task is triggered; The preset minimum execution rate, The preset base execution rate is set for this type of storage optimization task. The preset adjustment coefficient, The global internal perturbation for this storage optimization task. This represents the function that takes the maximum value.

6. The method for optimizing the storage of monitoring data on a monitoring system server as described in claim 3, characterized in that, The process of obtaining the disturbance impact window is as follows: taking the end time of the storage optimization task as the starting point, for all sampling times after the starting point, the first sampling time that meets the preset judgment condition is selected and defined as the recovery termination time. The task execution window of the storage optimization task and the time period contained between the end time and the recovery termination time are merged and defined as the disturbance impact window.

7. The method for optimizing and storing monitoring data on a monitoring system server as described in claim 6, characterized in that, The preset judgment condition is: the state offset of all links at the same sampling time is not greater than 0 and the task occupancy at that sampling time is not greater than the preset background occupancy baseline value.

8. The method for optimizing the storage of monitoring data on a monitoring system server as described in claim 1, characterized in that, The determination of the policy credibility weight of each link at each sampling time includes: for the disturbance impact window, multiplying the state offset of each link at each sampling time by the value of the internal disturbance of the link as the distortion degree of each link at each sampling time, and negatively mapping it as the policy credibility weight.

9. The method for optimizing and storing monitoring data on a monitoring system server as described in claim 2, characterized in that, The determination of the link targeted by the triggered strategy action and the time set referenced by it includes: parsing the strategy action triggered in the strategy reference stream, extracting the self-monitoring indicator requested by the strategy action when executing the query and the query time range, and determining the link corresponding to the self-monitoring indicator; within the query time range, extracting all sampling moments falling within the disturbance influence time window to form the time set actually referenced by the strategy action for the link.

10. The method for optimizing the storage of monitoring data on a monitoring system server as described in claim 9, characterized in that, The calculation process of the policy input value is as follows: based on the policy credibility weight, the original self-monitoring indicators and dynamic baseline values ​​of the link targeted by the policy action at each sampling time in the time set are weighted and summed as the policy substitution value; the mean of all policy substitution values ​​in the time set referenced by the policy action is taken as the policy input value of the policy action for the link.

Images