A database monitoring index recommendation method and system based on association recommendation
By calculating the frequency of co-occurrence of anomalies and pruning similarity among monitoring indicators, and combining this with a random walk algorithm, the redundancy problem in the correlation analysis of indicators in database monitoring is solved, and efficient fault location and diagnosis are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING XINSHU TECH CO LTD
- Filing Date
- 2024-10-16
- Publication Date
- 2026-06-09
Smart Images
Figure CN119396649B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a method and system for recommending database monitoring metrics based on association recommendation, belonging to the field of database monitoring. Background Technology
[0002] In production environments, system administrators face significant challenges in effectively managing numerous operational objects, including key performance indicators (KPIs), servers, and basic physical infrastructure such as common and custom components. These operational objects provide crucial support for system availability, but the associated monitoring metrics grow exponentially, often reaching hundreds or even thousands. As system observability increases, the number of monitoring metrics also grows, placing a heavy burden of observation and analysis tasks on system administrators.
[0003] Each system operation and maintenance object exposes a large number of monitoring metrics, covering various aspects of the system, from key performance indicators of application service capabilities to performance parameters of the underlying physical infrastructure. This requires system administrators to focus on multiple dimensions simultaneously to monitor the real-time status of the system. However, the existence of a large number of monitoring metrics makes it difficult for system administrators to effectively process this information, thus affecting the efficiency of fault detection and diagnosis.
[0004] In monitoring systems, correlation recommendations between metrics can help system administrators gain a more comprehensive understanding of the system's operational status, providing targeted information to facilitate faster problem detection and troubleshooting. Existing methods include:
[0005] (1) Automatic association rules: Using machine learning algorithms and data mining techniques, historical monitoring data can be analyzed to automatically identify association rules between different indicators. However, for large-scale systems, a large number of rules may be generated, some of which may be redundant.
[0006] (2) Expert system and rule engine: Establish association rules based on domain expertise, and manually adjust and optimize by adding rules. However, over-reliance on domain expertise may fail to capture unknown associations.
[0007] (3) Model-based correlation analysis: Establish statistical models to identify correlations between indicators. However, for nonlinear and complex relationships, establishing and maintaining models may require a large amount of computational resources.
[0008] In monitoring systems, single-metric alerting strategies are typically configured for each specific metric to detect whether it exceeds a predetermined threshold range, triggering an alert upon an anomaly. This approach can quickly respond to a specific problem, but in-depth investigation and root cause analysis require analyzing the co-occurrence frequency of anomalies to reveal patterns of simultaneous anomalies in multiple metrics and determine their correlations. Therefore, it necessitates combining analysis with other related metrics. However, in database operations and maintenance, there are often numerous monitoring metrics with weak correlations, leading to inefficient analysis. The system needs to address how to merge redundant metrics by analyzing their correlations. Furthermore, given a large number of monitoring metrics, how to filter out key metrics highly relevant to the problem addresses the issues of low accuracy and slow speed in problem localization. Attached Figure Description
[0009] Figure 1 This is a system flowchart.
[0010] Figure 2 This is a flowchart of the iterative calculation process. Summary of the Invention
[0011] To address the problems of existing methods, this application proposes a database monitoring indicator recommendation method based on association recommendation. In the indicator relationship construction stage, the correlation between abnormal events in the system is detected by calculating the co-occurrence frequency of anomalies among monitoring indicators, and redundant recommendations are avoided by combining curve similarity pruning techniques, generating a relationship graph of monitoring indicators. In the key indicator recommendation stage, a random walk algorithm is used to analyze the above relationship graph to determine the recommendation priority and ranking of monitoring indicators, ensuring effective identification and recommendation of key monitoring indicators. The specific steps are as follows:
[0012] (1) Obtain multi-indicator data, collect data related to monitoring, clean and preprocess the data, compare the monitoring indicators in the same time period when an anomaly occurs, and obtain multi-indicator data in that time period.
[0013] (2) Constructing indicator relationships: Calculate the frequency of abnormal co-occurrence between each pair of monitoring indicators.
[0014] (3) Recommend key indicators: Determine the priority and ranking of the monitoring indicators, and recommend key monitoring indicators.
[0015] Furthermore, when constructing the relationship between indicators, the correlation coefficient is calculated. Where the abnormal event is i, and the abnormal event is j. Let be the mean of the first-order difference sequence of event i at time m. Let be the mean of the first-order difference sequence of event j at time m. Let i be the value corresponding to event i at time m. Let be the value of time j at time m, p be the rate at which the control weight decays, n be the total number of times, and sim(i,j) be the similarity between i and j.
[0016] Furthermore, when recommending key indicators, the specific steps are as follows:
[0017] 3.1 Determine the objective function F(x);
[0018] 3.2 Determine the parameters before optimization: number of iterations N, initial step size λ, initial value of decision variable x0, control precision ξ, ξ>0;
[0019] 3.3 Determine whether the set number of iterations has been reached;
[0020] 3.4 If k = 1, perform the first iteration, randomly generating a d-dimensional vector μ = (μ1, μ2, ..., μ) between (0, 1). d ),calculate Let x1 = x + λμ′ to complete the first iteration, where x is a vector consisting of multiple decision variables.
[0021] 3.5 Calculate the objective function and perform a loop operation based on the judgment conditions.
[0022] 3.6 If the number of iterations reaches the set value but the optimal solution is still not found, output the current value, i.e., λ < ξ, and the algorithm terminates; otherwise, continue iterating.
[0023] Based on the above method, this application proposes a database monitoring metric recommendation system based on association recommendation, which includes:
[0024] (1) Multi-indicator data acquisition module: This module collects data related to monitoring, cleans and preprocesses the data, and compares the monitoring indicators in the same time period when an anomaly occurs to obtain multi-indicator data in that time period.
[0025] (2) Indicator Relationship Construction Module: This module calculates the frequency of abnormal co-occurrence between any two monitoring indicators.
[0026] (3) Key indicator recommendation module: This module determines the recommendation priority and ranking of monitoring indicators and recommends key monitoring indicators.
[0027] Furthermore, in the indicator relationship construction module, the correlation coefficient is calculated. Where the abnormal event is i, and the abnormal event is j. Let be the mean of the first-order difference sequence of event i at time m. Let be the mean of the first-order difference sequence of event j at time m. Let i be the value corresponding to event i at time m. Let be the value of time j at time m, p be the rate at which the control weight decays, n be the total number of times, and sim(i,j) be the similarity between i and j.
[0028] Furthermore, the key indicator recommendation module employs the following specific steps:
[0029] 3.1 Determine the objective function F(x);
[0030] 3.2 Determine the parameters before optimization: number of iterations N, initial step size λ, initial value of decision variable x0, control precision ξ, ξ>0;
[0031] 3.3 Determine whether the set number of iterations has been reached;
[0032] 3.4 If k = 1, perform the first iteration, randomly generating a d-dimensional vector μ = (μ1, μ2, ..., μ) between (0, 1). d ),calculate Let x1 = x + λμ′ to complete the first iteration, where x is a vector consisting of multiple decision variables.
[0033] 3.5 Calculate the objective function and perform a loop operation based on the judgment conditions.
[0034] 3.6 If the number of iterations reaches the set value but the optimal solution is still not found, output the current value, i.e., λ < ξ, and the algorithm terminates; otherwise, continue iterating.
[0035] After multiple rounds of random walk iterations, the weight of each node is updated, and the monitoring metrics are sorted according to the final weight values. Metrics with higher weights rank higher in the recommendation ranking, and the sorted monitoring metrics are output as the recommendation ranking results. This step dynamically adjusts the weights using the node's neighbor information, captures the hierarchical relationship between different monitoring metrics, and combines it with the relationship graph of database monitoring metrics, making the recommendation results more reflective of the true state of the database and helping operations and maintenance personnel quickly identify the metrics that need to be focused on.
[0036] This invention merges redundant monitoring indicators by calculating the co-occurrence frequency and correlation analysis between pairs of indicators, obtaining a relationship graph between the indicators. A random walk algorithm is then used to output a recommended ranking. The ranked monitoring indicators are used as the recommended ranking results to help operations and maintenance personnel quickly locate problems, reduce their workload, and help pinpoint the root cause. This invention comprehensively considers the relationships between multiple monitoring indicators through multi-indicator data analysis, moving beyond a focus on a single indicator. This facilitates a comprehensive analysis of the overall system status and improves the global perspective for problem discovery and troubleshooting. Simultaneously, anomaly co-occurrence frequency analysis is introduced. By statistically analyzing the frequency of simultaneous occurrence of abnormal events, co-occurrence patterns between anomalies are revealed, helping to determine the correlation between database indicators. To filter out indicators with weak relationships and capture recent trends and dynamic changes, a time-weighted factor is introduced into curve similarity pruning, and the weighting factor is dynamically adjusted to adapt to the changing trends over time. Detailed Implementation
[0037] To improve the efficiency of troubleshooting, this invention proposes a database monitoring indicator recommendation method based on association recommendation. It introduces an association indicator recommendation function, which recommends potential association indicators to system administrators by analyzing the relationship between multiple indicators, thereby assisting them in conducting more targeted manual analysis when locating root causes.
[0038] This invention consists of two main parts: indicator relationship construction and key indicator recommendation. First, the indicator relationship construction stage calculates the co-occurrence frequency of anomalies between each pair of monitoring indicators to detect the correlation between abnormal events in the system. Second, this stage introduces curve similarity pruning to avoid redundant recommendations of associated indicators and ultimately establishes a relationship graph between monitoring indicators. The key indicator recommendation stage uses a random walk algorithm to conduct in-depth analysis of the indicator relationship graph obtained in the previous step to determine the recommended ranking of monitoring indicators.
[0039] See the main steps Figure 1 Specifically:
[0040] 1. Multi-indicator data
[0041] Collect monitoring-related data, such as system logs and performance metrics. Clean and preprocess the data to ensure accuracy and consistency. Identify the metrics that need to be monitored, and compare metrics from the same time period when an anomaly occurs to obtain multi-metric data for that period.
[0042] 2. Construction of indicator relationships
[0043] (1) First, by analyzing the frequency of simultaneous occurrence of abnormal events within a given time window, the co-occurrence pattern among abnormalities is revealed, and an abnormal co-occurrence frequency analysis is conducted to determine the correlation between certain database indicators.
[0044] Assuming there are N anomalous events (the same as the number of iterations N below), construct the anomalous co-occurrence matrix M. ij This represents the number of times that exception events i and j co-occur across all time windows.
[0045] 1) Construct the anomaly co-occurrence matrix: M ij
[0046] 2) Normalized anomaly co-occurrence frequency:
[0047] NormalizedMatrix ij M represents the frequency of elements in the normalized anomalous co-occurrence matrix. ij This indicates the number of times that abnormal events i and j occur together. This represents the total number of occurrences of exception event i across all time windows.
[0048] (2) Curve similarity pruning
[0049] As previously discussed, there are close correlations among different monitoring metrics. To utilize these correlations more effectively, a curve similarity pruning method is introduced to avoid unnecessary redundant information during the recommendation phase.
[0050] Let exception events i and j be... Let be the mean of the first-order difference sequence of event i at time m. Let be the mean of the first-order difference sequence of event j at time m. Let i be the value corresponding to event i at time m. Let be the value of time j at time m, where p represents the rate of weight decay, m represents the specific time point, n is the total number of times, and sim(i,j) represents the similarity between anomalous event i and anomalous event j. To capture recent trends and dynamic changes, this similarity pruning introduces a time-weighted factor and dynamically adjusts the weight factor to adapt to the trend of time changes. The formula is as follows:
[0051]
[0052] The correlation coefficient calculated by this formula determines the degree of correlation between the indicators. To better perform pruning, a threshold is set to filter out indicators with weak relationships. The remaining indicator pairs can be used to construct the final indicator relationship graph based on the above method.
[0053] 3. Key Indicator Recommendations
[0054] Utilize the previously obtained relationship graph between monitoring metrics, where nodes represent monitoring metrics and edges represent the relationships between metrics. Assign an initial weight to each node (monitoring metric) as the starting point of the random walk. The iterative calculation process of the random walk algorithm is as Figure 2 shown, and the specific steps are as follows:
[0055] (1) Determine the objective function F(x), where the objective function has multiple decision variables;
[0056] (2) Before optimization, determine the number of iterations N, the initial step size λ, the initial value x0 of the decision variable, and the control precision ξ (where ξ > 0);
[0057] (3) Determine whether the set number of iterations is reached, that is, whether k < N is satisfied, where k = 1;
[0058] (4) If k = 1 for the first iteration, then randomly generate a d-dimensional vector μ = (μ1, μ2,..., μ d ) within (0, 1), and calculate Let x1 = x + λμ′ to complete the first iteration, where x is a vector composed of multiple decision variables;
[0059] (5) Calculate the objective function. If the condition F(x1) < F(x) is satisfied, then a point better than the starting value is found, making k = 1, and set x * = x1 (x * is the optimal value), and return to step (3). If the condition is not satisfied, then k = k + 1, and return to step (4);
[0060] (6) If the number of iterations reaches the setting but the optimal solution has not been found yet, then output the current value (instead of the optimal solution), that is, λ < ξ, and the algorithm terminates; if not satisfied, continue the iteration.
[0061] After multiple rounds of random walk iterations as above, update the weight of each node, and sort the monitoring metrics according to the final weight value. The higher the weight of the metric, the more prominent it is in the recommended ranking. Output the recommended ranking: Output the sorted monitoring metrics as the recommended ranking result.
[0062] Database monitoring metrics usually appear in the form of time series data, showing a curvilinear trend. These curves do not exist in isolation, but there are internal associations and correlations within the database system. The present invention merges redundant metrics by calculating the abnormal co-occurrence frequency and correlation analysis between pairs of monitoring metrics to obtain a relationship graph between monitoring metrics, and finally outputs a recommended ranking through the random walk algorithm: Output the sorted monitoring metrics as the recommended ranking result, which helps the operation and maintenance personnel quickly locate problems and reduce the workload of the operation and maintenance personnel, and helps to lock the root cause of the problem.
[0063] This algorithm comprehensively considers the relationships between multiple monitoring indicators through multi-indicator data analysis, rather than focusing on a single indicator. This helps to comprehensively analyze the overall status of the system and improve the global perspective for problem detection and troubleshooting. Furthermore, it introduces anomaly co-occurrence frequency analysis, revealing co-occurrence patterns between anomalies by statistically analyzing the frequency of simultaneous occurrences of abnormal events, thus helping to determine the correlation between database indicators. To filter out indicators with weak relationships and capture recent trends and dynamic changes, curve similarity pruning introduces a time-weighted factor and dynamically adjusts the weighting factor to adapt to trends changing over time.
[0064] A more specific example:
[0065] (1) Collect and monitor related data, compare the indicators in the same time period when an anomaly occurs, and obtain multi-indicator data in that time period.
[0066] (2) Assuming there are three monitoring indicators A, B, and C, the constructed anomaly co-occurrence matrix is as follows:
[0067] Where M ij Let represent the number of times that monitoring metrics i and j both exhibit anomalies across all time windows. The matrix for calculating the anomaly co-occurrence frequency is then given by... The denominator is the total number of times each monitoring metric occurs together with other metrics across all time windows. Each row of the result represents the probability that the corresponding monitoring metric occurs together with other metrics across all time windows.
[0068] Curve similarity pruning: Based on the abnormal co-occurrence frequencies obtained above, similarity is calculated, and pruning is performed based on a threshold. The resulting correlation coefficient matrix is as follows: If the threshold is set to 0.5, then the retained indicator pairs (A, B) with a correlation coefficient greater than 0.5 will form the final indicator relationship graph, as shown below.
[0069] (3) Utilize the relationship graph obtained from the monitoring indicators, where nodes represent monitoring indicators and edges represent the relationships between indicators. Set the objective function F(x) = a 2 +b 2 +c 2 The number of iterations is N = 100, λ = 0.01, x0 is the initial weight of each monitoring indicator, which is set to 1 / 3 here, and ξ = 0.001.
[0070] Iteration:
[0071] Initialize the number of iterations k = 1;
[0072] If k = 1, randomly generate a d-dimensional vector μ within (0, 1), and then transform μ to obtain
[0073] Perform the first iteration: x1 = x + λμ′;
[0074] Calculate the objective function F(x1): If F(x1) < F(x0), it means a better point than the starting value x0 has been found, and update x0 = x1.
[0075] If the condition is not satisfied, then k = k + 1, go back to the previous step, and repeat the steps until λ < ξ or the iteration count N is reached.
[0076] The first few steps of the iterative process will be demonstrated:
[0077] x1 = [0.35, 0.35, 0.3]
[0078] x2 = [0.22, 0.38, 0.4]
[0079] x3 = [0.27, 0.34, 0.39]
[0080] …
[0081] Finally, sort the monitoring indicators according to the final weight values. The higher the weight, the higher the indicator ranks in the recommendation.
[0082] Through curve similarity pruning, this application filters out pairs of monitoring indicators with relatively high abnormal co-occurrence frequency similarity. In the embodiment, a similarity threshold (0.5) is set, and only pairs of indicators with strong correlation are retained. Compared with the existing technologies that have problems such as high computational complexity and easy neglect of the correlation between multiple indicators, this application reduces the computational complexity and improves the accuracy of problem localization, helping to quickly discover the root causes of potential problems in complex systems. On the other hand, through computational operations such as repeated iteration in the embodiment, this application ensures that the finally output indicator weights are closer to the global optimal solution, dynamically adjusts the indicator weights, and makes the final ranking more accurate.
[0083] The units, devices, or modules described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. For ease of description, the above devices are described by dividing them into various modules according to their functions. Of course, in implementing this application, the functions of each module can be implemented in one or more software and / or hardware, or the module that implements the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection between the devices or units shown or discussed can be through some interfaces, and the indirect coupling or communication connection between the devices or units can be electrical, mechanical, or other forms.
[0084] Those skilled in the art will also know that, besides implementing the controller using purely computer-readable program code, the same functions can be achieved by logically programming the method steps, making the controller function as logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers (PLCs), and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the devices within it used to implement various functions can also be considered structures within that hardware component. Alternatively, the devices used to implement various functions can be considered as both software modules implementing the method and structures within a hardware component.
[0085] This application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, classes, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.
[0086] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, mobile terminal, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.
[0087] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. This application can be used in numerous general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable electronic devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices, etc.
[0088] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.
Claims
1. A database monitoring metric recommendation method based on association recommendation, characterized in that: The method includes the following steps: (1) Obtain multi-indicator data, collect data related to monitoring, clean and preprocess the data, compare the monitoring indicators in the same time period when an anomaly occurs, and obtain multi-indicator data in that time period. (2) Constructing indicator relationships: Calculating the frequency of abnormal co-occurrence between each pair of monitoring indicators; (3) Recommend key indicators: Determine the recommendation priority and ranking of monitoring indicators, and recommend key monitoring indicators; When constructing the relationship between indicators in step (2), the correlation coefficient is calculated. Among them, the abnormal events are The abnormal event is , For the event In time m The mean of the first-order difference sequence, For the event In time m The mean of the first-order difference sequence, For the event In time m The corresponding value, For the event In time m The corresponding value, To control the rate of weight decay, n For the total time, for and The degree of similarity between them; When recommending key indicators, the specific steps are as follows: (3.1) Determine the objective function , x It is a vector consisting of multiple decision variables; (3.2) Determine the parameters before optimization: total number of iterations N Initial step size Initial values of decision variables control precision , If the value is >0, set the current solution as the optimal solution. ; (3.3) Initialize the number of iterations k =1; (3.4) Randomly generate a number between (0,1). d dimensional vector ,calculate , order the k Decision variables for the next iteration ; (3.5) Calculate the objective function like If a better solution is found, then the optimal solution is updated. Then let k = k +1, proceed to step (3.6); (3.6) If the number of iterations reaches N And the current step size λ < ξ If the algorithm fails, it terminates and outputs the current optimal solution. Otherwise, return to step (3.4).
2. A database monitoring metric recommendation system based on association recommendation, characterized in that: The system includes: (1) Multi-indicator data acquisition module: This module collects data related to monitoring, cleans and preprocesses the data, and compares the monitoring indicators in the same time period when an anomaly occurs to obtain multi-indicator data in that time period. (2) Indicator Relationship Construction Module: This module calculates the frequency of abnormal co-occurrence between any two monitoring indicators; (3) Key indicator recommendation module: This module determines the recommendation priority and ranking of monitoring indicators and recommends key monitoring indicators; In the indicator relationship construction module, the correlation coefficient is calculated. Among them, the abnormal events are The abnormal event is , For the event In time m The mean of the first-order difference sequence, For the event In time m The mean of the first-order difference sequence, For the event In time m The corresponding value, For the event In time m The corresponding value, To control the rate of weight decay, n For the total time, for and The degree of similarity between them; The key indicator recommendation module employs the following specific steps: (3.1) Determine the objective function , x It is a vector consisting of multiple decision variables; (3.2) Determine the parameters before optimization: total number of iterations N Initial step size Initial values of decision variables control precision , If the value is >0, set the current solution as the optimal solution. ; (3.3) Initialize the number of iterations k =1; (3.4) Randomly generate a number between (0,1). d dimensional vector ,calculate , order the k Decision variables for the next iteration ; (3.5) Calculate the objective function like If a better solution is found, then the optimal solution is updated. Then let k = k +1, proceed to step (3.6); (3.6) If the number of iterations reaches N And the current step size λ < ξ If the algorithm fails, it terminates and outputs the current optimal solution. Otherwise, return to step (3.4).