Association recommendation-based database monitoring index recommendation method and system

Abstract

The invention provides a database monitoring index recommendation method and system based on association recommendation, and the method comprises the steps: merging redundant indexes through calculating the abnormal co-occurrence frequency between every two monitoring indexes and carrying out the correlation analysis, and obtaining a relation graph between the monitoring indexes; and the sorted monitoring indexes are used as recommended ranking results to be output, so that operation and maintenance personnel are helped to quickly position problems, the workload of the operation and maintenance personnel is reduced, and locking of root causes of the problems is facilitated. According to the method, the relationship among a plurality of monitoring indexes is comprehensively considered through multi-index data analysis, a single index is not concerned any more, the overall state of the system can be comprehensively analyzed, and the global view of problem discovery and troubleshooting is improved. And meanwhile, abnormal co-occurrence frequency analysis is introduced, and the co-occurrence mode between the anomalies is revealed by counting the simultaneous occurrence frequency of the abnormal events, so that the relevance between the database indexes is helped to be determined. In order to screen out indexes with weak relations and capture the latest change trend and dynamic change, a time weighting factor is introduced into curve similarity pruning, and the weight factor is dynamically adjusted to adapt to the trend of time change.

Description

Technical Field

[0001] The invention relates to a method and system for recommending database monitoring indicators based on association recommendation, and belongs to the field of database monitoring. Background Art

[0002] In a production environment, system administrators face huge challenges and need to effectively manage numerous operation and maintenance objects, including key performance indicators, servers, public and custom components, and other basic physical facilities. These operation and maintenance objects provide important support for system availability, but the monitoring indicators associated with them are growing exponentially, often reaching hundreds or thousands. As the observability of the system continues to improve, the number of monitoring indicators is also growing, which brings heavy observation and analysis tasks to system administrators.

[0003] Each operation and maintenance object exposes a large number of monitoring indicators, which cover all aspects of the system, from key performance indicators of application service capabilities to performance parameters of basic physical facilities. This requires system administrators to pay attention to multiple dimensions at the same time to monitor the real-time status of the system. However, the existence of large-scale monitoring indicators makes it difficult for system administrators to effectively process this information, thus affecting the efficiency of fault discovery and diagnosis.

[0004] In the monitoring system, the correlation recommendation between indicators can help system managers understand the system operation status more comprehensively and provide targeted information to detect problems and troubleshoot problems more quickly. Existing methods include:

[0005] (1) Automatic association rules: Using machine learning algorithms and data mining techniques, historical monitoring data can be analyzed to automatically identify association rules between different indicators. However, for large-scale systems, a large number of rules may be generated, some of which may be redundant.

[0006] (2) Expert system and rule engine: Establish association rules based on domain expertise and manually adjust and optimize by adding rules. However, over-reliance on domain expertise may fail to capture unknown associations.

[0007] (3) Model-based association analysis: Build statistical models to identify correlations between indicators. However, for nonlinear and complex relationships, building and maintaining models may require a lot of computing resources.

[0008] In the monitoring system, a single indicator alarm strategy is usually configured on each specific indicator to detect whether the indicator exceeds the preset threshold range, and an alarm is triggered once an abnormality occurs. This alarm strategy can respond quickly to a specific problem, but when deeply investigating and locating the root cause of the problem, it is necessary to analyze the co-occurrence frequency of abnormalities, reveal the pattern of simultaneous abnormalities of multiple indicators, and determine the correlation between them. Therefore, it is necessary to combine other related indicators for analysis. However, in database operation and maintenance, there are often a large number of monitoring indicators, and the correlation between them is weak, resulting in low efficiency during analysis. The system needs to solve how to merge redundant indicators by analyzing the correlation between monitoring indicators. On the other hand, faced with a large number of monitoring indicators, how to screen out key indicators that are highly relevant to the problem and solve the problems of low accuracy and slow speed in problem location. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 It is the system flow chart.

[0010] Figure 2 It is an iterative calculation flow chart. Summary of the invention

[0011] In response to the problems existing in the above existing methods, this application proposes a database monitoring indicator recommendation method based on association recommendation. In the indicator relationship construction stage, the correlation relationship of system abnormal events is detected by calculating the abnormal co-occurrence frequency between monitoring indicators, and the curve similarity pruning technology is combined to avoid redundant recommendations and generate a relationship diagram of monitoring indicators. In the key indicator recommendation stage, the random walk algorithm is used to analyze the above relationship diagram to determine the recommendation priority and ranking of the monitoring indicators to ensure the effective identification and recommendation of key monitoring indicators. The specific steps are as follows:

[0012] (1) Obtain multi-indicator data, collect data related to monitoring, clean and pre-process the data, compare the monitoring indicators in the same time period when an abnormality occurs, and obtain multi-indicator data within the time period.

[0013] (2) Construct indicator relationships: Calculate the abnormal co-occurrence frequency between monitoring indicators.

[0014] (3) Recommend key indicators: Determine the recommended priority and ranking of monitoring indicators and recommend key monitoring indicators.

[0015] Furthermore, when constructing the indicator relationship, the correlation coefficient is calculated The abnormal event is i, the abnormal event is j, is the mean of the first-order difference sequence of event i at time m, is the mean of the first-order difference sequence of event j at time m, is the value corresponding to event i at time m, is the value corresponding to time j at time m, p is the rate of weight decay, n is the total number of times, and sim(i,j) is the similarity between i and j.

[0016] Furthermore, when recommending key indicators, the specific steps are as follows:

[0017] 3.1 Determine the objective function F(x);

[0018] 3.2 Determine the parameters before optimization: number of iterations N, initial step size λ, initial value of decision variable x0, control accuracy ξ, ξ>0;

[0019] 3.3 Determine whether the set number of iterations has been reached;

[0020] 3.4 If k = 1, perform the first iteration and randomly generate a d-dimensional vector μ = (μ1, μ2, …, μ d ),calculate Let x1=x+λμ′ to complete the first iteration, where x is a vector consisting of multiple decision variables.

[0021] 3.5 Calculate the objective function and perform loop operations based on the judgment conditions.

[0022] 3.6 If the number of iterations reaches the setting but the optimal solution is still not found, the current value is output, that is, λ<ξ, and the algorithm terminates; if it is not satisfied, continue to iterate.

[0023] Based on the above method, the present application further proposes a database monitoring indicator recommendation system based on association recommendation, the system comprising:

[0024] (1) Multi-indicator data acquisition module: This module collects data related to monitoring, cleans and preprocesses the data, compares the monitoring indicators in the same time period when an abnormality occurs, and obtains multi-indicator data within the time period.

[0025] (2) Indicator relationship construction module: This module calculates the abnormal co-occurrence frequency between monitoring indicators.

[0026] (3) Key indicator recommendation module: This module determines the recommendation priority and ranking of monitoring indicators and recommends key monitoring indicators.

[0027] Furthermore, in the indicator relationship construction module, the correlation coefficient is calculated The abnormal event is i, the abnormal event is j, is the mean of the first-order difference sequence of event i at time m, is the mean of the first-order difference sequence of event j at time m, is the value corresponding to event i at time m, is the value corresponding to time j at time m, p is the rate of weight decay, n is the total number of times, and sim(i,j) is the similarity between i and j.

[0028] Furthermore, the key indicator recommendation module adopts the following specific steps:

[0029] 3.1 Determine the objective function F(x);

[0030] 3.2 Determine the parameters before optimization: number of iterations N, initial step size λ, initial value of decision variable x0, control accuracy ξ, ξ>0;

[0031] 3.3 Determine whether the set number of iterations has been reached;

[0032] 3.4 If k = 1, perform the first iteration and randomly generate a d-dimensional vector μ = (μ1, μ2, …, μ d ),calculate Let x1=x+λμ′ to complete the first iteration, where x is a vector consisting of multiple decision variables.

[0033] 3.5 Calculate the objective function and perform loop operations based on the judgment conditions.

[0034] 3.6 If the number of iterations reaches the setting but the optimal solution is still not found, the current value is output, that is, λ<ξ, and the algorithm terminates; if it is not satisfied, continue to iterate.

[0035] After multiple rounds of random walk iterations, the weight of each node is updated, and the monitoring indicators are sorted according to the final weight value. The higher the weight, the higher the recommended ranking, and the sorted monitoring indicators are output as the recommended ranking results. This step uses the node's neighbor information to dynamically adjust the weight, captures the hierarchy between different monitoring indicators, and combines the relationship diagram of the database monitoring indicators to make the recommendation results more reflective of the actual status of the database, helping operation and maintenance personnel quickly find the indicators that need to be focused on.

[0036] The present invention merges redundant indicators by calculating the abnormal co-occurrence frequency and correlation analysis between the monitoring indicators, obtains the relationship diagram between the monitoring indicators, and finally outputs the recommended ranking through the random walk algorithm: the sorted monitoring indicators are output as the recommended ranking results, which helps the operation and maintenance personnel to quickly locate the problem, reduce the workload of the operation and maintenance personnel, and help to lock the root cause of the problem. The present invention comprehensively considers the relationship between multiple monitoring indicators through multi-indicator data analysis, no longer focuses on a single indicator, helps to comprehensively analyze the overall state of the system, and improves the global vision of problem discovery and troubleshooting. At the same time, the abnormal co-occurrence frequency analysis is introduced, and the co-occurrence pattern between abnormalities is revealed by counting the frequency of abnormal events occurring simultaneously, which helps to determine the correlation between database indicators. In order to filter out indicators with weaker relationships and capture the recent change trends and dynamic changes, the time weighting factor is introduced in the curve similarity pruning, and the weight factor is dynamically adjusted to adapt to the trend of time changes. DETAILED DESCRIPTION

[0037] In order to improve the efficiency of troubleshooting, the present invention proposes a database monitoring indicator recommendation method based on association recommendation, introduces an associated indicator recommendation function, and recommends potential associated indicators to system administrators by analyzing the relationship between multiple indicators, thereby assisting them in performing more targeted manual analysis when locating the root cause.

[0038] The present invention consists of two main parts: indicator relationship construction and key indicator recommendation. First, the indicator relationship construction stage calculates the abnormal co-occurrence frequency between the monitoring indicators to detect the correlation between abnormal events in the system. Secondly, this stage introduces curve similarity pruning to avoid redundant recommendation of related indicators, and finally establishes a relationship diagram between monitoring indicators. The key indicator recommendation stage uses a random walk algorithm to conduct an in-depth analysis of the indicator relationship diagram obtained in the previous step to determine the recommended ranking of the monitoring indicators.

[0039] The main steps are shown in Figure 1 , specifically:

[0040] 1. Multi-indicator data

[0041] Collect monitoring-related data, such as system logs, performance indicators, etc. Clean and preprocess the data to ensure data accuracy and consistency. Determine the indicators that need to be monitored, and compare the indicators in the same time period when an anomaly occurs to obtain multi-indicator data within the time period.

[0042] 2. Indicator relationship construction

[0043] (1) First, we analyze the frequency of anomalous events occurring simultaneously within a given time window to reveal the co-occurrence pattern between anomalies, conduct anomaly co-occurrence frequency analysis, and determine the correlation between certain database indicators.

[0044] Assuming there are N abnormal events (the same as the number of iterations N below), construct the abnormal co-occurrence matrix M ij It represents the number of co-occurrences of abnormal events i and j in all time windows.

[0045] 1) Construct anomaly co-occurrence matrix: M ij

[0046] 2) Normalized anomaly co-occurrence frequency:

[0047] Where NormalizedMatrix ij represents the element frequency in the normalized abnormal co-occurrence matrix, M ij represents the number of times abnormal events i and j occur together, It represents the total number of occurrences of abnormal event i in all time windows.

[0048] (2) Curve Similarity Pruning

[0049] As mentioned before, there are close correlations between different monitoring indicator data. In order to make more effective use of these correlations, the curve similarity pruning method is introduced to avoid unnecessary redundant information in the recommendation stage.

[0050] Abnormal event i and abnormal event j, let is the mean of the first-order difference sequence of event i at time m, is the mean of the first-order difference sequence of event j at time m, is the value corresponding to event i at time m, is the value corresponding to time j at time m, where p represents the rate of weight decay, m represents the specific time point, n is the total time, and sim(i,j) represents the similarity between abnormal event i and abnormal event j. In order to capture the recent change trend and dynamic changes, the similarity pruning introduces the time weighting factor and dynamically adjusts the weight factor to adapt to the trend of time change. The formula is as follows:

[0051]

[0052] The correlation coefficient calculated according to the formula determines the correlation degree of the indicators. In order to better perform pruning operations, a threshold is set to filter out indicators with weaker relationships. According to the above method, the retained indicator pairs can be used to construct the final indicator relationship diagram.

[0053] 3. Key indicator recommendations

[0054] Utilize the previously obtained relationship graph among monitoring metrics, where nodes represent monitoring metrics and edges represent the relationships among metrics. Assign initial weights to each node (monitoring metric) as the starting point of random walk. The iterative calculation process of the random walk algorithm is as Figure 2 shown, and the specific steps are as follows:

[0055] (1) Determine the objective function F(x), where the objective function has multiple decision variables;

[0056] (2) Before optimization, determine the number of iterations N, the initial step size λ, the initial value x0 of the decision variable, and the control accuracy ξ (where ξ > 0);

[0057] (3) Determine whether the set number of iterations is reached, that is, whether k < N is satisfied, where k = 1;

[0058] (4) If k = 1 for the first iteration, then randomly generate a d-dimensional vector μ = (μ1, μ2, …, μ d ) between (0, 1), and calculate Let x1 = x + λμ′ to complete the first iteration, where x is a vector composed of multiple decision variables;

[0059] (5) Calculate the objective function. If the condition F(x1) < F(x) is satisfied, then a point better than the starting value is found, making k = 1, and set x * = x1 (x * is the optimal value), and return to step (3). If the condition is not satisfied, then k = k + 1, and return to step (4);

[0060] (6) If the number of iterations reaches the setting but the optimal solution has not been found yet, then output the current value (instead of the optimal solution), that is, λ < ξ, and the algorithm terminates; if not satisfied, continue the iteration.

[0061] After multiple rounds of random walk iterations as above, update the weights of each node, and sort the monitoring metrics according to the final weight values. The higher the weight of an indicator, the more prominent it is in the recommended ranking. Output the recommended ranking: Output the sorted monitoring metrics as the recommended ranking result.

[0062] Database monitoring metrics usually appear in the form of time series data, showing a curvilinear trend. These curves do not exist in isolation, but there are correlations and relationships among them within the database system. The present invention merges redundant metrics by calculating the abnormal co-occurrence frequency and correlation analysis between pairs of monitoring metrics to obtain the relationship graph among monitoring metrics, and finally outputs the recommended ranking through the random walk algorithm: Output the sorted monitoring metrics as the recommended ranking result, which helps the operation and maintenance personnel quickly locate problems and reduce their workload, and helps to lock the root cause of the problems.

[0063] The algorithm comprehensively considers the relationship between multiple monitoring indicators through multi-indicator data analysis, rather than focusing on a single indicator. This helps to comprehensively analyze the overall status of the system and improve the global vision of problem discovery and troubleshooting. In addition, the anomaly co-occurrence frequency analysis is introduced. By counting the frequency of simultaneous abnormal events, the co-occurrence pattern between anomalies is revealed, which helps determine the correlation between database indicators. In order to filter out indicators with weak relationships and capture recent trends and dynamic changes, the curve similarity pruning introduces a time weighting factor and dynamically adjusts the weight factor to adapt to the trend of time changes.

[0064] A more specific embodiment:

[0065] (1) Collect data related to monitoring, compare the indicators in the same time period when an abnormality occurs, and obtain multi-indicator data within the time period.

[0066] (2) Assuming there are three monitoring indicators A, B and C, the constructed abnormal co-occurrence matrix is:

[0067] Among them, M ij represents the number of times monitoring indicators i and j have common anomalies in all time windows, so the matrix for calculating the anomaly co-occurrence frequency is The denominator is the total number of times each monitoring indicator has an anomaly in all time windows. Each row of the calculation result represents the probability that the corresponding monitoring indicator has an anomaly in all time windows together with other indicators.

[0068] Curve similarity pruning: Based on the abnormal co-occurrence frequency obtained above, similarity calculation is performed, and pruning operation is performed according to the threshold. The calculated correlation coefficient matrix is ​​as follows: If the threshold is set to 0.5, the retained indicator pair is (A, B), and its corresponding correlation coefficient is greater than 0.5, which will form the final indicator relationship diagram. The final relationship diagram is as follows

[0069] (3) Using the relationship graph of the monitoring indicators obtained, where nodes represent monitoring indicators and edges represent the relationship between indicators. Set the objective function F(x) = a 2 +b 2 +c 2 , the number of iterations N = 100, λ = 0.01, x0 is the initial weight of each monitoring indicator, which is set to 1 / 3 here, ξ = 0.001.

[0070] Iteration:

[0071] Initialize the number of iterations k = 1;

[0072] If k = 1, randomly generate a d-dimensional vector μ between (0, 1), and then transform μ to obtain

[0073] Perform the first iteration: x1 = x + λμ′;

[0074] Calculate the objective function F(x1): If F(x1) < F(x0), it means a better point than the starting value x0 has been found, and update x0 = x1.

[0075] If the condition is not met, then k = k + 1, go back to the previous step, and repeat the steps until λ < ξ or the number of iterations N is reached.

[0076] The first few steps of the iterative process will be demonstrated:

[0077] x1 = [0.35, 0.35, 0.3]

[0078] x2 = [0.22, 0.38, 0.4]

[0079] x3 = [0.27, 0.34, 0.39]

[0080] …

[0081] Finally, sort the monitoring indicators according to the final weight values. The higher the weight, the more prominent the indicator is in the recommended ranking.

[0082] In this application, through curve similarity pruning, monitoring indicator pairs with a relatively high similarity of abnormal co-occurrence frequencies are screened out. In the embodiment, a similarity threshold (0.5) is set, and only indicator pairs with strong correlations are retained. Compared with the existing technologies that have problems such as high computational complexity and easy neglect of the correlations between multiple indicators, this application reduces the computational complexity and improves the accuracy of problem location, helping to quickly discover the root causes of potential problems in complex systems. On the other hand, in the embodiment of this application, through calculation operations such as repeated iterations, it is ensured that the finally output indicator weights are closer to the global optimal solution, dynamically adjust the indicator weights, and make the final ranking more accurate.

[0083] The units, devices or modules described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. For the convenience of description, the above devices are described separately by functions divided into various modules. Of course, when implementing this application, the functions of each module can be implemented in the same or more software and / or hardware, or the modules that implement the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0084] Those skilled in the art also know that, in addition to implementing the controller in a purely computer-readable program code, the controller can be made to implement the same function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered as a hardware component, and the devices for implementing various functions included therein can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules for implementing the method and structures within the hardware component.

[0085] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0086] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0087] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0088] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present application. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A database monitoring indicator recommendation method based on association recommendation, characterized in that: The method comprises the following steps: (1) Obtain multi-indicator data, collect monitoring-related data, clean and pre-process the data, compare the monitoring indicators in the same time period when an abnormality occurs, and obtain multi-indicator data within the time period. (2) Construct indicator relationships: Calculate the abnormal co-occurrence frequency between monitoring indicators. (3) Recommend key indicators: Determine the recommended priority and ranking of monitoring indicators and recommend key monitoring indicators.

2. The method for recommending database monitoring indicators based on association recommendation according to claim 1, characterized in that: Step (2) When constructing the indicator relationship, calculate the correlation coefficient The abnormal event is i, the abnormal event is j, is the mean of the first-order difference sequence of event i at time m, is the mean of the first-order difference sequence of event j at time m, is the value corresponding to event i at time m, is the value corresponding to time j at time m, p is the rate of weight decay, n is the total number of times, and sim(i,j) is the similarity between i and j.

3. A database monitoring indicator recommendation method based on association recommendation as claimed in claim 1, characterized in that when recommending key indicators, the specific steps are as follows: 3.1 Determine the objective function F(x); 3.2 Determine the parameters before optimization: number of iterations N, initial step size λ, initial value of decision variable x0, control accuracy ξ, ξ>0; 3.3 Determine whether the set number of iterations has been reached; 3.4 If k = 1, perform the first iteration and randomly generate a d-dimensional vector μ = (μ1, μ2, …, μ d ),calculate Let x1=x+λμ′ to complete the first iteration, where x is a vector consisting of multiple decision variables. 3.5 Calculate the objective function and perform loop operations based on the judgment conditions. 3.6 If the number of iterations reaches the setting but the optimal solution is still not found, the current value is output, that is, λ<ξ, and the algorithm is terminated; if it is not satisfied, continue to iterate.

4. A database monitoring indicator recommendation system based on association recommendation, characterized in that: The system includes: (1) Multi-indicator data acquisition module: This module collects data related to monitoring, cleans and preprocesses the data, compares the monitoring indicators in the same time period when an abnormality occurs, and obtains multi-indicator data within the time period. (2) Indicator relationship construction module: This module calculates the abnormal co-occurrence frequency between monitoring indicators. (3) Key indicator recommendation module: This module determines the recommendation priority and ranking of monitoring indicators and recommends key monitoring indicators.

5. A database monitoring indicator recommendation system based on association recommendation as claimed in claim 4, characterized in that: In the indicator relationship construction module, calculate the correlation coefficient The abnormal event is i, the abnormal event is j, is the mean of the first-order difference sequence of event i at time m, is the mean of the first-order difference sequence of event j at time m, is the value corresponding to event i at time m, is the value corresponding to time j at time m, p is the rate of weight decay, n is the total number of times, and sim(i,j) is the similarity between i and j.

6. A database monitoring indicator recommendation system based on association recommendation as claimed in claim 4, characterized in that: The specific steps of the key indicator recommendation module are as follows: 3.1 Determine the objective function F(x); 3.2 Determine the parameters before optimization: number of iterations N, initial step size λ, initial value of decision variable x0, control accuracy ξ, ξ>0; 3.3 Determine whether the set number of iterations has been reached; 3.4 If k = 1, perform the first iteration and randomly generate a d-dimensional vector μ = (μ1, μ2, …, μ d ),calculate Let x1=x+λμ′ to complete the first iteration, where x is a vector consisting of multiple decision variables. 3.5 Calculate the objective function and perform loop operations based on the judgment conditions. 3.6 If the number of iterations reaches the setting but the optimal solution is still not found, the current value is output, that is, λ<ξ, and the algorithm is terminated; if it is not satisfied, continue to iterate.

Images