A method and system for processing system performance monitoring data
By dynamically adjusting the performance transfer threshold and linear interpolation to fill data gaps, the problem of high storage cost of system performance monitoring data is solved, efficient data compression and fault alarm are achieved, and enterprise operation and maintenance costs are reduced.
Patent Information
- Application Number
- CN202010567891.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-06-19
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2040-06-19
AI Technical Summary
Current system performance monitoring generates massive monitoring data, resulting in high operating and maintenance costs for enterprises. The existing technology is unable to take into account the real-time data and storage efficiency of data while reducing the frequency of data acquisition and fixed threshold screening methods, which affects the timeliness of fault location and alarm.
By dynamically adjusting the performance transfer threshold, the monitoring data with absolute value less than the threshold is discarded, and the performance transfer threshold is calculated based on the threshold level of the monitoring data, only data close to the alarm threshold is stored, and linear interpolation is used to fill the data gap.
Effectively compress data, reduce storage resource requirements, improve the effectiveness of fault alarms and problem positioning, reduce enterprise operation costs, and store compression rate can reach 90%, ensuring data integrity and analysis accuracy.
Smart Images

Figure CN113704047B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of database operation and maintenance, and more specifically, to a method and system for optimizing processing of system performance monitoring data. Background Art
[0002] With the rapid development of the Internet era, the complexity of enterprise business and the diversity of user needs are increasing, and users and enterprises have put forward higher and higher requirements for the stability of business platforms. In order to improve the stability of the business platform, a good monitoring system is built to achieve timely warning and fault detection through real-time monitoring of the business platform, and to achieve fault tracing and problem location through stored monitoring data. It can not only provide users with stable, efficient and secure services, but also facilitate operation and maintenance personnel to carry out daily maintenance work and troubleshoot abnormal problems, which is of great significance to the development of enterprises.
[0003] In the process of business platform monitoring deployment, system performance monitoring is very important and is the basis of business platform monitoring. System performance monitoring mainly includes CPU (such as CPU usage, CPU idle rate, system load, etc.), memory (such as total memory, used memory items, remaining memory, file system buffer / cache occupied space, actual used space, etc.), disk (such as disk capacity monitoring, disk IO utilization monitoring, wait delay monitoring, etc.), network (such as network card traffic monitoring, network card packet loss rate monitoring, etc.), database (such as space utilization monitoring, database process system resource usage monitoring, request number monitoring, concurrent connection monitoring, etc.), etc. In the actual monitoring process, a server generally has 30 system performance monitoring items, and data is obtained once a minute, so the data volume is 43,200 data per day. For 1,000 servers, the data volume of threshold monitoring in one day is 43,200,000 data. Under normal circumstances, data storage is required for more than half a year. Therefore, the storage of massive monitoring data places very high requirements on the server. The pressure of large-scale monitoring in enterprises is mainly reflected in performance. In order to store massive amounts of monitoring data, enterprises need to spend huge amounts of resource usage costs and operation and maintenance costs. How to process massive amounts of monitoring data and greatly reduce enterprise costs while ensuring real-time performance is an important challenge and also a hot topic in the current monitoring operation and maintenance field.
[0004] Currently, a major way to deal with massive monitoring data is to reduce the frequency of data collection and reduce the storage time of monitoring data. However, this method sacrifices the real-time nature of the data, affects the timeliness of installation alarms, and is not conducive to fault location and troubleshooting because it cannot reflect the actual operating conditions of the system at the time.
[0005] The Chinese patent application "Real-time Data Compression and Storage Method for Monitoring System" (201711328408.1) proposes to compare the monitoring records collected at this moment with the monitoring records collected last time in bytes. If the monitoring records are exactly the same as the last monitoring records, the monitoring records are stored in the same way; if they are not the same, the monitoring records are stored in the same way by calculating the change records compression and storage method. Although the method introduced in the patent application compresses the monitoring data, reducing the storage space occupied by the monitoring data, it still stores a large amount of redundant and invalid data that has little effect on monitoring alarms and fault location.
[0006] The Chinese patent application "A system and method for processing massive data on an industrial monitoring platform" (201811130154.7) proposes three data compression methods: a single threshold screening method, a cumulative threshold screening method, and a single threshold and cumulative threshold combined screening method. The core idea of these three methods is to compare the difference between the current collected monitoring data and the most recent last collected data with a preset threshold, and to remove or retain the current monitoring data based on the comparison result. This patent application effectively saves storage resources by eliminating a large amount of complex and inefficient data. However, the preset threshold used for comparison in this patent application is fixed. In actual monitoring, data far below the alarm threshold does not have much effect on fault alarms and problem location, while data close to the alarm threshold requires more attention. Therefore, the data compression method with a fixed preset threshold in this patent application cannot achieve both efficient data compression and storage of valid data.
[0007] Therefore, although the current system performance monitoring realizes fault warning and fault location through real-time monitoring data collection and monitoring data storage, which greatly improves the stability of the business platform, the current system performance monitoring generates a large amount of monitoring data, requiring enterprises to spend a lot of money on data storage and operation and maintenance, which greatly increases the operating costs of enterprises. In order to further reduce the operation and maintenance costs of enterprises while ensuring that the monitoring data is reliable and effective, it is hoped to provide an improved system performance monitoring data calculation and processing method. Summary of the invention
[0008] This Summary is provided to introduce some concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
[0009] This method preprocesses the currently collected system performance monitoring data to obtain the absolute value of the difference between the current data and the previously stored data. When the absolute value is less than the given performance transfer threshold, the data collected this time will be discarded. When the difference is not less than the given performance transfer threshold, the data collected this time will be stored. Moreover, for system performance monitoring, the closer the monitoring data is to the alarm threshold, the more conducive it is to fault alarm and problem location, and the higher the effectiveness. The given performance transfer threshold in this method is dynamically changing and is correlated with the system performance monitoring data. The closer the data is to the alarm threshold, the smaller the corresponding performance transfer threshold, that is, as much as possible to store data with high effectiveness.
[0010] According to one aspect of the present invention, there is provided a method for processing system performance monitoring data, wherein the method includes:
[0011] Obtain the current monitoring data V * and the previous monitoring data V;
[0012] Based on the current monitoring data V * obtain the performance transfer threshold V^ corresponding to the current monitoring data V * ;
[0013] Calculate the performance difference Δ between the current monitoring data V * and the previous monitoring data V;
[0014] Compare the performance difference Δ with the performance transfer threshold V^; and
[0015] If Δ < V^, discard the current monitoring data V * , otherwise, store the current monitoring data V * in the monitoring database.
[0016] According to an embodiment of the present invention, the performance transfer threshold V^ decreases as the current monitoring data V * increases.
[0017] According to a further embodiment of the present invention, the performance transfer threshold V^ is obtained based on the following formula:
[0018]
[0019] wherein, v represents the threshold level corresponding to the current monitoring data and is calculated according to the following formula:
[0020]
[0021] where v1 to v n are preset values of the current monitoring data at the corresponding threshold levels, and L1 to L nIt is a preset value for dividing threshold levels into sections.
[0022] According to a further embodiment of the present invention, the v1 to v n and L1 to L n The specific values of are set according to different categories of monitoring data, and among them, v1 to v n Adopt a fixed value or a decreasing sequence.
[0023] According to a further embodiment of the present invention, the method further includes: filling the data gap between two storage periods by linear interpolation.
[0024] According to a further embodiment of the present invention, the following formula is used to calculate linear interpolation:
[0025]
[0026] Wherein, T represents the sampling period, T0 is the sampling period corresponding to the first storage period, T1 is the sampling period immediately preceding the sampling period corresponding to the second storage period, and V0 and V1 are the values of the threshold class monitoring corresponding to the T0 and T1 periods respectively.
[0027] According to another aspect of the present invention, a system performance monitoring system is provided, wherein the monitoring system includes:
[0028] A monitoring data acquisition module configured to obtain the current monitoring data V * and the previous monitoring data V;
[0029] A monitoring data evaluation module configured to obtain, based on the current monitoring data V * a performance transfer threshold V^ corresponding to the current monitoring data V * ;
[0030] A monitoring data detection module configured to calculate the performance difference Δ between the current monitoring data V * and the previous monitoring data V; and
[0031] A monitoring data processing module configured to:
[0032] Compare the performance difference Δ with the performance transfer threshold V^; and
[0033] If Δ < V^, then discard the current monitoring data V * , otherwise, store the current monitoring data V * in the monitoring database.
[0034] According to an embodiment of the present invention, the monitoring data evaluation module obtains the performance transition threshold V^ based on the following formula:
[0035]
[0036] Where v represents the threshold level corresponding to the current monitoring data and is calculated according to the following formula:
[0037]
[0038] Where v1~v n It is the preset value of the current monitoring data at the corresponding threshold level, L1~L n is a preset value for segmenting the threshold level, and v1~v n Use a constant value or a decreasing sequence.
[0039] According to a further embodiment of the present invention, the monitoring data processing module is further configured to fill the data gap between two storage cycles by linear interpolation, wherein the linear interpolation is calculated using the following formula:
[0040]
[0041] Wherein, T represents the sampling period, T0 is the sampling period corresponding to the first storage period, T1 is the sampling period before the sampling period corresponding to the second storage period, and V0 and V1 are the values of the threshold monitoring corresponding to the T0 and T1 periods, respectively.
[0042] According to another aspect of the present invention, a system for monitoring system performance is provided, wherein the system comprises:
[0043] a monitoring database configured to store monitoring data; and
[0044] A monitoring system is configured to execute the method as described above.
[0045] Compared with the solutions in the prior art, the monitoring system and method provided by the present invention have at least the following advantages:
[0046] (1) Effectively realize data compression, which can accurately and reliably obtain real-time monitoring data of system performance while eliminating a large amount of redundant and repeated data, effectively saving storage resources and reducing enterprise operating costs. In addition, conventional monitoring in enterprise monitoring is mainly system performance monitoring. This method can effectively improve the compression performance of conventional monitoring, and the compressed storage can reach 90%.
[0047] (2) The performance threshold of monitoring data storage can be adjusted according to the current system monitoring data, which is of great help in storing effective data that can be used for fault alarm and problem location and reducing the storage of low-effective data.
[0048] (3) It is universal and can adjust the monitoring alarm threshold and its corresponding monitoring data range (performance transfer threshold) according to the requirements of different business platforms for threshold-type system performance monitoring data. This makes this method applicable to system monitoring data processing on different business platforms in different industries.
[0049] (4) It has integrity. Linear interpolation is used to fill in data that floats within the allowable range and is not stored, so as to ensure the integrity of the data as much as possible without affecting the correctness of the data analysis.
[0050] These and other features and advantages will become apparent by reading the following detailed description and by reference to the associated drawings.It is to be understood that the foregoing general description and the following detailed description are illustrative only and are not restrictive of the aspects of what is claimed. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] In order to understand the manner in which the above features of the present invention are used in detail, the above briefly summarized contents can be described in more detail with reference to various embodiments, some of which are shown in the accompanying drawings. However, it should be noted that the accompanying drawings only show some typical aspects of the present invention and should not be considered to limit its scope, because the description may allow for other equally effective aspects.
[0052] Figure 1 is an example architecture diagram of a monitoring system for processing and optimizing system performance monitoring data according to an embodiment of the present invention.
[0053] Figure 2 is a flow chart of a method for processing and optimizing system performance monitoring data according to an embodiment of the present invention. DETAILED DESCRIPTION
[0054] The present invention will be described in detail below in conjunction with the accompanying drawings, and the features of the present invention will be further revealed in the following specific description.
[0055] Figure 1 FIG. 1 is an exemplary architecture diagram of a monitoring system 100 for processing and optimizing system performance monitoring data according to an embodiment of the present invention. Figure 1As shown in , the monitoring system 100 of the present invention includes: a monitoring data acquisition module 101, a monitoring data evaluation module 102, a monitoring data detection module 103, and a monitoring data processing module 104. The monitoring data acquisition module 101 collects the monitoring data of each system performance monitored, and can directly obtain the real-time system performance data (also called current or current monitoring data) from the monitored system, and can also obtain the previous monitoring data from a monitoring database separate from the monitoring system. Among them, the previous monitoring data obtained is called the standard data value, and the current monitoring data obtained is called the comparison data value. The monitoring data evaluation module 102 obtains the performance transfer threshold corresponding to the current comparison data value based on the current monitoring data, that is, the comparison data value. The monitoring data detection module 103 obtains the absolute value Δ of the difference between the comparison data value and the standard data value, which is called the performance difference. The monitoring data processing module 104 further compares the performance difference with the performance transfer threshold. If the performance difference is less than the performance transfer threshold, the current comparison data is discarded. If the performance difference is not less than the performance transfer threshold, the current comparison data value is stored in the monitoring database and used as the standard data value for the next monitoring data processing.
[0056] Those skilled in the art will appreciate that the monitoring system and its modules of the present invention may be implemented in hardware or software, and the modules may be combined or combined in any suitable manner. In addition, the monitoring system and the monitoring database are generally separate, for example, the monitoring system is configured in or near the monitored system, and the monitoring database may be a separate or cloud-based database server, but the monitoring system may also be implemented as an integrated monitoring database.
[0057] Figure 2 2 is a flow chart of a method 200 for processing and optimizing system performance monitoring data according to an embodiment of the present invention. The method starts at step 201, where the monitoring data acquisition module 101 of the monitoring system 100 obtains the current monitoring data, referred to as the comparison data value V * , and obtain the previous monitoring data from the monitoring database, which is called the standard data value V. Let the data acquisition cycle be T, that is, the monitoring system obtains system performance monitoring data every time T.
[0058] Step 202: the monitoring data evaluation module 102 evaluates the current monitoring data (i.e., the comparison data value V * ) obtains the performance transition threshold V^ corresponding to the current comparison data value. The performance transition threshold V^ can be obtained according to formula (1):
[0059]
[0060] From formula (1), it can be seen that the performance transfer threshold V^ changes with the comparison data value V *In addition, in formula (1), v represents the threshold level corresponding to the current monitoring data. The higher the level, the smaller the floating range of the monitoring data. The specific calculation is shown in formula (2):
[0061]
[0062] In formula (2), v1~v n It is the preset value of the current monitoring data at the corresponding threshold level, L1~L n It is a preset value used to divide the threshold level into segments. For example, in the CPU utilization monitoring scenario, the monitoring data V * It is divided into three levels: 0-80%, 80%-90%, and 90%-100%. The corresponding monitoring threshold fluctuation ranges are 10%, 5%, and 1%, respectively. When the CPU occupancy rate is 70%, the performance transfer threshold
[0063] As a non-limiting example, v1~v n and L1~L n The specific value of can be set according to different threshold types of monitoring, and according to the specific business scenario, v1~v n A fixed value or decreasing sequence can be used to meet the performance transfer threshold V^ with the comparison data value V * decreases with the increase of .
[0064] In step 203, the monitoring data detection module 103 calculates the standard data value V and the comparison data value V obtained in step 201. * The performance difference Δ is
[0065] Δ=|V * -V| (3)
[0066] In step 204, the monitoring data processing module 104 compares the performance threshold V^ obtained in step 2 according to formula (1) with the performance difference Δ obtained in step 203 according to formula (3). If Δ<V^, then in step 205, the comparison data value V^ is discarded. * Otherwise, the method proceeds to step 206, and the comparison data value V * Stored in the monitoring database.
[0067] If the monitoring values for multiple consecutive cycles do not exceed the performance threshold, a large data gap will be generated. Although the monitoring data generally fluctuates within the range allowed by the current level, the data gaps for multiple cycles will be detrimental to subsequent data analysis. Therefore, optionally, method 200 may also include step 207, filling the data gap between two storage cycles by linear interpolation. As a non-limiting example, the calculation formula of the interpolation function that can be used is as follows:
[0068]
[0069] Wherein, T represents the sampling period, T0 is the sampling period corresponding to the first storage period, T1 is the sampling period before the sampling period corresponding to the second storage period, V0 and V1 are the values of the threshold monitoring corresponding to the periods T0 and T1 respectively, and the monitoring data filling values of all periods between periods T0 and T1 can be calculated by formula (4).
[0070] What has been described above includes examples of various aspects of the claimed subject matter. Of course, it is not possible to describe every conceivable combination of components or methods for the purpose of describing the claimed subject matter, but one of ordinary skill in the art will recognize that many further combinations and permutations of the claimed subject matter are possible. Thus, the disclosed subject matter is intended to encompass all such alterations, modifications, and variations that fall within the spirit and scope of the appended claims.
Claims
1. A method for processing system performance monitoring data, characterized in that: The method comprises: Get the monitoring data V * and the previous monitoring data V; Based on the monitoring data V * Obtain the monitoring data V corresponding to this time * The performance transfer threshold V ^ ; Calculate the monitoring data V * and the performance difference Δ of the previous monitoring data V; The performance difference Δ and the performance transition threshold V ^ compared to; and If Δ <V ^ , then discard the monitoring data V * Otherwise, the monitoring data V * Stored in the monitoring database, The performance transfer threshold V is obtained based on the following formula: ^ : Where v represents the threshold level corresponding to the current monitoring data and is calculated according to the following formula: Where v1~v n It is the preset value of the current monitoring data at the corresponding threshold level, L1~L n is a preset value for segmenting the threshold level, and v1~v n Use a constant value or a decreasing sequence.
2. The method according to claim 1, characterized in that The performance transfer threshold V ^ With the monitoring data V * decreases with the increase of .
3. The method according to claim 1, characterized in that The v1~v n and L1~L n The specific value is set according to different monitoring data categories.
4. The method according to claim 1, characterized in that The method further comprises: filling the data gap between two storage cycles by linear interpolation.
5. The method according to claim 4, characterized in that The linear interpolation is calculated using the following formula: Wherein, T represents the sampling period, T0 is the sampling period corresponding to the first storage period, T1 is the sampling period before the sampling period corresponding to the second storage period, and V0 and V1 are the values of the threshold monitoring corresponding to the T0 and T1 periods, respectively.
6. A system performance monitoring system, characterized in that: The monitoring system comprises: Monitoring data acquisition module, the monitoring data acquisition module is configured to obtain the current monitoring data V * and the previous monitoring data V; A monitoring data evaluation module is configured to evaluate the monitoring data based on the monitoring data V * Obtain the monitoring data V corresponding to this time * The performance transfer threshold V ^ ; A monitoring data detection module is configured to calculate the monitoring data V * and the performance difference Δ of the previous monitoring data V; and A monitoring data processing module, wherein the monitoring data processing module is configured to: The performance difference Δ and the performance transition threshold V ^ compared to; and If Δ <V ^ , then discard the monitoring data V * Otherwise, the monitoring data V * Stored in the monitoring database, The monitoring data evaluation module obtains the performance transition threshold V based on the following formula: ^ : Where v represents the threshold level corresponding to the current monitoring data and is calculated according to the following formula: Where v1~v n It is the preset value of the current monitoring data at the corresponding threshold level, L1~L n is a preset value for segmenting the threshold level, and v1~v n Use a constant value or a decreasing sequence.
7. The monitoring system according to claim 6, characterized in that: The monitoring data processing module is further configured to fill the data gap between two storage cycles by linear interpolation, wherein the linear interpolation is calculated using the following formula: Wherein, T represents the sampling period, T0 is the sampling period corresponding to the first storage period, T1 is the sampling period before the sampling period corresponding to the second storage period, and V0 and V1 are the values of the threshold monitoring corresponding to the T0 and T1 periods, respectively.
8. A system for monitoring system performance, characterized in that: The system comprises: a monitoring database configured to store monitoring data; and A monitoring system, wherein the monitoring system is configured to execute the method according to any one of claims 1 to 5.
