Database monitoring operation and maintenance method and system based on time sequence data compression

Abstract

The invention provides a time sequence data compression-based database monitoring operation and maintenance method and system, which realize real-time monitoring of various index data of a database platform through an efficient data acquisition and compression technology. The system can efficiently collect various performance data generated in the operation process of the database, optimizes time series data through a compression algorithm, and reduces redundant storage of the data. In a data storage and management process, a primary differential sequence and a secondary differential sequence are calculated, so that the stored data volume is reduced; according to the method, a trend in an original data sequence is removed by using difference, so that a subsequent difference sequence is easier to compress, and meanwhile, for non-constant difference, by setting a dynamic threshold value and only recording a difference value greater than the threshold value, small changes are ignored and the compression ratio is improved on the premise of ensuring that important data information is not lost. And on the other hand, the regularity of the differential sequence is judged through Srelative, and a compression strategy can be selectively applied, so that proper compression methods can be specifically applied to time sequence data with different characteristics, and the processing efficiency is improved. Through the design, the storage and processing efficiency of the compressed data is improved, the system can decompress and analyze the data more quickly, and the running state of the database is monitored in real time.

Description

Technical Field

[0001] The invention relates to a database monitoring operation and maintenance method and system based on time series data compression, belonging to the field of database operation and maintenance. Background Art

[0002] In today's information-based world, more and more data is automatically collected by various monitoring devices, and the scale of the generated time series data continues to expand, which has brought tremendous pressure to the storage and management of time series data. In actual production environments, the disk throughput rate of traditional databases will gradually become its overall performance bottleneck. In order to ensure the read and write performance of the database system, using memory to cache recent data and matching it with disk persistent storage has become the current mainstream response solution. At the same time, with the rapid increase in the number of monitoring devices, the current existing operation and maintenance management platform has also encountered new pressures and challenges.

[0003] There are many compression technologies for time series data that can significantly reduce storage space and data transmission costs, but these technologies cannot always maintain high compression rates for all features and patterns of data. Existing high compression ratio technologies often require large-scale decompression of data during database queries, or even decompression of the entire data set, which cannot meet the actual retrieval needs of the database. Open TSDB uses a dictionary-like compression algorithm to reduce memory usage by encoding each label in the index name and each sequence name. However, it still has many useless fields that cannot be effectively compressed, and its aggregation ability is weak. The Delta-of-delta algorithm is a variable-length compression algorithm. When decompressing, it must be calculated sequentially from the first address of the data block, which increases the complexity of decompression. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Figure 1 It is a system flow chart of the present invention. Summary of the invention

[0005] In view of the above problems, the present invention proposes a database monitoring and operation method based on time series data compression, which realizes real-time monitoring of various indicator data of the database platform through efficient data collection and compression technology. The system will be able to efficiently collect various performance data generated during the operation of the database, and optimize the time series data through compression algorithms to reduce redundant storage of data. The method includes the following steps:

[0006] 1. Collect data: Collect relevant data from the database, including performance indicators, database activities, and log information.

[0007] 2. Database operation and maintenance management: compress the collected time series data, analyze the database operation status in real time, discover potential problems in time and issue early warnings.

[0008] 3. Display operation and maintenance status: Display the operation status of the database to the user in a visual form.

[0009] Furthermore, in step (2), when the data storage and management submodule compresses the timestamp of the time series data, the following steps are included:

[0010] 3.1 Input original time series data T = [t1, t2, t3, ... t n ,…t N ], calculate the difference between adjacent data points, where N is the number of data points, n∈[1,N], and obtain the first-level difference sequence ΔT=[δ1,δ2,…,δ N-1 ], where δ i =t i+1 -t i , i=1,2,…,N-1;

[0011] 3.2 Calculation in is the sample mean of the time series;

[0012] 3.3 If S relaction =1, the first-level difference sequence ΔT is a constant difference sequence, and the initial value t1, the difference value δ1 of the first-level difference sequence, and the number of sequences N are recorded and stored;

[0013] 3.4 If S relaction ≠1, the first-order difference sequence ΔT is a non-constant difference, using Δ adaptive Encoded as a threshold: |δ i |≥Δ adaptive When i , |δ i |<Δ adaptive When δ i ,in Standard deviation of ΔT α and β are weight parameters; the secondary difference sequence ΔT2 = [γ1,γ2,…,γ n-2 ], where γ i =δ i+1 -δ i , i=1,2,…,n-2; perform Huffman encoding on the secondary differential sequence.

[0015] Based on the above method, the present invention further proposes a database monitoring and operation system based on time series data compression, which includes the following modules:

[0016] 1. Data collection module: This module collects relevant data from the database, including performance indicators, database activities and log information.

[0017] 2. Database operation and maintenance management module: This module consists of data storage and management sub-module, data decompression sub-module, real-time analysis and early warning module sub-module. It compresses the collected time series data, analyzes the database operation status in real time, promptly detects potential problems and issues early warnings.

[0018] 3. Operation and maintenance status display module: This module displays the operation status of the database to the user in a visual form.

[0019] Furthermore, in the database operation and maintenance management module, when the data storage and management submodule compresses the timestamp of the time series data, the following steps are included:

[0020] 3.1 Input original time series data T = [t1, t2, t3, ... t n ,…t N ], calculate the difference between adjacent data points, where N is the number of data points, n∈[1,N], and obtain the first-level difference sequence ΔT=[δ1,δ2,…,δ N-1 ], where δ i =t i+1 -t i , i=1,2,…,N-1;

[0021] 3.2 Calculation in is the sample mean of the time series;

[0022] 3.3 If S relaction =1, the first-level difference sequence ΔT is a constant difference sequence, and the initial value t1, the difference value δ1 of the first-level difference sequence, and the number of sequences N are recorded and stored;

[0023] 3.4 If S relaction ≠1, the first-order difference sequence ΔT is a non-constant difference, using Δ adaptive Encoded as a threshold: |δ i |≥Δ adaptive When i , |δ i |<Δ adaptive When δ i ,in ,

[0024] Standard deviation of ΔT α and β are weight parameters; the secondary difference sequence ΔT2 = [γ1,γ2,…,γ n-2 ], where γ i =δ i+1 -δ i, i=1,2,…,n-2; perform Huffman encoding on the secondary differential sequence.

[0025] After completing the above compression algorithm, the initial value t1 and the first difference value δ1 of the first-level differential sequence are stored, and a suitable data storage structure is adopted, using relational database, time series database or distributed file system storage technology for data storage, so as to achieve efficient compressed data storage and fast retrieval.

[0026] The present invention reduces the amount of stored data by calculating the primary difference sequence and the secondary difference sequence during the data storage and management process; uses the difference to remove the trend in the original data sequence, making the subsequent difference sequence easier to compress. At the same time, for non-constant differences, by setting a dynamic threshold, only the difference value greater than the threshold is recorded, and minor changes are ignored to improve the compression rate while ensuring that important data information is not lost. On the other hand, through S relation By judging the regularity of the differential sequence, the compression strategy can be selectively applied, so that the appropriate compression method can be applied to time series data with different characteristics in a targeted manner to improve processing efficiency. Through the above design, the storage and processing efficiency of compressed data is improved, the system can decompress and analyze data more quickly, and realize real-time monitoring of the database operation status. DETAILED DESCRIPTION

[0029] In order to achieve real-time monitoring of various indicator data of the database platform and compress time series data to reduce redundant data storage, the present invention proposes a database monitoring and operation system based on time series data compression. The platform is mainly composed of the following parts: data acquisition is responsible for obtaining various key indicators and operation status data from the database platform; database operation and maintenance management is responsible for receiving, storing and analyzing data transmitted from the status acquisition device; operation and maintenance status display is a graphical display interface for operation and maintenance personnel and management personnel, which presents complex monitoring data and analysis results through intuitive charts and dashboards. The main process of the system is as follows: Figure 1 The system process design is as follows:

[0030] 1. Data acquisition module:

[0031] This module obtains various key indicators and operating status data from the database platform. It collects database performance parameters, resource usage, network traffic, error logs and other information in real time through sensors, data collectors and monitoring probes.

[0032] 2. Database operation and maintenance management module

[0033] (1) Data storage and management

[0034] According to the data type and purpose, the collected data is classified, including performance parameters, resource usage, network traffic, error logs and other information. The collected data is segmented based on the time standard:

[0035] Determine the length of time each data segment contains. Starting from the start time of the time series data, collect the data points belonging to this time period into a data segment according to the monthly cycle. Place the first cycle at the starting time point, and slide the window from the starting time until the entire time range is covered. Each time you move a step, a new data segment is generated, and the data stream is processed continuously. For each data segment, record its start time, end time, and the number of data points it contains.

[0036] In addition, the data storage and management module of the platform has designed a compression scheme for the timestamp of time series data, as shown below:

[0037] a. Input original time series data T = [t1, t2, t3, ... t n ,…t N ], calculate the difference between adjacent data points, where N represents the number of data points, n∈[1,N], and obtain the first-level difference sequence ΔT=[δ1,δ2,…,δ N-1 ], where δ i =t i+1 -t i , i=1,2,…,N-1.

[0038] b. For the first-order difference sequence, determine the regularity of the difference, through is the sample mean of the time series. The regularity of the difference series is analyzed based on the result of this function.

[0039] c. If S relaction =1, the sequence is a constant difference sequence, then the initial value t1, the difference value δ1 of the first-level difference sequence and the number of sequences N are recorded and stored.

[0040] d. If S relaction ≠1, the sequence is a non-constant difference: then use Δ adaptive Encoded as a threshold: when |δ i |≥Δ adaptive , then record δ i , when |δ i |<Δ adaptive Ignore δ i , the dynamic threshold formula is: ,

[0041] in is the average value, α and β are weight parameters;

[0042] Then calculate the adjacent differences of the first-level difference sequence to obtain the second-level difference sequence:

[0043] ΔT2=[γ1,γ2,…,γ n-2 ], where γ i =δ i+1 -δ i , i = 1, 2, ..., n-2. Huffman encoding is performed on the secondary differential sequence to further reduce data redundancy.

[0044] After completing the above compression algorithm, store the initial value t1 and the first difference value δ1 of the first-level difference sequence, design a suitable data storage structure, and use storage technologies such as relational databases, time series databases, or distributed file systems.

[0045] (2) Data decompression

[0046] The constant difference data decompression steps are as follows: input the initial value t1 of the compressed information, the difference value δ1, and the sequence length N;

[0047] a. Iteratively recover the sequence and iteratively calculate each subsequent data point: For each data point t i+1 According to the previous data point t i , calculate: t i+1 =t i +δ1, and repeat this step until all data points in the sequence are calculated.

[0048] b. After completing the above iterative calculation, output the restored original sequence.

[0049] Decompression of non-constant differential data: To recover the original time series data from the compressed data, the following steps are required:

[0050] a. Input the compressed data storage initial value t1, the first difference value δ1 of the first-level difference sequence, and the second-level difference sequence after Huffman encoding.

[0051] b. Decode the Huffman-encoded data and restore the secondary differential sequence ΔT2.

[0052] c. Restore the first-order differential sequence δ1=δ1,δ i+1 =δ i +γ i , where i=1,2,...,n-2.

[0053] d. Restore the original time series data: t1 = t1, t i+1 =t i +δ i , where i=1,2,...,n-1.

[0054] By dynamically adjusting the threshold according to the data characteristics, the algorithm can better adapt to data with different characteristics, improve compression efficiency and data restoration accuracy, and improve data compression ratio and data fidelity.

[0055] (3) Real-time analysis and early warning

[0056] Utilize machine learning algorithms and rule engines to analyze database operation status in real time, detect potential problems promptly and issue warnings.

[0057] 3. Operation and maintenance status display module

[0058] Intuitive chart and dashboard interface displays complex monitoring data and analysis results, mainly including the following parts:

[0059] Real-time monitoring: displays the real-time operation status of the database, including key indicators such as CPU usage, memory usage, disk IO, network traffic, etc.

[0060] Historical data analysis: Trend analysis of historical data to show the changing patterns of database performance;

[0061] Early warning notification: When the monitoring data exceeds the preset threshold, the early warning function is displayed through the interface to notify relevant personnel.

[0062] In response to the needs of in-memory time series databases, the database operation and maintenance management module has designed a compression scheme for the timestamps of time series data. By dynamically adjusting the threshold and multi-level differential encoding, the data compression efficiency and restoration accuracy are improved. It is suitable for processing a large amount of high-frequency and complex time series data, effectively reducing data storage costs. The operation and maintenance status display system uses intuitive charts and dashboards to display the real-time operating status of the database and trend analysis of historical data, helping operation and maintenance personnel understand the changing patterns of database performance. Through the organic combination of data acquisition magic, database operation and maintenance management module and operation and maintenance status display, the database monitoring and operation and maintenance platform of the present invention can realize comprehensive, real-time and intelligent monitoring and operation and maintenance management of the database platform. A more specific embodiment:

[0063] In order to better understand the practical application of the database monitoring and operation system, the whole process is described in detail below through a specific embodiment, starting from the acquisition of data by the data acquisition module to the compression and decompression of data.

[0064] Assume that status collection devices are running on the database platform. These devices monitor the following key indicators in real time: CPU usage, memory usage, network traffic, etc.

[0065] In this example, assuming that the collected indicator is the CPU usage (%) per minute, the original time series data obtained is:

[0066] T=[("2024-05-2700:00:00",45),("2024-05-2700:01:00",48),("2024-05-2700:02:00",50),("2024-05-2700:03 :00",55),("2024-05-2700:04:00",60),("2024-05-2700:05:00",58),("2024-05-2700:06:00",56),("2024-05-2 700:07:00",54),("2024-05-2700:08:00",57),("2024-05-2700:09:00",59),("2024-05-2700:10:00",63),("2024-05-2700:11:00",65),("2024-05-2700:12:00",68),("2024-05-2700:13:00",70),("2024-05-2700:14:00",69)]

[0067] First convert the timestamp to the number of seconds since the start time, and then calculate the first-level difference sequence of the timestamp.

[0068] The start time "2024-05-27 00:00:00" is converted to the timestamp of 0 seconds, and other timestamps are incremented by 60 seconds.

[0069] T seconds =[0,60,120,180,240,300,360,420,480,540,600,660,720,780,840]

[0070] Calculate the first-order difference sequence of timestamps: ΔT ts =[60,60,60,60,60,60,60,60,60,60,60,60,60,60,60]

[0071] The first-order difference sequence of calculated values: ΔV = [3, 2, 5, 5, -2, -2, -2, 3, 2, 4, 2, 3, 2, -1]

[0072] Calculate the dynamic threshold Δ adaptive , assuming α = 0.5, β = 0.5, calculate and σ T ≈7.43, we get Δ adaptive ≈2.

[0073] Encode the first-level difference sequence, when |δ i |≥Δ adaptive , then record δi , that is, ΔT′=[3,5,5,-2,-2,3,-5,3,4,3]; calculate the headphone differential sequence ΔT2=[2,0,-7,0,5,-8,8,1]; perform Huffman encoding on the secondary differential sequence ΔT2 to reduce data redundancy. This part of the encoding process can be implemented using the Huffman encoding algorithm.

[0076] The operation and maintenance status display system graphically displays the data obtained from the operation and maintenance management platform. It uses various charts and dashboards to display data such as CPU usage, memory usage, network traffic, and error logs.

[0077] The present invention realizes efficient data compression and transmission through the above embodiments, introduces dynamic thresholds, and adjusts relevant parameter values ​​according to actual data fluctuations. For example, assuming α=0.5 and β=0.5 in the embodiment, the data compression ratio will be improved. Compared with the traditional time series data compression method and database operation and maintenance system, the data compression method of the present invention is fast. In the present invention, only the secondary differential sequence ΔT2 after Huffman encoding needs to be stored, and the original time series data is:

[0078] T=[("2024-05-2700:00:00",45),("2024-05-2700:01:00",48),("2024-05-2700:02:00",50),("2024-05-2700:03 :00",55),("2024-05-2700:04:00",60),("2024-05-2700:05:00",58),("2024-05-2700:06:00",56),("2024-05-27 00:07:00",54),("2024-05-2700:08:00",57),("2024-05-2700:09:00",59),("2024-05-2700:10:00",63),("2024-05-2700:11:00",65),("2024-05-2700:12:00",68),("2024-05-2700:13:00",70),("2024-05-2700:14:00",69)]. Compared with the two, the algorithm greatly reduces the demand for memory and significantly improves the performance and efficiency of the database monitoring and operation system.

[0079] The units, devices or modules described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. For the convenience of description, the above devices are described separately by functions divided into various modules. Of course, when implementing this application, the functions of each module can be implemented in the same or more software and / or hardware, or the modules that implement the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0080] Those skilled in the art also know that, in addition to implementing the controller in a purely computer-readable program code, the controller can be made to implement the same function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered as a hardware component, and the devices for implementing various functions included therein can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules for implementing the method and structures within the hardware component.

[0081] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0082] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0083] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0084] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present application. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A database monitoring and operation method based on time series data compression, characterized in that: The method comprises the following steps:

1. Collect data: collect relevant data from the database, including performance indicators, database activities and log information; 2. Database operation and maintenance management: compress the collected time series data, analyze the database operation status in real time, discover potential problems in time and issue early warnings; 3. Display operation and maintenance status: Display the operation status of the database to the user in a visual form.

2. A database monitoring and operation method based on time series data compression as claimed in claim 1, characterized in that: In step (2), when the data storage and management submodule compresses the timestamp of the time series data, the following steps are included: 3.1 Input original time series data T = [t1, t2, t3, ... t n ,…t N ], calculate the difference between adjacent data points, where N is the number of data points, n∈[1,N], and obtain the first-level difference sequence ΔT=[δ1,δ2,…,δ N-1 ], where δ i =t i+1 -t i , i=1,2,…,N-1; 3.2 Calculation in is the sample mean of the time series; 3.3 If S relaction =1, the first-level difference sequence ΔT is a constant difference sequence, and the initial value t1, the difference value δ1 of the first-level difference sequence, and the number of sequences N are recorded and stored; 3.4 If S relaction ≠1, the first-order difference sequence ΔT is a non-constant difference, using Δ adaptive Encoded as a threshold: |δ i |≥Δ adaptive When i , |δ i |<Δ adaptive When δ i ,in , standard deviation of ΔT α and β are weight parameters; the secondary difference sequence ΔT2 = [γ1,γ2,…,γ n-2 ], where γ i =δ i+1 -δ i , i=1,2,…,n-2; perform Huffman encoding on the secondary differential sequence.

3. A database monitoring and operation system based on time series data compression, characterized in that: The system includes the following modules:

1. Data collection module: This module collects relevant data from the database, including performance indicators, database activities and log information; 2. Database operation and maintenance management module: This module consists of data storage and management submodule, data decompression submodule, and real-time analysis and early warning module submodule. It compresses the collected time series data, analyzes the database operation status in real time, discovers potential problems in time and issues early warnings; 3. Operation and maintenance status display module: This module displays the operation status of the database to the user in a visual form.

4. A database monitoring and operation and maintenance system based on time series data compression as claimed in claim 3, characterized in that: In the database operation and maintenance management module, when the data storage and management submodule compresses the timestamp of the time series data, the following steps are included: 3.1 Input original time series data T = [t1, t2, t3, ... t n ,…t N ], calculate the difference between adjacent data points, where N is the number of data points, n∈[1,N], and obtain the first-level difference sequence ΔT=[δ1,δ2,…,δ N-1 ], where δ i =t i+1 -t i , i=1,2,…,N-1; 3.2 Calculation in is the sample mean of the time series; 3.3 If S relaction =1, the first-level difference sequence ΔT is a constant difference sequence, and the initial value t1, the difference value δ1 of the first-level difference sequence, and the number of sequences N are recorded and stored; 3.4 If S relaction ≠1, the first-order difference sequence ΔT is a non-constant difference, using Δ adaptive Encoded as a threshold: |δ i |≥Δ adaptive When i , |δ i |<Δ adaptive When δ i ,in , standard deviation of ΔT α and β are weight parameters; the secondary difference sequence ΔT2 = [γ1,γ2,…,γ n-2 ], where γ i =δ i+1 -δ i , i=1,2,…,n-2; perform Huffman encoding on the secondary differential sequence.

Images