Adaptive data compression system, method and device in time series database

Abstract

The adaptive data compression system, method and device in the time series database belong to the field of computer technology. In order to solve the problem that the current compression algorithm for the time series database cannot perform targeted selection and processing on the data and cannot take into account the lightweight requirements at the same time. The system of the present invention includes a time series database plug-in and a data compression subsystem that calls different compression algorithms for compression through an interface based on configuration information; wherein the time series database plug-in is a classification decision model, and the classification decision model includes a feature extraction unit and a classification decision unit that automatically selects the compression algorithm using a neural network model; the data compression subsystem includes a compression strategy configuration unit that feeds back the selected data compression algorithm to the user and writes the compression strategy into the configuration center, and an adaptive data compression unit that creates a new activity to perform data compression tasks and is regularly awakened to perform data sorting and compression tasks. It is mainly used for adaptive compression of data in a time series database.

Description

Technical Field

[0001] The invention relates to an adaptive data compression system, method and equipment, belonging to the technical field of computers. Background Art

[0002] Today we are in an era of big data with astonishing data scale. In recent years, with the promotion of technologies such as artificial intelligence, 5G, and AIoT, the amount of global data is increasing infinitely. In 2018, the total amount of global data was 33ZB, and in 2019 it reached about 45ZB. According to this growth trend, by 2025, 175ZB of data will be generated throughout the year. As of 2020, there will be 50 billion device data on the cloud in the world. These devices cover many practical scenarios, such as: smart life, smart cities, smart agriculture, and more worthy of attention is smart manufacturing, that is, the field of industrial Internet of Things. Under the background of 5G and Industry 4.0, the industrial Internet of Things will also be the next technological trend. We need a new storage technology that specializes in processing time series data to solve it. Time series databases have developed rapidly under this background.

[0003] At present, the storage and processing of time series big data are often handled by relational databases. However, due to the inherent disadvantages of the B+ tree storage architecture of relational databases, it is impossible to efficiently store and query data. The time series big data solution uses a special storage method to enable time series big data to efficiently store and quickly process massive time series big data, which is an important technology for solving massive data processing. This technology uses a special data storage method to greatly improve the processing ability of time-related data. Compared with relational databases, its storage space is halved and the query speed is greatly improved. The superior query performance of time series functions far exceeds that of relational databases. Informix TimeSeries is very suitable for IoT analysis applications.

[0004] After solving the problem of efficient data storage and query, new problems naturally arise. The volume of these time series data will be extremely large, and as time goes by, the value of the data generated earlier will become lower and lower. Therefore, it is necessary to compress the data to a certain extent. Taking InfluxDB, the most stable time series database in the time series database open source community, as an example, the string compression algorithm used is the snappy compression algorithm open sourced by Google. Compared with other compression algorithms, this compression algorithm has an advantage in compression speed and decompression speed, but its data compression ratio is not very good. As the query frequency of data generated earlier decreases, the decompression speed is no longer so important, and the significance of improving the data compression ratio to save storage space becomes more and more important. Summary of the invention

[0005] The present invention aims to solve the problems that the current compression algorithms for time series databases cannot selectively process data in a targeted manner and cannot simultaneously take into account lightweight requirements.

[0006] The adaptive data compression system in the time series database includes:

[0007] A time series database plug-in for determining the compression algorithm type encoding of the final decision, the expected data compression ratio, and the data decompression rate, and a data compression subsystem for compressing by calling different compression algorithms through an interface based on configuration information; wherein,

[0008] The time series database plug-in is a classification decision model; the classification decision model includes a feature extraction unit and a classification decision unit;

[0009] Feature extraction unit: used to obtain user input features, workload features, time series data features and algorithm performance features, and perform preprocessing to obtain feature vector V;

[0010] The classification decision unit uses a neural network model to make classification decisions and realize automatic selection of a compression algorithm;

[0011] The data compression subsystem includes a compression strategy configuration unit and an adaptive data compression unit;

[0012] The compression strategy configuration unit selects a data compression algorithm based on the input user data and the classification decision model corresponding to the sequence database plug-in, and feeds it back to the user, and writes the compression strategy to the configuration center;

[0013] The adaptive data compression unit creates a new activity to perform data compression tasks, and the new activity is awakened periodically to perform data sorting and compression tasks.

[0014] Furthermore, the data compression subsystem also includes a user information receiving unit for receiving a data compression ratio and compression rate set by a user, as well as an applied compressed data range.

[0015] Furthermore, the process of the adaptive data compression unit performing the data sorting and compression task includes the following steps:

[0016] S221: compact_thread reads the configuration of the data compression strategy of the configuration center. If the data is not configured within the corresponding time period, the default configuration strategy is executed, and a default data compression algorithm is selected for compression; otherwise, the compression strategy of the configuration center is selected for compression;

[0017] S222: according to different compression strategy configurations, the data is divided into different time periods, and the data sets are read in different time periods;

[0018] S223: Decompress the data according to the current data compression format to obtain the source data, re-compress it according to the new data compression strategy configuration, and write it back to the data set.

[0019] Furthermore, the user input characteristics include: planned compression time period, user expected data compression ratio; the workload characteristics include: execution frequency of data workload in the corresponding time period; the time series data characteristics include: cardinality, mode; the algorithm performance characteristics include: data decompression rate.

[0020] Furthermore, the preprocessing process is to obtain input data information, perform normalization processing, and organize it into a feature vector V.

[0021] Furthermore, the neural network model of the classification decision unit includes a first convolutional layer, a second convolutional layer, a maximum pooling layer, a Flatten layer, a fully connected layer and an output layer.

[0022] Preferably, the network parameters of the neural network model are as follows:

[0023] The network parameters of the first convolutional layer and the second convolutional layer are the same: filters = 16, kernel_size = 3, and the activation function is tanh;

[0024] The pool_size of the maximum pooling layer is 3;

[0025] The dropout of the fully connected layer is 0.1, and the activation function is relu;

[0026] The output layer has dropout=0.1 and the activation function is sigmiod.

[0027] The adaptive data compression method in the time series database includes the following steps:

[0028] The user proposes the data compression ratio and compression rate as well as the compressed data range to be applied;

[0029] Then the adaptive data compression system in the time series database obtains the user information and performs adaptive data compression.

[0030] A storage medium stores at least one instruction, which is loaded and executed by a processor to implement an adaptive data compression system in a time series database.

[0031] An adaptive data compression device in a time series database, the device comprises a processor and a memory, the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement an adaptive data compression system in the time series database.

[0032] Beneficial effects:

[0033] 1. The data compression ratio and decompression rate of different compression algorithms vary greatly. Multiple compression algorithms are built into the database to cope with more diverse scenarios with different requirements and adopt different compression strategies for data.

[0034] 2. A compression algorithm decision model is built into the database. Due to limited training data, the model is very lightweight, so results can be obtained quickly both in training and use without consuming too many resources.

[0035] 3. Dynamic configuration of compression algorithms can help users more easily select different compression strategies for data according to their own needs, balance the performance of different features in different business scenarios, and maximize resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a processing flow chart of a feature extraction unit;

[0037] Figure 2 It is a schematic diagram of the neural network model structure of the classification decision unit;

[0038] Figure 3 Schematic diagram of the processing process of the adaptive data compression method. DETAILED DESCRIPTION

[0039] The current time series database uses only one or one type of method by default to compress data. Often, for the sake of query performance, the decompression speed of the compression algorithm is more important. However, as the data "ages", its value will decrease over time, and the query frequency will also decrease. At the same time, the data volume caused by its low compression ratio is very large, so a dynamic compression method is needed. According to the needs of different scenarios, the data compression algorithm is used reasonably by comprehensively considering the decompression rate and data compression ratio. The present invention provides an adaptive data compression method in a time series database, which can select an adaptive compression algorithm in the time series database. Specific implementation method one:

[0041] This embodiment is an adaptive data compression method in a time series database, comprising the following steps:

[0042] S1. Classification decision model design:

[0043] The classification decision model is used to extract the feature vectors of the data in the time series database and decide the type of compression algorithm encoding as well as the expected data compression ratio and data decompression rate.

[0044] The classification decision model can be used as a lightweight time series database plug-in. The data compression algorithm that can be implemented is very limited. Therefore, the classification decision model implemented by the present invention is also very lightweight, so results can be obtained quickly both in training and use without consuming too many resources. At the same time, the update frequency of the classification decision model can be dynamically adjusted according to the workload of the database.

[0045] The classification decision model includes a feature extraction unit and a classification decision unit;

[0046] (1) Feature extraction unit:

[0047] The present invention needs to make compression algorithm decisions in different business scenarios, so it is necessary to consider a variety of environmental factors. The feature extraction unit is used to obtain user input features, workload features, time series data features and algorithm performance features, and perform preprocessing, such as Figure 1 shown.

[0048] Input: user input characteristics, workload characteristics, time series data characteristics, and algorithm performance characteristics;

[0049] The user input features include: planned compression time period, user expected data compression ratio;

[0050] The workload characteristics include: execution frequency of data workload in corresponding time period;

[0051] The time series data features include: cardinality and mode;

[0052] The algorithm performance characteristics include: data decompression rate.

[0053] Output: processed feature vector containing the meaning of each attribute.

[0054] Preprocessing process: normalize the acquired input data information and organize it into a feature vector V;

[0055] The normalization operation can be defined as mapping the data to the interval [0, 1]. The calculation formula is as follows, where I is the original data, and I new is the normalized data.

[0056]

[0057] (2) Classification decision unit:

[0058] Use neural network models to make classification decisions and realize automatic selection of compression algorithms;

[0059] Input: feature vector V;

[0060] Output: The final compression algorithm type encoding and the expected data compression ratio and data decompression rate.

[0061] Neural Network Design:

[0062] The neural network for selecting the compression algorithm is a convolutional neural network. Figure 2 As shown in FIG, it includes the first convolutional layer (convolutional layer 1), the second convolutional layer (convolutional layer 2), the maximum pooling layer, the Flatten layer, the fully connected layer and the output layer; the network parameter settings are shown in Table 1.

[0063] Table 1

[0064]

[0065] The network parameters of the present invention can not only ensure the lightweight of the time series database plug-in, so that results can be obtained quickly in both training and use without consuming too many resources; more importantly, the selection of such parameters can also ensure the effectiveness of the selection (the determined compression algorithm is effective, that is, the relative accuracy of the selection result) while ensuring the lightweight and fast rate of the neural network model.

[0066] S2. The processing process of the compression algorithm decision model is as follows:

[0067] We first designed interface calls for various compression algorithms in the database and designed a configuration center to store the configurations of different data compression strategies.

[0068] S21, Adaptive data compression algorithm configuration process:

[0069] S211. The user proposes the expected data compression ratio and compression rate within a reasonable range and the applied compressed data range according to the characteristics or needs of his / her own data, such as: all data before a certain date. The reasonable range can be defined as the range that meets the following constraints:

[0070] 0≤Ratio≤4.0

[0071] 50MB / s≤Compression≤1000MB / s

[0072] Among them, Ratio is the data compression ratio, and Compression is the compression rate;

[0073] S212: The classification decision model is ready and can make decisions based on the parameters passed in by the user, select the data compression algorithm that best meets the user's expectations, and provide feedback to the user on which data compression algorithm he / she has selected. At the same time, the compression strategy is written into the configuration center.

[0074] S22, Adaptive data compression algorithm application process:

[0075] Since the database main activity is mainly used to monitor external interface calls, the task will be very heavy, so we will create a new activity to perform data compression tasks. The new activity will be awakened regularly to perform data sorting and compression tasks.

[0076] S221: compact_thread reads the configuration of the data compression strategy of the configuration center. If the data is not configured in the corresponding time period, the default configuration strategy is executed. At this time, a default data compression algorithm is selected. For example, for string type data, the snappy compression algorithm is used by default. Otherwise, compression is performed according to the specified compression strategy.

[0077] S222: According to different compression strategy configurations, the data is divided into different time periods, and the data is read in the data sets in different time periods. The time periods here can be defined by the user according to the characteristics of their own data, such as one week or 24 hours.

[0078] S223: Decompress the data according to the current data compression format to obtain the source data, re-compress it according to the new data compression strategy configuration, and write it back to the data set.

[0079] For example: the original data is [1 2 3 4 5], and the current compression algorithm is simple8b. Therefore, the original data format before compression is [16 0 0 0 0 0 0 0 2 192 0 64 0 128 1 0 2]. According to the simple8b decompression method, the original data is obtained, and then compressed using the rle compression algorithm. The compressed format is [32 0 0 0 0 0 0 0 2 2 4].

[0080] At the same time, the compression system designed according to the above process is the adaptive data compression system in the time series database. After the adaptive data compression system in the time series database is designed, the adaptive data compression of the data can be realized. Figure 3 shown.

[0081] The present invention has the following advantages:

[0082] 1. Dynamic configuration of data compression algorithm: This dynamic configuration method allows data in different time periods and scenarios to select a more appropriate data compression algorithm according to demand, balance data compression ratio and decompression rate, and further improve resource utilization.

[0083] 2. Deep learning model for compression algorithm decision-making: This model allows users to choose data compression strategies according to their needs in an almost "foolproof" way. At the same time, because the model can learn from the experience of multiple benchmarks, it can make decisions more effectively than manual control. Specific implementation method 2:

[0085] This embodiment is a storage medium, in which at least one instruction is stored. The at least one instruction is loaded and executed by a processor to implement the adaptive data compression system in the time series database.

[0086] It should be understood that any method described in the present invention may be provided as a computer program product, software or computerized method, which may include a non-transitory machine-readable medium having instructions stored thereon, which may be used to program a computer system, or other electronic device, to perform the process. The storage medium may include, but is not limited to, magnetic storage media, optical storage media; magneto-optical storage media include: read-only memory ROM, random access memory RAM, erasable programmable memory (e.g., EPROM and EEPROM) and flash memory layers; or other types of media suitable for storing electronic instructions. Specific implementation method three:

[0088] This embodiment is an adaptive data compression device in a time series database, and the device includes a processor and a memory. It should be understood that the device includes any device including a processor and a memory described in the present invention, and the device may also include other units and modules that perform display, interaction, processing, control, etc. and other functions through signals or instructions;

[0089] At least one instruction is stored in the memory, and the at least one instruction is loaded and executed by the processor to implement the adaptive data compression system in the time series database.

[0090] The present invention may also have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.

Claims

1. An adaptive data compression system in a time series database, characterized in that: include: A time series database plug-in for determining the compression algorithm type encoding of the final decision, the expected data compression ratio, and the data decompression rate, and a data compression subsystem for compressing by calling different compression algorithms through an interface based on configuration information; wherein, The time series database plug-in is a classification decision model; the classification decision model includes a feature extraction unit and a classification decision unit; Feature extraction unit: used to obtain user input features, workload features, time series data features and algorithm performance features, and perform preprocessing to obtain feature vector V; The classification decision unit uses a neural network model to make classification decisions and realize automatic selection of a compression algorithm; The data compression subsystem includes a compression strategy configuration unit, an adaptive data compression unit, and a user information receiving unit; The compression strategy configuration unit selects a data compression algorithm based on the input user data and the classification decision model corresponding to the time series database plug-in, feeds it back to the user, and writes the compression strategy to the configuration center; The adaptive data compression unit creates a new activity to perform the data compression task. The new activity is awakened periodically to perform the data sorting and compression task. The process of performing the compression task includes the following steps: S221: compact_thread reads the configuration of the data compression strategy of the configuration center. If the data is not configured within the corresponding time period, the default configuration strategy is executed, and a default data compression algorithm is selected for compression; otherwise, the compression strategy of the configuration center is selected for compression; S222: according to different compression strategy configurations, the data is divided into different time periods, and the data sets are read in different time periods; S223: decompress the data according to the current data compression format to obtain the source data, re-compress it according to the new data compression strategy configuration and write it back to the data set; The user information receiving unit is used to receive the data compression ratio and compression rate set by the user, as well as the applied compressed data range.

2. The adaptive data compression system in a time series database according to claim 1, characterized in that: The user input features include: planned compression time period, user expected data compression ratio; the workload features include: execution frequency of data workload in the corresponding time period; the time series data features include: cardinality, mode; the algorithm performance features include: data decompression rate.

3. The adaptive data compression system in a time series database according to claim 2, characterized in that: The preprocessing process is to obtain input data information, perform normalization processing, and organize it into a feature vector V.

4. The adaptive data compression system in a time series database according to claim 3, characterized in that: The neural network model of the classification decision unit includes a first convolutional layer, a second convolutional layer, a maximum pooling layer, a Flatten layer, a fully connected layer and an output layer.

5. The adaptive data compression system in a time series database according to claim 4, characterized in that: The network parameters of the neural network model are as follows: The network parameters of the first convolutional layer and the second convolutional layer are the same: filters = 16, kernel_size = 3, and the activation function is tanh; The pool_size of the maximum pooling layer is 3; The dropout of the fully connected layer is 0.1, and the activation function is relu; The output layer has dropout=0.1 and the activation function is sigmiod.

6. An adaptive data compression method in a time series database, characterized in that: The following steps are involved: The user proposes the data compression ratio and compression rate as well as the compressed data range to be applied; Then the adaptive data compression system in the time series database described in claim 5 obtains user information and performs adaptive data compression.

7. A storage medium, characterized in that: The storage medium stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the adaptive data compression system in the time series database as described in any one of claims 1 to 5.

8. An adaptive data compression device in a time series database, characterized in that: The device includes a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the adaptive data compression system in the time series database as described in any one of claims 1 to 5.

Images