Lossless compression method for estimating compression ratio of compression algorithm based on time sequence data characteristic value in database

Abstract

The invention discloses a lossless compression method and device for estimating a compression ratio of a compression algorithm based on a time sequence data characteristic value in a database. The method comprises the following steps: acquiring time sequence data blocks grouped according to columns; extracting characteristic values of the data blocks; aiming at a plurality of lossless compression algorithms, calculating each algorithm estimation compression ratio based on a preset compression ratio estimation formula and the characteristic value; selecting the algorithm with the highest estimation compression ratio as a target algorithm; and performing lossless compression on the data block by utilizing the target algorithm. According to the invention, through a lightweight estimation mechanism, high cost and high overhead of an AI model are avoided, and adaptive compression is realized with low calculation overhead; the time sequence data compression ratio is effectively improved (the time sequence data is close to the theoretical limit when the potential data is high and reaches the highest compression ratio of 75% when the potential is low), the compression potential is fully explored, the storage cost is remarkably reduced, and high efficiency and practicability are achieved.

Description

Technical Field

[0001] This application belongs to the field of data compression technology, and specifically relates to a method, apparatus, computer-readable storage medium, and electronic device for lossless compression of a database based on the compression ratio of a compression algorithm estimated by time-series data feature values. Background Technology

[0002] In the context of the information age, the rapid development of fields such as the Internet of Things (IoT), the Industrial Internet, environmental monitoring, and artificial intelligence (AI) has driven an explosive growth in time-series data. This type of data is characterized by high-frequency generation, massive volume (up to terabytes in a single scenario), and the overlapping of multiple business scenarios further exacerbates the capacity pressure on storage systems. High-efficiency compression technology has become a key path to reduce storage costs, and by achieving compression rates of several to tens of times, storage resource utilization can be significantly improved.

[0003] Currently, mainstream time-series databases generally employ columnar storage architectures to manage time-series data. The inherent high degree of structure (regular format), strong temporal correlation, and significant compression potential of time-series data make it an ideal target for efficient compression. For numerical data (which constitutes the majority of time-series data), existing technologies have developed various classic lossless compression algorithms (such as run-length encoding / RLE, incremental encoding / DELTA, second-order difference / DELTA2, SIMPLE8B, bit-packing / BITPACKING, and variable-length integer / VARINT). However, mainstream time-series database compression mechanisms still have significant limitations. The aforementioned algorithms all rely on specific data distribution characteristics (such as stationarity, continuity, and sparsity) to achieve optimal results, lacking the ability to dynamically adapt to different data characteristics. In industrial practice, database systems are typically statically bound to a single compression algorithm, failing to intelligently switch strategies based on real-time data characteristics, resulting in limited compression efficiency and the untapped potential compression space of time-series data.

[0004] Some cutting-edge research attempts to introduce artificial intelligence (AI) models to predict data features and dynamically select compression algorithms. While such solutions can theoretically achieve adaptive compression, their high deployment costs (requiring the training of complex models) and significant runtime overhead (real-time inference computation load) limit their practical application. Summary of the Invention

[0005] To address the aforementioned problems in the existing technology, this application proposes a novel method for lossless compression of databases based on the compression ratio estimation algorithm using time-series data feature values.

[0006] The purpose of this invention is to provide a method for lossless compression based on the compression ratio estimation of compression algorithms using time-series data feature values. Through a lightweight estimation mechanism, with low computational overhead and implementation cost, the method dynamically selects an appropriate lossless compression algorithm based on the characteristics of time-series data to perform compression, thereby effectively improving the compression ratio of time-series data and achieving adaptability to different data characteristics.

[0007] In summary, this invention proposes a lightweight alternative: based on the rapid extraction of key statistical features of time-series data (such as variance, entropy, and sequence correlation), a compression ratio prediction model is constructed by combining the mathematical principles of various compression algorithms. By calculating the expected compression ratio of different algorithms on the target data, the optimal algorithm is dynamically selected for lossless compression. This mechanism avoids the complexity of AI models, is simple to implement, and achieves compression results close to the theoretical limit with extremely low computational overhead, combining high efficiency and practicality.

[0008] Specifically, this application provides the following technical solutions: The first aspect of this application provides a method for lossless compression of a database based on estimating the compression ratio of a compression algorithm using time-series data feature values, such as... Figure 3 As shown, the method includes: S1. Obtain a data block of time-series data, wherein the data block is a subset of time-series data grouped by columns; S2. Extract at least one time-series data feature value from the data block; S3. For a variety of preset lossless compression algorithms, based on the compression ratio estimation formula corresponding to each algorithm, the estimated compression ratio of the data block under each algorithm is calculated using the feature value; S4. Select the algorithm with the highest estimated compression ratio from the various lossless compression algorithms as the target compression algorithm; S5. Call the target compression algorithm to perform lossless compression on the data block, and record the selected algorithm identifier in the metadata of the compressed data block.

[0009] Furthermore, in the method of this application, the data block mentioned in step S1 is the largest data unit after being grouped by column, and each data block contains a preset number of time-series data points. Furthermore, in the method of this application, the lossless compression algorithms mentioned in step S3 include at least two of RLE (run-length encoding algorithm), SIMPLE8B algorithm, BITPACKING (bit packing algorithm), VARINT (variable-length integer encoding algorithm), DELTA (first-order differential encoding algorithm), and DELTA2 (second-order differential encoding algorithm); The time series data feature values ​​extracted in step S2 include: data repetition rate, data average value, data maximum value, data minimum value, first-order difference value, and second-order difference value.

[0010] Furthermore, in the method of this application, when the data block contains negative values, before using the SIMPLE8B algorithm, BITPACKING algorithm, or VARINT algorithm for compression, the data is first subjected to a ZIGZAG transformation to map the signed integer to an unsigned integer.

[0011] Furthermore, in the method of this application, the compression ratio estimation formula of the RLE algorithm is: C<(valueSize×R) / ((2×valueSize+1)(1-R)); Where C is the compression ratio, valueSize is the byte length of the data type, and R is the data repetition rate, defined as the statistical proportion of consecutive repetitions of values, and 0≤R<1.

[0012] Furthermore, in the method of this application, the compression ratio estimation formula of the SIMPLE8B algorithm is as follows: C<(valueSize×60.0 / bitwidth(avg)) / 8, min≥0; C<(valueSize×60.0 / bitwidth(2|avg|)) / 8, min<0; Where C is the compression ratio, valueSize is the byte length of the data type, avg is the average data value, min is the minimum data value, and bitwidth() is the function to calculate the effective bit width after removing the high-order zeros.

[0013] Furthermore, in the method of this application, the compression ratio estimation formula of the BITPACKING algorithm is as follows: C<8×valueSize / bitwidth(max); Where C is the compression ratio, valueSize is the byte length of the data type, bitwidth() is the function to calculate the effective bit width after removing the high-order zeros, and max is the maximum value of the data. When min≥0, ​​max takes the original value, and when min<0, max = max(|min|, 2|max|).

[0014] Furthermore, in the method of this application, the compression ratio estimation formula of the VARINT algorithm is as follows: C <valueSize / floor((bitwidth(avg) + 6) / 7)),min≥0; C <valueSize / floor((bitwidth(2|avg|) + 6) / 7)),min<0; Where C is the compression ratio, valueSize is the byte length of the data type, bitwidth() is the function to calculate the effective bit width after removing the high-order zeros, and avg is the average value of the data.

[0015] Furthermore, in the method of this application, the compression ratio estimation formula of the DELTA algorithm is as follows: C<(valueSize×60.0 / bitwidth(avg(2|delta|))) / 8; The compression ratio estimation formula for the DELTA2 algorithm is as follows: C<(sizeof×60.0 / bitwidth(avg(2|delta2|))) / 8; Where C is the compression ratio, valueSize is the byte length of the data type, bitwidth() is the function to calculate the effective bit width after removing the high-order 0s, avg(2|delta|) is twice the average of the absolute values ​​of the first-order difference, and avg(2|delta2|) is twice the average of the absolute values ​​of the second-order difference.

[0016] A second aspect of this application provides an apparatus for lossless compression of a database based on the compression ratio of a compression algorithm estimated from the feature values ​​of time-series data. When the apparatus operates, it implements the steps of the aforementioned method for lossless compression of a database based on the compression ratio of a compression algorithm estimated from the feature values ​​of time-series data. Figure 4 As shown, this device includes: The data acquisition module is used to acquire data blocks of time-series data, wherein the data blocks are subsets of time-series data grouped by columns; The feature extraction module is used to extract at least one time-series data feature value from the data block; The compression ratio estimation module is used to calculate the estimated compression ratio of the data block under each algorithm based on the compression ratio estimation formula corresponding to each algorithm and using the feature value for a variety of preset lossless compression algorithms. The algorithm selection module is used to select the algorithm with the highest estimated compression ratio from the various lossless compression algorithms as the target compression algorithm; The compression execution module is used to call the target compression algorithm to perform lossless compression on the data block and record the selected algorithm identifier in the metadata of the compressed data block.

[0017] A third aspect of this application provides an electronic device, including: a memory and a processor; Memory: Used to store computer programs; Processor: Used to execute the computer program to implement the steps of the aforementioned method for lossless compression based on the compression ratio of a compression algorithm using time-series data feature values ​​in the database.

[0018] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the steps of the aforementioned method for lossless compression based on the compression ratio estimation algorithm of time-series data feature values ​​in a database.

[0019] In summary, this invention avoids the high cost and overhead of AI models by using a lightweight estimation mechanism (dynamically estimating the compression ratio of each compression algorithm based on the feature values ​​of time-series data and selecting the optimal one), achieving adaptive compression with low computational overhead; effectively improving the compression ratio of time-series data (approaching the theoretical limit for high-potential data and reaching the highest compression ratio of 75% for low-potential data), fully exploring the compression potential, combining high efficiency and practicality, and significantly reducing storage costs.

[0020] Other features and advantages of this application will be set forth in detail in the following description, or will become apparent through the implementation of the relevant technical solutions of this application. The objectives and other advantages of this application can be achieved through the technical features and means explicitly pointed out in the description, claims, and drawings, and will be obtained through the implementation of these technical contents. Attached Figure Description

[0021] To more clearly illustrate the technical solution of this application, the accompanying drawings involved in the description of this solution will be briefly introduced below. It should be noted that the drawings only show some embodiments of this application. For those skilled in the art, other related drawings can be derived from these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating the implementation process of the proposed solution.

[0023] Figure 2 This is an example diagram of the CU compression code class in the embodiments of this application.

[0024] Figure 3 This is a flowchart illustrating the overall implementation of the lossless compression method based on the feature value estimation of time-series data for compression algorithm compression.

[0025] Figure 4 This is a structural diagram of the apparatus for lossless compression based on the compression ratio estimation algorithm of time-series data feature values, as described in this application.

[0026] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be noted that the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the protection scope of this application.

[0028] In this document, the term "comprising" and any variations thereof (such as "including," "including," etc.) are open-ended expressions and should be understood as "including but not limited to," meaning that the listed content is not exhaustive and may include other content not explicitly mentioned. The term "based on" should be understood as "at least partially based on," meaning that the basis or condition referred to may not be the only factor and may involve other relevant factors. The term "one embodiment" should be understood as "at least one embodiment," meaning that the described embodiment is not the only possible implementation, and other similar embodiments may exist.

[0029] In this application, the terms "a" and "a plurality of" are used to modify related elements or features, and their expression is illustrative rather than restrictive. Unless otherwise expressly stated in the context, "a" should be understood as "at least one," and "a plurality of" should be understood as "at least two." Those skilled in the art should reasonably interpret these terms based on the semantic and logical relationships of the context to ensure that they cover the possibility of "one or more."

[0030] Example: A method for lossless compression of a database based on estimating the compression ratio of a compression algorithm using time-series data feature values. The following six algorithms are commonly used for lossless compression of numerical data in time-series databases: RLE, DELTA, DELTA2, SIMPLE8B, BITPACKING, and VARINT (of course, other numerical compression algorithms can also be applied as long as their compression ratio estimation formulas can be derived). These six compression algorithms do not have strict standard definitions, and their implementations vary slightly across different database products. To explain how to derive the compression ratio estimation formulas based on the algorithm principles, it is necessary to first define the implementation principles of these algorithms. Different implementation schemes will result in slightly different compression ratio estimation formulas, but the underlying principles are similar.

[0031] 1. ZIGZAG Algorithm In computers, negative numbers are stored in two's complement form. For example, two similar small integers 1 and -1 are represented in binary as follows:

[0032] It can be seen that their values are similar, while the binary differences are quite large, which is not conducive to the compression of algorithms such as SIMPLE8B and BITPACKING.

[0033] ZIGZAG itself does not have compression ability, but is an auxiliary algorithm that converts data into positive integers to facilitate the compression of other algorithms.

[0034] Its core principle is to map positive numbers to even numbers and negative numbers to odd numbers. The mapping rules are as follows:

[0035] The corresponding C language code is as follows:

[0036] 2. RLE algorithm The full English name of RLE is Run Length Encoding, that is, run-length encoding. Its core idea is to replace the original continuous repeated sequence with "repetition count + data value". For example, the sequence AAABBBBBCD can be compressed into 3A4B1C1D. This encoding method of "repetition count + data value" will instead expand when encountering non-repeated data. The repeated segments and non-repeated segments can be distinguished through flag bits to suppress the expansion.

[0037] In implementation, the encoding method of MARKER (mask) + REPEATS (repetition count) + SYMBOL (repeated value) can be used. Specifically: (1) When REPEATS < minRepeats and SYMBOL != MARKER, the data is encoded as it is; (2) When REPEATS < minRepeats and SYMBOL == MARKER, MARKER + REPEATS (1 byte) is used for encoding; (3) When REPEATS >= minRepeats, MARKER + REPEATS + SYMBOL is used for encoding; Among them, minRepeats >= 4, and the byte length of MARKER is the same as that of SYMBOL.

[0038] 3. SIMPLE8B algorithm SIMPLE8B packs consecutive unsigned small integers in 64-bit blocks with fewer bits, which is suitable for the compression of small integers in time-series data. If the original data contains negative numbers, ZIGZAG processing needs to be performed first.

[0039] The SIMPLE8B algorithm works by dividing the storage space into contiguous 64-bit blocks. Each block has 4 bits reserved at the beginning as a selector, and the remaining 60 bits are used to store integer data. There are 16 different selectors, each determining the number of data items stored in the 60 bits and the data width, as follows:

[0040] When compressing data, selectors are tested one by one in ascending order of bit width. If the remaining data can fill a selector, that selector can be used for encoding.

[0041] For example, if the data consists of 152 consecutive 1s, we first check selector-0. We find that selector-0 cannot fill all 240 data points, so we cannot use selector-0 for encoding. However, it can fill all 120 data points of selector-1, so we encode those 120 1s using selector-1 first. This leaves 32 1s, enough to fill all 30 numbers in selector-3. Finally, there are 2 1s left, enough to fill all two numbers in selector-14. The total encoding is 3 uint64 bytes, totaling 24 bytes, as shown below:

[0042] 4. BITPACKING Algorithm There are several implementation variations of BITPACKING. This embodiment adopts a globally minimum bit width scheme, which is simple and efficient. That is, the maximum value in a set of unsigned integers to be compressed is determined, thereby determining the maximum bit width required to store this batch of numbers. For example, if the maximum value in a set of numbers is 50, then each number only needs 6 bits to be encoded. All numbers are encoded in 6-bit increments and then sequentially filled into the binary buffer. If the last byte is not completely filled, high-order bits are padded with 0s. An example is shown below:

[0043] If the original data contains negative numbers, the BITPACKING algorithm will be completely invalid, so ZIGZAG processing is required first.

[0044] 5. VARINT Algorithm The VARINT algorithm, similar to UTF-8, is a variable-length integer compression encoding algorithm. Its core principle is to dynamically adjust the number of bytes to represent integers of different sizes, making it particularly effective at compressing small values ​​(typically requiring only 1 byte) while maintaining compatibility with large values. If the original data contains negative numbers, ZIGZAG processing is required first. Its encoding principle is illustrated in the following example:

[0045] In the example, the original value to be encoded is 123456, and it is encoded according to the following steps: (1) Represent 123456 in network order binary; (2) Divide the network sequence binary into 7-bit groups from least significant bit to most significant bit, and fill any insufficient groups with bit 0 in the high bits; (3) Change the 7-bit grouping to little-endian order, that is, reverse the order, so that the high-order 0 can be discarded during subsequent encoding; (4) Each 7-bit group is composed of a flag bit added to the highest bit to form a byte. A flag bit of 1 indicates that the subsequent byte still belongs to the current value, and a flag bit of 0 indicates that the current byte is the last byte of the value. Thus, an integer 123456, which would normally require 4 bytes of storage, is encoded into binary data 11000000 11000100 00000111, which only requires 3 bytes.

[0046] 6. DELTA Algorithm In time-series scenarios, adjacent values ​​generally do not change much, and the DELTA algorithm takes advantage of this.

[0047] (1) First, calculate the first difference of the sequence. The resulting difference sequence values ​​are all very small. (2) Then, the difference between positive and negative numbers is converted into an unsigned integer using ZIGZAG; (3) Finally, the SIMPLE8B algorithm is used to perform compression.

[0048] 7. DELTA2 Algorithm The only difference between DELTA2 and the DELTA algorithm is that DELTA2 calculates the second difference of the sequences. Therefore, DELTA2 is better able to adapt to various time series patterns than DELTA, such as data changing exponentially or parabolically.

[0049] The compression algorithm described above needs to summarize a formula for estimating the compression ratio based on the feature values ​​of the data. The notation for the formula is as follows:

[0050] Therefore, the compression ratio estimation formula for the above compression algorithm is as follows: (a) RLE compression ratio estimation formula

[0051] Feature value: Data repetition rate R, 0 <= R < 1.

[0052] Assuming that the non-repeating data is randomly and uniformly distributed in the sequence, then R=0.9 indicates that on average, one out of every 10 numbers is different, and R=0.8 indicates that one out of every 5 numbers is different. That is, on average, the length of a continuous repeating sequence is R / (1-R), and the number of bytes it occupies is valueSize * R / (1-R).

[0053] This continuous repeating sequence, after being encoded by RLE, has a length of 2 * valueSize + 1.

[0054] The compression ratio is obtained by dividing the original data length by the encoded length.

[0055] For example, if the repetition rate of the sequence R = 0.99 and the data type is int32 (4 bytes), then the estimated compression ratio C = 44.

[0056] (b) SIMPLE8B

[0057] Feature value: The average value of the data.

[0058] When min>=0, the data does not require ZIGZAG processing, so the average bit width of the data can be estimated to be bitwidth(avg).

[0059] When min < 0, ZIGZAG processing is required first. After ZIGZAG processing, positive numbers will double, so 2avg can be used as the upper bound of the average of the data. Of course, avg may be negative, so the absolute value of avg is taken first.

[0060] Under the simple8b algorithm, the average number of data items that can be packed in a 64-bit block is N = 60 / bitwidth.

[0061] The original number of bits required for N data points was 8N * valueSize, but the actual number of bits used was 64. After dividing, the compression ratio is N * valueSize / 8.

[0062] For example, assuming all data in the sequence are non-negative, the average value is 7, and the data type is int64 (8 bytes), then the estimated compression ratio C = 15.

[0063] (c) BITPACKING

[0064] Eigenvalues: Extreme values ​​in the data.

[0065] The compression ratio estimation formula for BITPACKING is relatively simple. First, the bit width of the BITPACKING encoded data is obtained based on min and max (of course, if min < 0, ZIGZAG processing is required first). Then, the original number of bits of the data type is divided by the bit width to get the compression ratio.

[0066] For example, assuming all data in the sequence are non-negative numbers of data type int64 (8 bytes), with a maximum value of 50, then each number can be encoded using 6 bits, resulting in an estimated compression ratio C = 10.667.

[0067] (d) VARINT

[0068] Feature value: The average value of the data.

[0069] Taking min>0 as an example, no ZIGZAG processing is needed at this time. bitwidth(avg) is the average bit width of the data. Under the varint encoding rule, bitwidth needs to be encoded by floor((bitwidth + 6) / 7) bytes.

[0070] For example, assuming all data in the sequence are non-negative numbers, the data type is int64 (8 bytes), and the average value is 25, this means that each number only needs one byte to be encoded on average, so the estimated compression ratio C = 8.

[0071] (e)DELTA

[0072] Eigenvalue: avg(2|delta|), twice the average of the absolute values ​​of the first-order differences.

[0073] Because the DELTA algorithm ultimately uses SIMPLE8B compression, its essence is the same as that of SIMPLE8B, except for the difference in calculating the average coding bit width.

[0074] (f)DELTA2

[0075] Eigenvalue: avg(2|delta2|), twice the average of the absolute values ​​of the second-order differences.

[0076] Similar to DELTA, but the average coding bit width of SIMPLE8B is calculated by avg(2|delta2|), which is the average of the absolute values ​​of the second-order differences.

[0077] After summarizing the compression ratio estimation formulas for all numerical data types, the compression steps for a time-series database columnar data are as follows: 1. Traverse the data once to obtain the feature values ​​required by the estimation formulas of all compression algorithms; 2. Based on the data feature values, substitute them into the formula for each compression algorithm to obtain the estimated compression ratio for each compression algorithm; 3. Select the algorithm with the highest compression ratio to compress the data.

[0078] Effects of the invention: The compression ratios obtained through actual testing using this invention, and the highest compression ratios obtained using various compression algorithms, are shown in the table below: Comparison table of actual test results

[0079] The practical effects of this invention can be summarized as follows: 1. The higher the maximum compression ratio of the data (the greater the data compression potential), the more accurate the estimation result of this invention will be. It can be seen that when the maximum compression ratio is greater than 5, the compression ratio obtained by this invention is 100% consistent with the maximum compression ratio, and the optimal compression algorithm is selected 100%.

[0080] 2. When the data compression potential is small and the maximum compression ratio is relatively average (less than 5), this method can achieve 75% of the maximum compression ratio.

[0081] 3. The compression ratio obtained by this invention is sometimes higher than the highest compression ratio obtained by a single algorithm. This is because the method of this invention can sense and automatically adapt to changes in data characteristics over time, selecting the optimal compression algorithm for each data block.

[0082] In summary, this method is simple to implement, has low operating overhead, and can adapt to the characteristics of time-series data, fully exploring its compression potential. Compared to existing time-series database products that can only use a few fixed compression algorithms, this method can effectively improve the compression ratio of time-series data and simplify database usage.

[0083] Figure 1 The following is an example flowchart illustrating the implementation process of the present invention. As shown in the figure, the present invention is implemented through the following process: 1. Business data is written to the time-series database row by row, and the rows are sorted by time. 2. The time-series database internally groups data by column; 3. The data in each column is divided into blocks, for example, a block of data can contain a maximum of 10,000 values; 4. The data blocks calculate feature values, and all compression algorithms calculate the estimated compression ratio based on these feature values; 5. Select the algorithm with the highest estimated value to compress the data block. As shown in the figure, the delta algorithm has the highest estimated value, so delta is selected to compress the data block.

[0084] To more clearly illustrate the technical solution of this application, the following will provide further explanation through specific scenario embodiments.

[0085] The following is an example of the syntax for creating a time series table in Vastbase time series database:

[0086] The columns of the timing table can explicitly specify the compression algorithm to be used, such as specifying the delta2 algorithm in the power column; for columns that do not explicitly specify an algorithm, adaptive compression is performed, i.e., the solution of this invention is applied.

[0087] Vastbase's time-series database uses the openGauss column-oriented storage engine. After grouping time-series data by column, it is split into CUs (Compressed Values) for compression and storage. Each CU can contain a maximum of 60,000 values. The CU compression code is similar to... Figure 2 As shown in the figure: CU: The CU class provides the ability to compress and decompress CU blocks; CUStatistics: Collects various feature values ​​and related metadata from CU statistics, such as column type and number of bytes per type; ICompressor: An interface definition for compression algorithms, defining three interfaces: compression, decompression, and estimation of compression ratio based on CUStatistics; RleCompressor: This is the RLE algorithm implementation for the ICompressor interface; other algorithm implementations are similar. VTimeCompressor: This function actually performs the compression and decompression of Vastbase time-series data (CU). The compression strategy is implemented by the Compress method. The execution process of the VTimeCompressor::Compress method is as follows: 1. Call collectStatistics to collect the feature values ​​of the CU data block, which is passed in by the char *inBuf parameter; 2. Call the `chooseCompression` method to select the optimal compression algorithm. Of course, if the user has explicitly specified a compression algorithm for this column, the user-specified algorithm will be selected; otherwise, the estimation method for each compression algorithm will be called to perform an estimation, and the algorithm with the highest estimated value will be selected. 3. Call the `Compress` method of the specific compression algorithm to perform compression; Finally, the CU object can call the VTimeCompressor::InfoMode method to obtain the compression algorithm actually used by the CU and write it into the CU's metadata. When the CU is decompressed, it can directly use this compression algorithm to decompress.

[0088] The flowcharts and block diagrams in the accompanying drawings illustrate possible implementations of apparatus, methods, and computer program products according to various embodiments of this application, including architecture, functionality, and operation. In these figures, each block may represent a module, program segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should be noted that each block in the block diagrams and / or flowcharts, and combinations thereof, can be implemented using either a dedicated hardware-based system or a combination of dedicated hardware and computer instructions to achieve the specified function or operation.

[0089] like Figure 5 As shown in the illustration, an embodiment of this application also discloses an electronic device, including: a processor 310, a communication interface 320, a memory 330 for storing a processor-executable computer program, and a communication bus 340. The processor 310, communication interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 executes the executable computer program to implement the steps of the aforementioned method for lossless compression based on estimating the compression ratio of a compression algorithm using time-series data feature values ​​in a database.

[0090] It is understood that, in addition to memory and a processor, this electronic device may also include input devices (such as a keyboard), output devices (such as a display), and other communication modules. These input devices, output devices, and other communication modules all communicate with the processor through I / O interfaces (i.e., input / output interfaces).

[0091] The operations described in this application can be implemented by writing computer program code using one or more programming languages ​​or a combination thereof. The programming languages ​​include, but are not limited to, the following types: Object-oriented programming languages, such as Java, Smalltalk, C++, etc. Conventional procedural programming languages, such as "C" or similar programming languages.

[0092] The execution methods of program code include, but are not limited to: It runs entirely on the user's computer; Part of it executes on the user's computer, and part of it executes on a remote computer; Execute as a standalone software package; It is executed entirely on a remote computer or server.

[0093] In scenarios involving remote computers, the remote computer can connect to the user's computer via any type of network, including but not limited to local area networks (LANs) or wide area networks (WANs). Furthermore, the remote computer can also connect to external computers through an internet service provider, for example, by utilizing the internet for connection.

[0094] Furthermore, this application also discloses a computer-readable storage medium, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform the various steps of the lossless compression method disclosed in this application, which estimates the compression ratio of a compression algorithm based on time-series data feature values ​​in a database.

[0095] In the context of this application, a computer-readable storage medium refers to a tangible medium capable of storing computer program code and related data. Specific examples include, but are not limited to, the following: (1) Portable computer disk: such as floppy disks and other removable magnetic storage media.

[0096] (2) Hard disk: including mechanical hard disks and solid-state hard disks and other fixed storage devices.

[0097] (3) Random Access Memory (RAM): A volatile storage medium used for temporary storage of data and program code.

[0098] (4) Read-only memory (ROM): a non-volatile storage medium used to store fixed programs and data.

[0099] (5) Erasable programmable read-only memory (EPROM) or flash memory: non-volatile storage media that supports multiple erasures and reprogrammings.

[0100] (6) Fiber optic storage devices: storage media based on fiber optic technology.

[0101] (7) Portable compact disc read-only memory (CD-ROM): a read-only medium that stores data in the form of an optical disc.

[0102] (8) Optical storage devices: such as DVDs, Blu-ray discs and other storage media based on optical principles.

[0103] (9) Magnetic storage devices: such as magnetic tapes, disks and other storage media based on magnetic principles.

[0104] (10) Any suitable combination of the above: for example, combining multiple storage media to meet different storage needs.

[0105] These computer-readable storage media can be used to store the program code and related data described in this application to support program execution and persistent data storage.

[0106] Specifically, according to embodiments of this application, the processes described in the flowcharts can be implemented as computer software programs. For example, embodiments of this application relate to a computer program product comprising a computer program carried on a non-transitory computer-readable medium. This computer program includes program code for performing a method for lossless compression based on estimating the compression ratio of a compression algorithm using time-series data feature values ​​from a database disclosed in this application. When this computer program is executed by a processing device, it can achieve the functions defined in the embodiments of this application.

[0107] While the foregoing discussion contains several specific implementation details, these details should not be construed as limiting the scope of this application. The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this application is not limited to technical solutions formed by specific combinations of the above-described technical features. Furthermore, this application should also cover other technical solutions formed by any combination of the above-described technical features or their equivalents without departing from the foregoing disclosed concept.

[0108] Those skilled in the art should also understand that modifications can be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features, without departing from the spirit and scope of the technical solutions of the embodiments of this application. These modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the core spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for lossless compression in a database based on the estimated compression ratio of a compression algorithm for time series data feature values, characterized in that, The method comprises: S1, acquiring a data block of time series data, the data block being a time series data subset grouped by column; S2, extracting at least one time series data characteristic value of the data block; S3, for a plurality of preset lossless compression algorithms, calculating an estimated compression ratio of the data block under each algorithm based on a compression ratio estimation formula corresponding to each algorithm using the characteristic value; S4, selecting an algorithm with the highest estimated compression ratio from the plurality of lossless compression algorithms as a target compression algorithm; S5, calling the target compression algorithm to perform lossless compression on the data block, and recording the selected algorithm identification in the metadata of the compressed data block.

2. The method of claim 1, wherein, The data block in step S1 is the largest data unit after grouping by column, and each data block contains a preset number of time series data points.

3. The method of claim 1, wherein, The plurality of lossless compression algorithms in step S3 includes at least two of RLE, SIMPLE8B, BITPACKING, VARINT, DELTA, and DELTA2 algorithms. The time series data characteristic values extracted in step S2 include data repetition rate, data average value, data maximum value, data minimum value, first-order difference value, and second-order difference value.

4. The method of claim 3, wherein, When the data block contains negative values, perform ZIGZAG transformation on the data before using the SIMPLE8B algorithm, BITPACKING algorithm, and VARINT algorithm for compression, and map signed integers to unsigned integers.

5. The method of claim 3, wherein, The compression ratio estimation formula of the RLE algorithm is: C < (valueSize x R) / ((2 x valueSize + 1) (1 - R)); Wherein, C is the compression ratio, valueSize is the byte length of the data type, R is the data repetition rate, which is defined as the statistical proportion of consecutive repeated values, and 0 ≤ R < 1.

6. The method of claim 3, wherein, The compression ratio estimation formula of the SIMPLE8B algorithm is: C < (valueSize x 60.0 / bitwidth(avg)) / 8, min ≥ 0; C < (valueSize x 60.0 / bitwidth(2 | avg |)) / 8, min < 0; Wherein, C is the compression ratio, valueSize is the byte length of the data type, avg is the data average value, min is the data minimum value, and bitwidth() is the effective bit width calculation function after removing high bits 0.

7. The method of claim 3, wherein, The compression ratio estimation formula of the BITPACKING algorithm is: C < 8 x valueSize / bitwidth(max); Wherein, C is the compression ratio, valueSize is the byte length of the data type, bitwidth() is the effective bit width calculation function after removing high bits 0, and max is the data maximum value, when min ≥ 0, max takes the original value, when min < 0, max = max (| min |, 2 | max |).

8. The method of claim 3, wherein, The compression ratio estimation formula of the VARINT algorithm is: C < valueSize / floor((bitwidth(avg) + 6) / 7)),min≥0; C < valueSize / floor((bitwidth(2|avg|) + 6) / 7)),min < 0; Wherein, C is compression ratio, valueSize is byte length of data type, bitwidth() is effective bit width calculation function after removing high bit 0, and avg is data average value.

9. The method of claim 3, wherein, The compression ratio estimation formula of the DELTA algorithm is: C < (valueSize x 60.0 / bitwidth(avg(2|delta|))) / 8; The compression ratio estimation formula of the DELTA2 algorithm is: C < (sizeof x 60.0 / bitwidth(avg(2|delta2|))) / 8; Wherein, C is compression ratio, valueSize is byte length of data type, bitwidth() is effective bit width calculation function after removing high bit 0, avg(2|delta|) is 2 times of average value of absolute value of first-order difference value, and avg(2|delta2|) is 2 times of average value of absolute value of second-order difference value.

10. An apparatus for lossless compression of a database based on the compression ratio estimation algorithm using time-series data feature values, characterized in that... The device realizes the steps of the method for estimating the compression ratio of the time series data feature value based lossless compression algorithm in the database according to any one of claims 1-9 when the device runs, and the device comprises: A data acquisition module is configured to acquire a data block of time series data, wherein the data block is a time series data subset grouped according to columns. A feature value extraction module is configured to extract at least one time series data feature value of the data block. A compression ratio estimation module is configured to calculate the estimated compression ratio of the data block under each algorithm according to the feature value based on the compression ratio estimation formula corresponding to each algorithm for a plurality of preset lossless compression algorithms. An algorithm selection module is configured to select the algorithm with the highest estimated compression ratio from the plurality of lossless compression algorithms as a target compression algorithm. A compression execution module is configured to call the target compression algorithm to perform lossless compression on the data block, and record the selected algorithm identification in the metadata of the compressed data block.

Images