Data compression method adaptive to data change trend

Abstract

The invention discloses a data compression method adaptive to a data change trend. The method comprises the following steps: sequentially obtaining original values in a data stream to be compressed according to a time sequence; writing the obtained first original numerical value into a compressed data area through a preset coding mode, and taking the first original numerical value as a reference value; determining a data change trend, and dividing the data stream into N sub-data streams; calculating a threshold value of each sub-data stream; calculating a difference value between a subsequent original numerical value and an adjacent numerical value of the sub-data stream, or taking a value calculated through a fluctuation amplitude function as an intermediate value; performing corresponding calculation on the difference value or the intermediate value and a corresponding threshold value to obtain a target value; coding the plurality of target values of each sub-data stream by adopting a preset coding mode; and organizing and storing the reference value and the coded target value. The method can significantly improve the data compression ratio.

Description

Technical Field

[0001] The present invention relates to the technical field of databases, and more particularly to a data compression method that is adaptive to data change trends. Background Art

[0002] At present, in modern database systems, with the explosive growth of data volume, how to efficiently store and transmit large-scale data has become an important technical challenge. Especially in applications such as time series databases, column storage databases, industrial databases, and streaming computing systems, data compression technology is crucial to improving system performance and reducing storage costs. Efficient data compression can not only reduce storage space, but also significantly reduce disk I / O overhead and improve data transmission efficiency.

[0003] The existing mainstream data compression algorithms are mainly divided into two categories: symbol compression and arithmetic compression. Specific compression algorithms such as variable-length coding, Delta-Encoding, ZigZag coding, and Huffman coding can achieve certain compression effects in specific application scenarios, but they still have some significant shortcomings:

[0004] Variable-length encoding: Fixed-length integers or floating-point numbers are encoded in a variable-length manner according to the value size, and unnecessary bytes are removed. This encoding method is too simple, has limited applicable data models, and the compression effect is not ideal.

[0005] ZigZag encoding: This encoding method does not change the compressed data itself, but adjusts the data to be more friendly to the general compression algorithm to facilitate further compression. The original data size is not reduced, the compression effect depends on the external compression algorithm, and the stability of the compression quality is difficult to guarantee consistency.

[0006] Delta-Encoding: This is a popular compression method for integers. However, since it only stores the difference between the current value and the previous value, it does not consider the global characteristics of data changes. Therefore, the compression ratio is greatly reduced in the case of non-normal distribution or normal jitter of the data stream. In addition, the large change range of adjacent data will also seriously affect the compression quality.

[0007] Huffman coding: This coding requires the construction of a tree-structured dictionary, which takes up a lot of memory. In some cases, the Huffman tree takes up too much space, and the compression and decompression efficiency is not high enough.

[0008] Dictionary encoding: uses the repetitiveness of data for compression. However, this algorithm either becomes a static model because of storing a global dictionary, or takes up more compression space because the dictionary is stored in the compressed data.

[0009] Run-length encoding: replaces continuous repeated data with a shorter encoding form. This compression method pays too much attention to local changes and does not take into account the global data model. The actual compression effect is not ideal.

[0010] The above compression technologies cannot flexibly adapt to data change trends when faced with complex and changeable data distribution models, and the compression effect is not ideal. Especially in some data streams with large data fluctuations and normal or power-law distribution, the existing compression algorithms are difficult to fully utilize the changing characteristics of the data, resulting in limited compression ratio and efficiency.

[0011] Therefore, there is an urgent need for a compression method that can adaptively adjust according to data change trends to achieve efficient compression of different types of data streams while maintaining a high compression ratio and good performance. Summary of the invention

[0012] In view of this, the present invention provides a data compression method that is adaptive to data change trends. The method introduces a dynamically adjusted threshold value, divides the data stream into multiple sub-data streams according to the change trend, and calculates and further optimizes the change characteristics and threshold values ​​of each sub-data stream, thereby significantly improving the data compression efficiency. The method is suitable for the storage and processing of large-scale data.

[0013] In order to achieve the above object, the present invention adopts the following technical solution:

[0014] The present invention provides a data compression method that is adaptive to data change trends, comprising the following steps:

[0015] S1. Obtaining original values ​​in the data stream to be compressed in chronological order; writing the first original value obtained into the compressed data area by a preset encoding method and using it as a reference value;

[0016] S2, judging the difference between the subsequent original value and the adjacent value, and determining the data change trend;

[0017] S3, dividing the data stream into N sub-data streams according to the data change trend; calculating the threshold value of each sub-data stream;

[0018] S4, calculating the difference between the subsequent original value and the adjacent value of the sub-data stream, or the value calculated by the fluctuation amplitude function as the intermediate value;

[0019] S5, performing corresponding calculations on each difference or intermediate value belonging to the sub-data stream and the threshold value of the corresponding sub-data stream to obtain a target value;

[0020] S6, encoding multiple target values ​​of each sub-data stream using the preset encoding method;

[0021] S7. Organize and store the reference value and the encoded target value.

[0022] Furthermore, the step S2 comprises:

[0023] S21, obtaining a current value other than the first value, and comparing it with an adjacent value;

[0024] S22. The method for determining the data change trend according to the comparison result includes any of the following methods:

[0025] Determine whether the data is increasing, decreasing, or remaining the same, and use the direction of data increase or decrease as the data change trend;

[0026] Or determine whether the data changes significantly, slightly, or remains unchanged, and use the magnitude of the data change as the data change trend;

[0027] Or determine the calculation formula that fits the data change trend, and use several calculation formulas with higher fitting degree as the data change trend.

[0028] Furthermore, the step S3 comprises:

[0029] S31, dividing the data into multiple sub-data streams according to the data change trend, the sub-data streams include any of the following division methods:

[0030] dividing into a data growth data stream and / or a data reduction data stream;

[0031] or divided into a data stream with a large data change and / or a data stream with a small data change;

[0032] Or divide each data that meets the calculation formula of data change trend into a sub-data stream;

[0033] S32. Dynamically calculate a threshold value according to the data in each sub-data stream; the threshold value calculation method includes any of the following methods:

[0034] The maximum common factor of the numerical increase or decrease of the current sub-data stream is used as the threshold value;

[0035] Or the majority of the numerical increase or decrease amplitude values ​​of the current sub-data stream is used as the threshold value;

[0036] Or calculate the average difference between the current value and the historical data value as the threshold value;

[0037] Or the result of dynamic calculation of the fluctuation amplitude function is used as the threshold value;

[0038] Or the average change increment of the previous batch of data streams is used in combination with the change increment of the existing data in the current sub-data stream to calculate and predict the average change increment of the data as the threshold value.

[0039] Furthermore, in step S1, the preset encoding method includes:

[0040] There are three types of coding: dictionary coding, arithmetic coding and symbol coding. Specific coding methods include but are not limited to: variable length coding, ZigZag, Delta-Encoding and Huffman coding.

[0041] Furthermore, the step S5 comprises:

[0042] S51, a method for calculating the difference or the intermediate value and the threshold value of the corresponding sub-data stream, including two types of numerical calculation and data change trend function calculation; for example, specific calculation methods such as: subtraction, addition, multiplication, exponentiation, division, square root or fluctuation amplitude function calculation operation;

[0043] S52. Obtain a final compression target value according to the calculation result.

[0044] Further, the step S7 comprises:

[0045] S71: The organization method of the reference value and the encoded target value includes any of the following methods:

[0046] Arranging the reference value and the encoded target value in the order of acquisition;

[0047] or organizing the reference value and the encoded target value by classification of sub-data streams;

[0048] Or the reference value and the encoded target value are sorted according to a specified rule.

[0049] Furthermore, the step S7 further includes:

[0050] S72: When storing the reference value and the encoded target value, configure 0 or several bits for each compressed value to identify the sub-data stream to which it belongs.

[0051] Compared with the prior art, the technical solution adopted by the present invention has the following advantages:

[0052] By processing the numerical relationship in the data stream, a dynamically adjustable threshold value is set to record the data change trend. The data stream is divided into multiple sub-data streams according to the numerical change, and each sub-stream is combined with the threshold value and the reference value for calculation. In this way, data can be compressed more efficiently, and the compression ratio can be further improved when a general compression algorithm is used for further compression. This method is particularly suitable for storage and query services in the fields of time series databases, streaming computing storage, industrial databases, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0054] Figure 1 This is a flow chart of the data compression method for adaptive data change trend provided by the present invention.

[0055] Figure 2 The present invention provides a student achievement curve chart.

[0056] Figure 3 A schematic diagram of identifying the ascending data flow in the student scores provided by the present invention.

[0057] Figure 4 The present invention provides an electrocardiogram.

[0058] Figure 5 This is a schematic diagram showing a relatively obvious change trend in the electrocardiogram provided by the present invention. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] The present invention provides a data compression method that is adaptive to data change trends. The method can dynamically divide sub-data streams and perform corresponding compression processing according to the change trend of data; the method does not focus on the value itself, but focuses on the relationship between the values, and considers how to express the relationship between the values ​​with the smallest value to achieve the purpose of compression. The method can be widely used in database indexes by dynamically adapting to data change trends, and is also suitable for data files of columnar storage databases, such as storage and query services in the fields of time series databases, streaming computing storage, and industrial databases.

[0061] Reference Figure 1 As shown, the embodiment of the present invention discloses a data compression method that is adaptive to data change trends, comprising the following steps:

[0062] S1. Obtaining original values ​​in the data stream to be compressed in chronological order; writing the first original value obtained into the compressed data area by a preset encoding method and using it as a reference value;

[0063] S2, judging the difference between the subsequent original value and the adjacent value, and determining the data change trend;

[0064] S3, dividing the data stream into N sub-data streams according to the data change trend; calculating the threshold value of each sub-data stream;

[0065] S4, calculating the difference between the subsequent original value and the adjacent value of the sub-data stream, or the value calculated by the fluctuation amplitude function as the intermediate value;

[0066] S5, performing corresponding calculations on each difference or intermediate value belonging to the sub-data stream and the threshold value of the corresponding sub-data stream to obtain a target value;

[0067] S6, encoding multiple target values ​​of each sub-data stream using the preset encoding method;

[0068] S7. Organize and store the reference value and the encoded target value.

[0069] This embodiment divides different fragments of a data stream into N sub-data streams according to the characteristics of data changes and compresses them independently. Each independent sub-data stream has one or more threshold values ​​for indicating the trend of data changes. The purpose of setting the threshold value is to make the absolute value of the data finally written into the compressed data area as small as possible.

[0070] The threshold value represents the cardinality of the data change trend and range in the compressed data. The compressed data actually generated is the result of calculation based on the change characteristics of the current value and its corresponding threshold value. The calculation method includes: subtraction, addition, multiplication, exponentiation, division, square root or other matching calculation methods. The calculated value usually has a smaller absolute value, and then it is encoded using preset encoding methods such as variable length encoding, ZigZag or Delta-Encoding to generate shorter compressed data.

[0071] The following is a detailed description of each of the above steps:

[0072] In step S1, it is a process of obtaining the original values ​​in sequence, which conforms to the actual situation of data acquisition one by one, and is particularly suitable for streaming computing or time series data. Specifically, it includes: obtaining a data value from the data source in chronological order, and judging whether the current data is the first data. If it is the first data, the data is directly stored through the preset coding and used as the reference value. Among them, the preset coding method mainly includes three categories: dictionary coding, arithmetic coding and symbol coding. The specific coding methods include but are not limited to: variable length coding, ZigZag, Delta-Encoding coding and Huffman coding.

[0073] In step S2, it specifically includes:

[0074] S21, obtaining a current value other than the first value, and comparing it with an adjacent value;

[0075] S22. The method for determining the data change trend according to the comparison result includes any of the following methods:

[0076] Determine whether the data is increasing, decreasing, or remaining the same, and use the direction of data increase or decrease as the data change trend;

[0077] Or determine whether the data changes significantly, slightly, or remains unchanged, and use the magnitude of the data change as the data change trend;

[0078] Or determine the calculation formula that fits the data change trend, and use several calculation formulas with higher fitting degree as the data change trend.

[0079] In this step, if the global trend of the data stream cannot be known in advance, the trend is dynamically determined based on the change rules and trends of the previous and next data values. By calculating the difference between the current data (not the first data) and the previous data, it can be known in real time whether the data is rising, falling or remaining unchanged, thereby dynamically determining the trend; by calculating the magnitude of the change in different areas of the data, it can be known in real time whether the data is changing significantly, changing slightly or remaining unchanged, thereby dynamically determining the characteristics of the data change trend.

[0080] In this step, when the data trend can be estimated in advance (eg, the data generated by a specific industrial equipment is a sine curve), the formula that fits the data change trend (eg, sin(x)) can be used as the data change trend.

[0081] In step S3, it includes:

[0082] S31, dividing the data into multiple sub-data streams according to the data change trend, the sub-data streams include any of the following division methods:

[0083] dividing into a data growth data stream and / or a data reduction data stream;

[0084] or divided into a data stream with a large data change and / or a data stream with a small data change;

[0085] Or divide the data that fits each data change trend calculation formula into a sub-data stream.

[0086] S32. Dynamically calculate a threshold value according to the data in each sub-data stream; the threshold value calculation method includes any of the following methods:

[0087] The maximum common factor of the numerical increase or decrease of the current sub-data stream is used as the threshold value;

[0088] Or the majority of the numerical increase or decrease amplitude values ​​of the current sub-data stream is used as the threshold value;

[0089] Or calculate the average difference between the current value and the historical data value as the threshold value;

[0090] Or the result of dynamic calculation of the fluctuation amplitude function is used as the threshold value;

[0091] Or the average change increment of the previous batch of data streams is used in combination with the change increment of the existing data in the current sub-data stream to calculate and predict the average change increment of the data as the threshold value.

[0092] Dividing data into different sub-data streams according to their changing trends is similar to grouping the same type of data into the same group for processing.

[0093] The division of sub-data streams can adopt any of the three methods described in step S31, and the ultimate goal is to classify data with similar data change characteristics into the same sub-data stream, so as to facilitate the use of specific compression parameters to optimize compression for different types of data.

[0094] The threshold value can be calculated by any of the five methods described in step S32, with the ultimate goal of making the absolute value of the data finally written into the compressed data area as small as possible, and saving the threshold value of each sub-data stream for subsequent processing.

[0095] In step S4, the difference or intermediate value is calculated: for the subsequent original values ​​belonging to each sub-data stream, the difference between the data value and its adjacent original value is calculated, or the intermediate value is calculated by the fluctuation amplitude function to reduce the fluctuation amplitude of the data and improve the compression efficiency. Among them, the first original value in the data stream to be compressed does not belong to any sub-data stream, and there is no need to calculate the difference. The difference or intermediate value calculated in this step has been significantly reduced compared to directly storing the original value.

[0096] In step S5, the target value is calculated, which specifically includes:

[0097] S51, the difference or the intermediate value is calculated with the threshold value of the corresponding sub-data stream, which includes numerical calculation and data change trend function calculation, and the specific calculation method is as follows: subtraction, addition, multiplication, exponentiation, division, square root or fluctuation amplitude function calculation operation;

[0098] S52. Obtain a final compression target value according to the calculation result.

[0099] In step S51, for each sub-data stream, the difference or intermediate value obtained in step S4 is further calculated with the threshold value to obtain a final compression target value. The compression target value calculated in this step is further reduced compared with the difference or intermediate value obtained in step S4. Since the threshold value is closely related to the data characteristics, the compression range is relatively large.

[0100] In step S6, the multiple target values ​​of each sub-data stream are encoded using the preset encoding method. The preset encoding method includes three types: dictionary encoding, arithmetic encoding and symbol encoding. The specific encoding method is variable length encoding, ZigZag, Delta-Encoding and Huffman encoding, and the specific selection depends on the data characteristics.

[0101] In step S7, the compressed data is organized and stored, specifically including:

[0102] S71, organizing the reference value and the encoded target value in any of the following ways:

[0103] Arrange the reference value and the encoded target value in the order of acquisition. This organization method can ensure that the data can be read in the order of acquisition during the decompression process, and is suitable for time-sensitive data;

[0104] Or the reference value and the encoded target value are organized by sub-data stream classification. This organization method can ensure that the data can be read by data category during the decompression process. It is suitable for data that is not sensitive to time sequence but sensitive to data classification, such as: it is desired to read data of the same classification adjacently;

[0105] Or the reference value and the encoded target value are sorted according to a specified rule. This organization method is used to meet the scenario where data is read in a specific order during the decompression process, such as: data is written in chronological order but needs to be read in chronological reverse order.

[0106] S72. When storing the reference value and the encoded target value, configure 0 or several bits for each compressed value to identify the sub-data stream to which it belongs, so as to ensure that the original value can be correctly restored during decompression.

[0107] If you choose to organize and sort the compressed values ​​in the order in which they are obtained, you need additional space to record the number of the sub-data stream to which each target value belongs, so that you can correctly obtain the reference value and threshold value corresponding to each target value during decompression. For the case of two sub-data streams, only one bit is needed to identify the sub-data stream corresponding to each target value: bit 0 and bit 1 can identify the sub-data stream to be reduced and the sub-data stream to be added, respectively.

[0108] If the values ​​in the sub-data streams have been normalized during the process of calculating the target value, there is no need to identify the sub-data stream to which the target value belongs. In this case, 0 bits are used in S72.

[0109] Example A: Compression of student achievement classification data

[0110] For ease of understanding, the student score data shown in Figure 2 is used as an example for description. This embodiment is a typical integer data compression scheme that uses the average difference of adjacent values ​​as the data change threshold. Student scores have two change trends: rising and falling, so Figure 2 All the upward and downward performance trends in the data are considered as two sub-data streams. Figure 3 All the score changes marked with bold orange segments are classified as ascending data flow, and the score changes marked with other blue segments are classified as descending data flow. Among them, the value of failing is 1, the value of passing is 2, the value of good is 3, the value of excellent is 4, and the value of full score is 5.

[0111] First, a threshold value of the data change trend is calculated according to the characteristics of the descending data stream. For example, (2+1+3+2+3+2)÷6=2. The threshold value of the descending data stream is set to the mean change of the descending amount, 2. Then the original data of the descending data stream is as follows (the first value is written into the original data, and the subsequent values ​​are written into the descending difference):

[0112] 4 2 1 3 2 3 2

[0113] (The above is called array 1), except for the first value 4 which remains unchanged as the reference value, the other values ​​of array 1 are subtracted from the threshold value 2. The compressed data is:

[0114] 4 0 -1 1 0 1 0

[0115] (The above is called array 2); By comparison, it can be clearly seen that the absolute value of the compressed data is significantly reduced, which achieves the purpose of compression.

[0116] Next, a data change trend model is calculated based on the characteristics of the rising data flow (where the same adjacent scores are considered rising):

[0117]

[0118] The average difference of the ascending data is less than 1, and after rounding, 0 is taken as the threshold value of the ascending data stream. Therefore, the data after compression of the ascending data stream is as follows:

[0119] 2 0 2 0 2 0 0 0 0 1 0 0 0 1 3

[0120] Then organize the two batches of compressed data together in the order of writing time:

[0121] drop 4 0 -1 1 0 1 0 Lift 2 0 2 0 2 0 0 0 0 1 0 0 0 1 3

[0122] So the final compressed data is:

[0123] 4 0 2 -1 0 2 1 0 2 0 0 0 0 1 0 0 0 0 1 1 3 0

[0124] Of course, in this embodiment, an additional bit must be added to each value to distinguish which sub-data stream each data item corresponds to.

[0125] The original data of the student's performance chart is as follows:

[0126] 4 2 4 3 3 5 2 2 4 4 4 4 4 5 5 3 3 3 4 1 4 2

[0127] Through the above data comparison, it can be observed that when the method of this embodiment is used, the absolute value of the value of the compressed data written is significantly reduced compared with the original data.

[0128] Example B: Compression of ECG curve data

[0129] Reference Figure 4 , Figure 5 As shown, the following describes another embodiment using electrocardiogram data. This embodiment is a typical solution for dividing different data change trends according to the magnitude of data change. Figure 4 As shown in the figure, the overall curve is relatively flat, but each QRS complex generated by ventricular contraction will produce a large amplitude data change. Figure 5 As shown, we treat the data generated by each QRS wave group as an independent sub-data stream (that is, the wave data with a larger change amplitude marked by the orange line segment in the figure), and the other data as an independent sub-data stream. According to the compression method provided by the present invention, a larger threshold value can be calculated for the QRS wave group, while a smaller threshold value can be calculated for other gentle wave types. According to the compressed data stream calculated by the above threshold value, the expected wave type is very close to the straight line.

[0130] The data compression method for adaptive data change trend provided by the present invention, in its simplest embodiment, is to divide the change relationship between all values ​​in the data stream into two situations: one is an increasing change, and the other is a decreasing change. The change situations of the above two values ​​in the same data stream are processed separately, or the above two value change relationships in the same data stream are divided into two data streams (one is a data stream with an increasing trend, and the other is a data stream with a decreasing trend) and compressed separately. Then the two compressed values ​​are organized together to become compressed data.

[0131] Example C: Structured Data Stream Compression

[0132] When storing the position data of the two-dimensional coordinate system, the data of the X-axis and the Y-axis are divided into an independent sub-data stream, and the data streams on the two axes are compressed respectively using the compression method of the present invention, and then the compressed data of the two axes are organized together.

[0133] When storing the position data of the three-dimensional coordinate system, the data of the X-axis, Y-axis and Z-axis are each divided into an independent sub-data stream. The data streams on the three axes are compressed using the compression method of the present invention, and then the compressed data of the three axes are organized together.

[0134] When storing a two-dimensional table with multiple fields, the data of each field is divided into an independent sub-data stream, each data stream is compressed using the compression method of the present invention for different data types, and then the compressed data of multiple fields are organized together. When decompressing, the data lines can be read one by one.

[0135] Example D: Complex structured data stream compression

[0136] For example, the data continuously generated by an acquisition device can be represented by three graphs: a straight line, a sine curve, and a cosine curve. Each generated data contains one value corresponding to each of the three graphs. When storing such a data stream, the data of the straight line, sine, and cosine are divided into an independent sub-data stream. The data on the three data streams are compressed using the compression method of the present invention. During the compression process, different methods are used to calculate the threshold values ​​according to the characteristics of the data stream: for a straight line, the average difference of continuous values ​​is used as the threshold value; for sine and cosine lines, the formulas or results of sine and cosine are used as the threshold value. The compressed data of the three data streams are then organized together through preset coding.

[0137] Example E: Compression of integer data stream

[0138] The data growth threshold and the data reduction threshold are recorded somewhere inside or outside the compressed data area. The threshold value can be initially set to be invalid, or filled according to the experience value obtained in the previous compression process, or the two corresponding threshold values ​​can be copied from the most recently used compressed data area to the current position. As an embodiment, the data type of the threshold value can be a signed integer or a signed floating point number.

[0139] If the value to be written is the first value in the compressed data area, the original value is directly written into the data area through the preset encoding.

[0140] If the value to be written is not the first value in the compressed data area, the value to be written is first calculated with the value written most recently to obtain a smaller value A (for example, the value to be written is subtracted from the value written most recently to obtain the Delta-Encoding difference).

[0141] If the value to be written increases compared to the value written last time, first determine whether the growth threshold is valid. If not, calculate the growth threshold.

[0142] The purpose of calculating the growth threshold is to obtain the maximum common factor of the value growth amplitude in each process of the value in the data stream. As an embodiment, the average value of the absolute value of the Delta-Encoding difference can be used as a calculation method for the growth threshold. Other embodiments also include addition, multiplication, exponentiation, division, square root and other functions that match the data change model.

[0143] If the growth threshold value is valid, the smaller value A that has been obtained is calculated with the growth threshold value. The purpose of the calculation is to use a smaller value to represent the relationship between the two. This calculation process is referred to as calculating the growth target value. Among the methods for calculating the growth target value, the easiest embodiment to think of is the difference obtained by subtraction. Other embodiments also include addition, multiplication, exponentiation, division, square root and other matching calculation methods.

[0144] Next, the result of calculating the growth target value is written into the compressed data area. Usually, the value actually written into the compressed data area is significantly reduced compared to the absolute value of the original value. (Variable length coding or ZigZag coding is an embodiment of compression coding that can be used in this writing process)

[0145] If the value to be written is smaller than the value written last time, it is first determined whether the threshold reduction is valid. If not, it is necessary to calculate the threshold reduction.

[0146] The purpose of calculating the threshold reduction is to obtain the maximum common factor of the value reduction amplitude in each process of the value in the data stream becoming smaller. As an embodiment, the average value of the absolute value of the Delta-Encoding difference can be used as a method for calculating the threshold value. Other embodiments also include addition, multiplication, exponentiation, division, square root and other functions that match the model.

[0147] If the threshold value reduction is effective, the smaller value A that has been obtained is calculated with the threshold value reduction, and the purpose of the calculation is to use a smaller value to represent the relationship between the two. This calculation process is hereinafter referred to as calculating the target value reduction. Among the methods for calculating the target value reduction, the easiest embodiment to think of is subtraction, and other embodiments also include addition, multiplication, exponentiation, division, square root and other matching calculation methods.

[0148] Next, the result of calculating the target value reduction is written into the compressed data area. Usually, the value actually written into the compressed data area is significantly reduced compared to the absolute value of the original value. (Variable length coding or ZigZag coding is an embodiment of compression coding that can be used in this writing process)

[0149] From the above description, it can be seen that the compression algorithm divides the values ​​in a data stream into two data streams and compresses them separately, but after compression, the values ​​belonging to different data streams are still stored in the order in which they were written, making the decompression algorithm simpler.

[0150] Example F: Compression of floating point data stream

[0151] In actual business data, the integer part of the floating point number is compressed by the compression method of the present invention, and a good compression ratio can be obtained. The decimal part of the floating point number is the difficulty of compression because its decimal part varies greatly. In actual business data, the floating point number needs to retain the decimal places of a specified length. Based on the characteristics of business data, we can split the integer part and the fixed-point decimal part of the floating point number into two integers respectively. The high bits of the two integers usually have a limited range of variation, and the lower the number of bits, the greater the variation. This data feature is very suitable for the compression method adopted by the present invention.

[0152] When compressing a floating point number (taking the floating point number 3.1415926 as an example), the process is as follows:

[0153] 1) Extract the integer part 3 of the floating point number.

[0154] 2) After truncating the decimal part of the floating-point number to the specified number of digits of precision (such as 6 decimal places), a fixed-point number 0.141593 is formed.

[0155] 3) Remove the decimal point of the above fixed-point number to form an integer 141593.

[0156] 4) Split each floating-point number obtained in sequence into two integers according to the above method, such as splitting 3.1415926 into two integers 3 and 141593, and so on.

[0157] 5) In summary, the floating point value data stream is split into two sub-data streams, one storing the integer part and the other storing the decimal part, and the number of digits of all decimal parts must be the same.

[0158] 6) Compress each sub-data stream according to the aforementioned integer value compression method. During the compression process, each integer sub-data stream and decimal sub-data stream generates an increase threshold value and a decrease threshold value respectively, and stores them in the compressed data.

[0159] In addition, the method can detect the data stream in real time, regularly recalculate the current optimal data stream jitter range base and adjust the threshold value, so that the compression ratio of subsequent written data continues to approach the optimal.

[0160] Embodiment G: Storing different fragments of a compressed data stream in data blocks

[0161] The compressed data generated by the compression method can be stored in a data block. By dividing the data stream into different data fragments, multiple fragments can be stored in different areas of the data block. The threshold value calculated by each data fragment in real time is used as the base number of the jitter range of the data stream at that time, or multiple data fragments in each data block share a unified threshold value as the base number of the data stream jitter range for all data fragments stored in the current data block.

[0162] Embodiment H: Taking an integer data stream as an example, the detailed compression process is described.

[0163] Assume that the following table shows the operating costs of a company in the past decade of cost reduction and efficiency improvement:

[0164] Year Operating cost (ten thousand yuan) 2014 1286 2015 1213 2016 1132 2017 1059 2018 1010 2019 926 2020 813 2021 763 2022 683 2023 768

[0165] Table 1

[0166] Now we need to compress the integer data in the operating cost column in chronological order. Assuming that the data is obtained one by one in chronological order and compressed in real time, we cannot know the direction and trend of the data in advance.

[0167] Divide the data into two sub-data streams: increase and decrease:

[0168] When the data 1286 of 2014 is obtained, it is found that this is the first data in the compressed data area, so the data is directly written into the data area, and this value is used as the benchmark value. When the data 1213 of 2015 and other subsequent data are obtained, it is found that each data has the previous data to be compared, so the difference between them and the previous adjacent values ​​is calculated. The calculation results are shown in Table 2 below:

[0169] 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 1286 -73 -81 -73 -49 -84 -113 -50 -80 +85

[0170] Table 2

[0171] So far, the data is divided into three parts: the base value, the reduced sub-data stream and the increased sub-data stream, as shown in Table 3 below.

[0172] 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 1286 -73 -81 -73 -49 -84 -113 -50 -80 +85

[0173] Table 3

[0174] Calculate a threshold value for each sub-data stream:

[0175] Because there are only two ways for data to change: either decrease or increase (no change can be classified as increase), we can treat data decrease and data increase separately. Calculate a threshold value for data decrease and data increase respectively. The goal of the threshold value is to make the absolute value of the data finally written into the compressed data area as small as possible. If there can be only one threshold value for each case, the simplest calculation method is the average value of all value change increments.

[0176] In real-time data processing scenarios, it is impossible to calculate the average value of all data change increments in advance, so this requirement can be adjusted to predict the average change increment of the next batch of data by the average change increment of the previous batch of data. Since data values ​​in the data stream are continuously generated, and these data are usually divided into several parts and stored in data blocks. Regularly recalculating the average change increment for different batches of data in the data stream is a good way to respond to data changes and dynamically adjust the calculated average change increment to approach the optimal solution.

[0177] When there is no historical data for reference, the calculation method is as follows. When the first baseline value is obtained, because there is no historical data, the value needs to be written directly into the compressed data area, and the average increment cannot be calculated. When the second value is obtained, the difference calculated based on the first two original values ​​is used as the average change increment, that is, the threshold value. As for whether the calculated threshold value belongs to the increase sub-data stream or the decrease sub-data stream, it depends on the change relationship between the second value and the first value. Since the number of values ​​is too small at this time, the calculated threshold value may have errors. However, this threshold value only works for the current batch of data. Until the next batch of data begins, the accurate average change increment of the previous batch of data can be combined with the change increment of the first two data of the next batch of data to predict and calculate the average increment of the next batch of data. The threshold value calculated in this way is usually more accurate and will approach the optimal solution infinitely as time goes by.

[0178] Assume that the threshold value for reducing the sub-data stream is calculated to be -75 based on the average value of the previous data segment and the first two values ​​of the current data segment.

[0179] The intermediate value is obtained by calculating with the threshold value:

[0180] The logic of dividing the two sub-data streams by difference is completed synchronously with the calculation of each value in the above processing. Similarly, the calculation of the first threshold value is also generated during the calculation of the second value. This processing ensures that each value is scanned and processed only once, and no additional buffer is required, which reduces memory usage.

[0181] The difference between the original data and the threshold value is calculated again, and the result is shown in Table 4 below:

[0182]

[0183] Table 4

[0184] The incremental data +85 for adding a sub-data stream does not appear until 2023, at which time the threshold value for adding a sub-data stream has not yet been calculated. According to the above method, since there is only one increment, +85 is used as the threshold value for adding a sub-data stream. The final result is shown in Table 5 below:

[0185]

[0186]

[0187] Table 5

[0188] Generate the final compressed data:

[0189] According to the above content, there are three types of data to be written into the final compressed data:

[0190] A base value of 1286; those belonging to the decreasing sub-data stream: 2, -6, 2, 26, 9, -38, 25, -5; and 0 belonging to the increasing sub-data stream.

[0191] It should be emphasized here that since the data is acquired, compressed and appended to the compressed data area one by one in real time, each data item must have a bit to identify the sub-data stream ID to which the data item belongs (e.g. 0 means to reduce the sub-data stream; 1 means to add the sub-data stream), and one bit is required to represent a positive number 0 or a negative number 1. This is to enable the ability to restore the values ​​one by one during the decompression process.

[0192] The data finally written into the compressed data area is shown in Table 6 below:

[0193]

[0194] Table 6

[0195] The above table is just a simple encoding method for easy description and understanding. The final encoded data is 11 bytes, while the original data requires at least 20 bytes to represent.

[0196] As shown in the table above, there are many consecutive binary 0s or repeated contents in the final binary data, such as binary 10 or binary 01. This kind of data is particularly suitable for binary general text compression algorithms, such as Huffman, etc. Using the aforementioned binary general text compression algorithm on the encoded final data can still greatly improve the compression efficiency.

[0197] Embodiment I: Taking floating point data stream as an example, the compression process is described.

[0198] This embodiment adopts the business scenario of the aforementioned embodiment H, except that the enterprise operating cost data is changed from an integer value to a floating point value. See the following table:

[0199]

[0200]

[0201] Table 7

[0202] Now we need to compress the floating point data stream of the operating cost column in chronological order. Assuming that the data is obtained one by one in chronological order and compressed in real time, the direction and trend of the data cannot be known in advance.

[0203] Before compression, the original data is 44 bytes, based on 4 bytes for each single-precision floating point number.

[0204] According to the characteristics of floating-point numbers, the change of the integer part is relatively linear, while the change of the decimal part is relatively random. If each floating-point number is compressed as a whole, the compression efficiency will be low due to the inconsistent change characteristics of the integer part and the decimal part. For the basic method of compressing floating-point data using the present invention, please refer to the above-mentioned embodiment F.

[0205] First, a floating point data stream is divided into an integer part sub-data stream and a decimal part sub-data stream.

[0206] Then, using the idea of ​​the aforementioned embodiment H, the integer part sub-data stream and the fractional part sub-data stream are respectively regarded as a common integer data for compression. Therefore, the aforementioned two sub-data streams are further divided into: an integer part increase sub-data stream, an integer part decrease sub-data stream, a fractional part increase sub-data stream, and a fractional part decrease sub-data stream.

[0207] Next, a threshold value is calculated for each data sub-stream, for example, by using an average difference calculation method to calculate the threshold value for each data sub-stream.

[0208] Then, the difference between the two values ​​in each sub-data stream and the threshold value is calculated again, so that the absolute value of the original value can be reduced by multiples.

[0209] If necessary, during the encoding and output process, a sign bit may be added to each integer sub-data stream to indicate positive and negative numbers.

[0210] In the process of encoding and outputting, the order of data writing is maintained, and an integer value and a decimal value of variable length coding are output for each floating point number. Among them: the integer value comes from the aforementioned integer part increase sub-data stream or integer part decrease sub-data stream; the decimal value comes from the aforementioned decimal part increase sub-data stream or decimal part decrease sub-data stream.

[0211] One more bit is added to each value to distinguish which sub-data stream the two values ​​come from.

[0212] Next, the compression process is described in conjunction with specific data. Since the integer part is exactly the same as in the aforementioned embodiment H, the integer part is compressed as follows:

[0213] 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 1286 2 -6 2 26 9 -38 25 -5 0

[0214] Table 8

[0215] The original value of the decimal part is divided into increasing and decreasing data streams as shown in the following table:

[0216] 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 280 740 685 761 856 925 190 240 327 375

[0217] Table 9

[0218] The decimal difference is:

[0219] 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 280 460 495 521 529 550 550 445 434 481

[0220] Table 10

[0221] The decimal part increases the data flow threshold: (460+495+521+529+550)÷4≈428

[0222] The fractional part reduces the data flow threshold: (550+445+434+481)÷4≈478

[0223] The difference between the decimal part and the threshold value is as follows:

[0224] 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 280 32 67 93 101 122 72 -33 -44 3

[0225] Table 11

[0226] So far, the original data is compressed as follows:

[0227]

[0228] Table 12

[0229] Finally, the compressed data in Table 12 is output to the compressed data area through variable length coding. For example, 1286 and 280 are stored adjacently; 2 and 32 are stored adjacently; -6 and 72 are stored adjacently, and so on. Finally, 20 compressed integers are encoded and output. It is expected that the compressed data will not exceed 25 bytes, while the original data is 44 bytes, achieving the purpose of compression.

[0230] It should be emphasized that even with such compression, there are still many consecutive binary 0s or repeated contents in the final binary data, such as binary 10 or binary 01. This kind of data is particularly suitable for binary general text compression algorithms, such as Huffman, etc. Using the aforementioned binary general text compression algorithm on the encoded final data can still greatly improve the compression efficiency.

[0231] Embodiment J: Compression using a function that conforms to the data change trend as a threshold value

[0232] Suppose you need to monitor the east-west swing of the top of a building during a typhoon, and use a device to periodically report the swing distance (unit: meter). The device has a fixed point on the roof, from which a 10-meter rope is hung, with a heavy object hanging. If the rope swings to the east, the reported value is negative; if the rope swings to the west, the reported value is positive. Assume that the reporting interval is 1 second.

[0233] The 11 values ​​collected during a certain period are shown in Table 8 below:

[0234] Swing distance (m) 0.1745 0.1745 0.3490 0.8716 1.7365 1.5643 0.6976 -0.1745 -2.2495 -2.4192 -2.0791

[0235] Table 13

[0236] Table 13 shows the floating point data stream that is now to be compressed.

[0237] We assume that each floating point number is a 4-byte single-precision floating point number, so the original data size is 4×11=44 bytes.

[0238] Next, the data compression method for adaptive data change trend provided by the present invention is used for compression. First, the data change trend is analyzed and a suitable data trend fitting formula (also called function) is found, which is the process of finding a threshold value. Of course, a fixed value can be found as the threshold value. However, if the absolute value of the compressed data is expected to be smaller, it is necessary to make the data change trend fitting formula coincide with the trend of the rope pendulum as much as possible.

[0239] Since the rope swing trend conforms to the sine curve, and since its swing angle is definitely an acute angle, it can be known that the absolute value of each swing distance can be calculated by the following formula, as shown in Table 14:

[0240] sin(swing angle)

[0241]

[0242]

[0243] Table 14

[0244] Therefore, the swing angle can be calculated according to the swing distance by the following formula, and the data is shown in Table 15:

[0245] Swing angle (absolute value) = arcsin(|swing distance|) ÷ rope length

[0246] Swing distance (m) Calculation formula Swing angle (degrees) 0.1745 arcsin(0.1745)÷10 1 0.1745 arcsin(0.1745)÷10 1 0.3490 arcsin(0.3490)÷10 2 0.8716 arcsin(0.8716)÷10 5 1.7365 arcsin(1.7365)÷10 10 1.5643 arcsin(1.5643)÷10 9 0.6976 arcsin(0.6976)÷10 4 -0.1745 -arcsin(0.1745)÷10 -1 -2.2495 -arcsin(2.2495)÷10 -13 -2.4192 -arcsin(2.4192)÷10 -14 -2.0791 -arcsin(2.0791)÷10 -12

[0247] Table 15

[0248] The above formula is generally derived from the experience of business experts, or before the system is officially put into operation, the data collection state is installed to collect data in advance, and then the operating parameters are calculated offline before being deployed online for formal operation.

[0249] Now you can set the threshold for the westward swing to the formula: arcsin(x) ÷ 10. The threshold for the eastward swing to the formula:

[0250] -arcsin(|x|)÷10.

[0251] The intermediate value is obtained by calculating with the threshold value (that is, the value dynamically calculated by the fluctuation amplitude function is used as the intermediate value):

[0252] Therefore, for the input of 0.1745 swinging westward, the middle value 1 can be calculated by this formula, and so on. Further, for the input of -0.1745 swinging eastward, the middle value -1 can be calculated by this formula, and so on. As can be seen from the table below, the number of valid data bits has been significantly reduced.

[0253] Finally, the intermediate value sequence is obtained, as shown in Table 16:

[0254] 1 1 2 5 10 9 4 -1 -13 -14 -12

[0255] Table 16

[0256] Then keep the first value as the reference value and calculate the difference of the remaining values, and get the following data in Table 17:

[0257] 1 degree 0 1 3 5 -1 -5 -5 -8 -6 -6

[0258] Table 17

[0259] A reference value, an increase sub-data stream and a decrease sub-data stream are obtained from the above data, as shown in Table 18:

[0260] 1 degree 0 1 3 5 -1 -5 -5 -8 -6 -6

[0261] Table 18

[0262] The next step is to write the above data into the compression target in the order in which it was obtained. If variable length encoding is used, about 11 bytes are required, while the original data length is 4 times that length.

[0263] The maximum value stored in one byte by the variable length coding method is 126. However, the maximum absolute value of the value after compression in this embodiment is only 8, and there are a large number of 0-bit values ​​in the compressed data. If any other general binary compression algorithm is used for the compressed data, the size of the compressed data can be further reduced.

[0264] After the data is split into the reference value, the increased sub-data stream and the decreased sub-data stream, the subsequent compression method can be carried out completely with reference to the ideas in the aforementioned embodiments, and no further elaboration is made here.

[0265] The data compression method for adaptive data change trend provided by the present invention can significantly improve the data compression ratio by dynamically dividing sub-data streams, calculating threshold values ​​and combining Delta-Encoding, variable-length coding and the like, and can ensure the sequentiality and decompressibility of the compressed data, and is suitable for data compression in various scenarios.

[0266] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0267] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data compression method that is adaptive to data change trends, characterized in that: The following steps are involved: S1. Obtaining original values ​​in the data stream to be compressed in chronological order; writing the first original value obtained into the compressed data area by a preset encoding method and using it as a reference value; S2, judging the difference between the subsequent original value and the adjacent value, and determining the data change trend; S3, dividing the data stream into N sub-data streams according to the data change trend; calculating the threshold value of each sub-data stream; S4, calculating the difference between the subsequent original value and the adjacent value of the sub-data stream, or the value calculated by the fluctuation amplitude function as the intermediate value; S5, performing corresponding calculations on each difference or intermediate value belonging to the sub-data stream and the threshold value of the corresponding sub-data stream to obtain a target value; S6, encoding multiple target values ​​of each sub-data stream using the preset encoding method; S7. Organize and store the reference value and the encoded target value.

2. The data compression method according to claim 1, characterized in that: The step S2 comprises: S21, obtaining a current value other than the first value, and comparing it with an adjacent value; S22. The method for determining the data change trend according to the comparison result includes any of the following methods: Determine whether the data is increasing, decreasing, or remaining the same, and use the direction of data increase or decrease as the data change trend; Or determine whether the data changes significantly, slightly, or remains unchanged, and use the magnitude of the data change as the data change trend; Or determine the calculation formula that fits the data change trend, and use several calculation formulas with higher fitting degree as the data change trend.

3. The data compression method according to claim 1, characterized in that: The step S3 comprises: S31, dividing the data into multiple sub-data streams according to the data change trend, the sub-data streams include any of the following division methods: dividing into a data growth data stream and / or a data reduction data stream; or divided into data streams with large data changes and / or data streams with small data changes; Or divide each data into a sub-data stream according to the calculation formula that fits the data change trend; S32. Calculate a threshold value according to the data in each sub-data stream; the threshold value calculation method includes any of the following methods: The maximum common factor of the numerical increase and / or decrease of the current sub-data stream is used as the threshold value; Or adopting the majority of the numerical increase and / or decrease amplitude values ​​of the current sub-data stream as the threshold value; Or calculate the average difference between the current value and the historical data value as the threshold value; Or the result of dynamic calculation of the fluctuation amplitude function is used as the threshold value; Or the average change increment of the previous batch of data streams is used in combination with the change increment of the existing data in the current sub-data stream to calculate and predict the average change increment of the data as the threshold value.

4. The data compression method according to claim 1, characterized in that: In the step S1, the preset encoding methods include three types: dictionary encoding, arithmetic encoding and symbol encoding.

5. The data compression method according to claim 1, characterized in that: The step S5 comprises: S51, a method for calculating the difference or the intermediate value and the threshold value of the corresponding sub-data stream, including numerical calculation and data change trend function calculation; S52. Obtain a final compression target value according to the calculation result.

6. The data compression method according to claim 1, characterized in that: The step S7 comprises: S71: The organization method of the reference value and the encoded target value includes any of the following methods: Arranging the reference value and the encoded target value in the order of acquisition; or organizing the reference value and the encoded target value by classification of sub-data streams; Or the benchmark value and the encoded target value are sorted according to specific rules.

7. The data compression method according to claim 6, characterized in that: The step S7 further comprises: S72: When storing the reference value and the encoded target value, configure 0 or several bits for each compressed value to identify the sub-data stream to which it belongs.

Images