Processing method and processing device for time sequence floating-point number exclusive-or compression and readable storage medium
By performing cumulative XOR calculations and bit rearrangement on the time-series floating-point dataset, the problem of the number of leading and trailing zeros affecting compression efficiency in existing technologies is solved, achieving more efficient storage space utilization.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-30
- Publication Date
- 2026-04-10
AI Technical Summary
Existing XOR operation compression methods are highly dependent on the number of leading and trailing zeros in the floating-point XOR result, resulting in low compression efficiency and large space consumption when the number of leading and trailing zeros is low.
By performing cumulative XOR calculations on the time series floating-point dataset, a bit rearrangement mapping relationship is constructed, increasing the number of leading and trailing zeros, and rearranging the bits of the floating-point number to adapt to the XOR compression algorithm.
It improves the compression efficiency and storage space utilization of floating-point time series and enhances the performance of the XOR compression method in cases with low leading zeros and trailing zeros.
Smart Images
Figure CN121841367A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology. Background Technology
[0002] Time series data refers to a sequence of data representing the same statistical indicator arranged chronologically. Its most important characteristic is the temporal dependency between data points. With the development of the Internet of Things (IoT) and big data technologies, time series data plays a crucial role in numerous fields. In the financial sector, it can be used to predict stock price trends and assess risk; in industrial production, predictive maintenance and production process optimization are achieved by analyzing equipment sensor data; the internet industry uses time series data for user behavior modeling and analysis to optimize recommendation systems; and in smart cities, traffic flow patterns at different times are analyzed based on road sensor time series data to dynamically adjust traffic light timing schemes and optimize road traffic efficiency. The common thread in the applications of time series data across various fields is that by capturing the trends and cyclical patterns of data, it enables predictions of the future or the detection of anomalies.
[0003] The application scenarios of time-series data are increasingly penetrating almost all fields, leading to an explosive growth in the storage of time-series data. The storage of time-series data faces challenges such as massive data volume, high read / write performance requirements, and complex management of hot and cold data. Time-series data compression has become a key means to address these challenges. Compression technology can significantly reduce the space occupied by data storage, reduce equipment investment and storage costs, while improving data transmission efficiency and alleviating network bandwidth pressure. For example, in industrial IoT scenarios, the massive amounts of data collected at high frequencies by sensors can be compressed to preserve historical data for a long time at a lower cost, meeting the needs of subsequent in-depth analysis.
[0004] Based on the presence or absence of recovery errors, time-series data compression methods can be divided into lossy compression and lossless compression methods. Lossless compression methods can also be called reversible compression, meaning that the compression and decompression operations are completely reversible and do not lead to data errors; lossy compression introduces information loss, and the decompressed data has irreversible errors compared to the original data. The IEEE 754 standard is the most commonly used technical standard for representing floating-point numbers in computer systems, specifying the underlying binary representation of floating-point numbers (see...). Figure 3 , Figure 4Floating-point numbers are represented in three parts: the sign bit, the exponent bit, and the mantissa bit. Double-precision floating-point numbers require 64 bits (1 sign bit, 11 exponent bits, and 52 mantissa bits). In time-series data, adjacent floating-point numbers often have the same sign bit and similar exponent bits due to limited variation. Based on this characteristic, performing an XOR operation on two adjacent floating-point numbers produces a large number of consecutive zero bits. The XOR result of a double-precision floating-point number often exhibits a three-part structure: leading zero – significant bit – trailing zero. Applying targeted encoding strategies to these three parts is the idea behind XOR compression methods (currently the most advanced class of lossless floating-point time-series compression algorithms). Typical examples of this type of compression algorithm include the Gorilla algorithm, Chimp algorithm, and Elf algorithm.
[0005] Taking the Gorilla algorithm as an example, this paper introduces a compression method based on XOR operation. Before compression, the floating-point sequence is first XORed to obtain the XOR result sequence. Then, a dynamic length encoding strategy is adopted to process it in three cases: (1) The first value is stored directly; (2) If the XOR result is all zeros (the value has not changed), only one bit of the identifier 0 is stored; (3) If it is a non-zero value, first store one bit of the identifier 1, and then process it according to the following rules: 3.1) When the number of leading zeros and trailing zeros of the current XOR result is greater than or equal to the previous value, store one bit of the identifier 0 and only retain the valid bits; 3.2) Otherwise, use five bits to encode the number of leading zeros, six bits to encode the length of the valid bits, and finally store the content of the valid bits. Through statistical analysis of the XOR results of various datasets, the Chimp algorithm further divides the XOR result into four cases for dynamic length encoding. In order to increase the number of trailing zeros in the XOR result, the Chimp128 algorithm dynamically selects the data point that can produce the most trailing zeros in the XOR result from the 128 historical data points before the current data point as the object of the XOR operation of the current data point. To further increase the number of trailing zeros in the XOR result, the Elf algorithm performs a trailing bit erasure operation on floating-point numbers. The objects of the XOR operation are all floating-point numbers after the erasure operation. Although the erasure operation introduces errors, they can be recovered losslessly during decompression. Both the Chimp algorithm and the Elf algorithm achieve improved compression ratios.
[0006] Figure 1 and Figure 2 Two examples of the underlying representation of a double-precision floating-point number and its XOR result are given. It can be observed that the XOR result produces many leading and trailing zeros. Floating-point number compression methods based on the XOR operation utilize this redundancy of repeated zeros to achieve data compression. For example... Figure 2 The relative number of leading zeros and trailing zeros Figure 1 More often than not, XOR compression is more suitable, and it takes up the least amount of space.
[0007] Problems with compression methods based on XOR operations:
[0008] Existing XOR-based compression methods are highly dependent on the number of leading and trailing zeros in the XOR result. When the number of leading and trailing zeros is high, there is a high degree of data redundancy. XOR compression methods can efficiently utilize this redundancy for compression and reduce storage space usage. Conversely, when the number of leading and trailing zeros is low, the degree of data redundancy is low, which is detrimental to XOR compression, resulting in low compression efficiency and high storage space usage. Therefore, the number of leading and trailing zeros directly determines the compression efficiency of the XOR compression method and directly affects the amount of storage space that can be saved. These issues urgently need to be addressed. Summary of the Invention
[0009] The purpose of this invention is to address the problem that existing XOR operation compression methods are highly dependent on the number of leading and trailing zeros in the original XOR result value, which affects the compression efficiency and the amount of computer storage space occupied by the XOR operation compression method. This invention provides a processing method, processing device, and readable storage medium for XOR compression of time-series floating-point numbers.
[0010] Methods for XOR compression of floating-point numbers in time series processing include:
[0011] (a) Preprocessing stage:
[0012] S1. Data Acquisition and Preprocessing: Convert each value in the acquired time series data into binary format according to a preset standard to obtain floating-point numbers, forming a floating-point time series dataset. ;
[0013] S2, Data Analysis: Analysis of datasets Perform a cumulative XOR operation to obtain the cumulative XOR result sequence. For each cumulative XOR result sequence The frequency of occurrence of 0 elements in the bit index of each XOR result is sorted to obtain the sorted frequency distribution table T.
[0014] The sorted frequency distribution table T is indexed by bit position. and the frequency of 0 elements at that index position The structure is composed of elements, and sorted in descending order of the frequency of the element 0; where, For the first bit bit It is an integer;
[0015] S3. Construct a mapping relationship for bit rearrangement based on the sorted frequency distribution table T;
[0016] S4. Based on the bit rearrangement mapping relationship, process the dataset. The bits of each floating-point number in the dataset are rearranged to obtain the rearranged dataset. ;
[0017] (II) Compression and Decompression Stage:
[0018] Use the XOR compression algorithm to process the dataset. Compress the file to obtain a compressed file;
[0019] After decompressing the compressed file using the XOR compression algorithm, the decompressed file is obtained;
[0020] (III) Post-processing stage:
[0021] The bit rearrangement mapping constructed in the inversion preprocessing stage restores the bit order of each floating-point number in the decompressed file, thus recovering the floating-point time series dataset. .
[0022] Preferably, in step S2, the dataset is processed... Perform a cumulative XOR operation to obtain the cumulative XOR result sequence. The implementation methods include:
[0023] Preserving floating-point time series datasets The original value of the first floating-point number is obtained, and this original value is used as an XOR result; at the same time, for all floating-point numbers, the binary number of each bit in the current floating-point number is XORed with the binary number of the corresponding bit in the previous floating-point number to obtain the XOR result.
[0024] Arrange all XOR results in chronological order to obtain the cumulative XOR result sequence. .
[0025] Preferably, the implementation method of constructing the first mapping relationship of bit reordering based on the sorted frequency distribution table T in step S3 includes:
[0026] Extract the data with the highest percentage from the sorted frequency distribution table T as the first sublist, and the remaining data as the second sublist; create a new permutation and fill the positions according to the following rules:
[0027] For the first sublist, extract the positions corresponding to the bit indices of the original permutation and the order of the 0 elements in the first sublist in descending order, and use these as the first batch of positions. Then, insert the first batch of positions extracted from the original permutation into the front and back positions of the new permutation in descending order of the frequency of the 0 elements, until the first batch of positions is exhausted.
[0028] For the second sublist, extract the positions corresponding to the bit indices of the original permutation and the order of the 0 elements in the second sublist in descending order, and use these as the second batch of positions. Then, alternately insert the extracted second batch of positions from the original permutation before and after the center position of the new permutation in order of the frequency of the 0 elements from low to high, until the second batch of positions is exhausted.
[0029] Preferably, the insertion direction for the first batch of alternating insertions is: alternating insertion from both ends towards the center.
[0030] The first alternating insertion method for the first batch of positions is as follows: according to the order of the frequency of 0 elements from high to low, the position with the highest frequency of 0 elements is inserted into the frontmost position of the new arrangement, the position with the second highest frequency is inserted into the backmost position of the new arrangement, and so on; among them, the bit index of the frontmost position of the new arrangement is the smallest, and the bit index of the backmost position of the new arrangement is the largest.
[0031] The second alternating insertion method for the first batch of positions is as follows: according to the order of the frequency of 0 elements from high to low, the position with the highest frequency of 0 elements is inserted into the last position of the rear end of the new arrangement, the position with the second highest frequency is inserted into the first position of the front end of the new arrangement, and so on; among them, the bit index of the first position of the front end of the new arrangement is the smallest, and the bit index of the last position of the rear end of the new arrangement is the largest.
[0032] Preferably, the implementation method of constructing the second mapping relationship of bit reordering based on the sorted frequency distribution table T in step S3 includes:
[0033] Create a new permutation and fill the positions according to the following rules:
[0034] Extract the positions corresponding to the bit indices in the original permutation and the descending order of the frequency distribution table T after sorting. Then, alternately insert the extracted positions from the original permutation from both ends toward the center of the new permutation in descending order of the frequency of the 0 element, until exhaustion.
[0035] Preferably, the floating-point number is a single-precision floating-point number, a double-precision floating-point number, or a half-precision floating-point number.
[0036] A data processing apparatus includes a processor and a memory, the memory storing program instructions that, when executed by the processor, implement the data preprocessing method for time-series floating-point XOR compression as described above.
[0037] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the data preprocessing method for time-series floating-point XOR compression as described above.
[0038] The beneficial effects of this invention are:
[0039] XOR compression is currently the most advanced lossless compression method for floating-point time series. This type of method relies on the widespread use of leading and trailing zeros in the binary representation of the accumulated XOR value to eliminate redundancy and achieve compression. The processing method of this invention increases the number of leading and trailing zeros in the accumulated XOR value, making floating-point time series more suitable for XOR compression, saving storage space after compression, and improving compression efficiency.
[0040] This invention is mainly applied in the field of computers, and is used to improve the storage space occupied by computers. Attached Figure Description
[0041] Figure 1 This is a schematic diagram of the underlying representation of the double-precision floating-point numbers 314.1592 and 3141.592 and their XOR result;
[0042] Figure 2 This is a schematic diagram of the underlying representation of the double-precision floating-point numbers 4.144444 and 5.14444 and their XOR result;
[0043] Figure 3 This is a schematic diagram of the IEEE 754 standard double-precision floating-point representation;
[0044] Figure 4 This is a schematic diagram of the single-precision floating-point number representation in the IEEE 754 standard;
[0045] Figure 5 It is a permutation function With inverse permutation function Relationship diagram between them;
[0046] Figure 6 This is a flowchart of the XOR compression method for time-series floating-point numbers described in this invention. Detailed Implementation
[0047] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0048] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.
[0049] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but this is not intended to limit the scope of the invention.
[0050] The following definitions are provided in this invention:
[0051] (a) Floating-Point Time Series: A time series is a set of data points arranged in chronological order, typically representing the observed values of a variable as a function of time. For a floating-point time series, the data points are floating-point numbers (computer representations of real numbers). Let... It is a set of time points (usually arranged in ascending order, i.e.) Time series It is a point in time A set or function that maps to floating-point numbers:
[0052]
[0053] Where: each time-value pair For time Observed floating-point values of variables Time series data are essentially multiple numerical values arranged in chronological order. These values are treated as a whole and have specific meanings, such as daily temperature data or daily stock trading data in January.
[0054] (ii) Bit Representation of Floating-Point Numbers: Floating-point numbers are the form in which real numbers are stored in a computer. Unlike fixed-point numbers, floating-point numbers balance the range and precision by adjusting the position of the decimal point (the meaning of "floating"). The IEEE 754 standard is the de facto unified standard for floating-point number representation, including multiple precision levels such as half-precision, single-precision, and double-precision. Figure 3 and Figure 4 These are schematic diagrams illustrating the representation of double-precision floating-point numbers and single-precision floating-point numbers according to the IEEE 754 standard; based on the IEEE 754 double-precision floating-point standard (see...). Figure 3 For example, the 64 bits occupied by a double-precision floating-point number can be represented as a bit sequence. .in, Corresponding sign bit ; Corresponding exponent , corresponding mantissa .
[0055] (iii) Permutation functions for bit rearrangement The goal of bit rearrangement is to define a bijective function that maps each bit of the original 64-bit sequence to a new position, forming a new bit sequence. If the original bit sequence is: The rearrangement of this sequence of floating-point numbers can be achieved using a permutation function. describe:
[0056] ;
[0057] in: It is a bijective function, i.e., each original position Mapped to a unique new location And all positions are covered. The new bit sequence is: in: .
[0058] Figure 5 A permutation function is given. In the example (from top to bottom), the permutation function completely reverses the order of a double-precision floating-point number, that is, the bit that was originally in the start position 0 is moved to the end position 63, the bit that was originally in the end position 63 is moved to the start position 0, and so on. This represents the bit position before the permutation. Represents the bit position after the permutation. .
[0059] (iv) Inverse permutation function for restoring the original bit arrangement Due to the permutation function It is a bijective function, therefore it exists. inverse function As an inverse permutation function, it can rearrange the bits of a floating-point number. Restore to the original sequence Inverse permutation function Defined as:
[0060]
[0061] in, yes The inverse function of satisfies: .
[0062] Figure 5 An inverse permutation function is given. The example (from bottom to top) shows how the inverse permutation function restores the bit order of a completely reversed double-precision floating-point number to its original, unreversed state; it is a permutation function. The reverse process.
[0063] Based on the above concept, this invention analyzes the bit distribution of a floating-point time series dataset, constructs a bit rearrangement mapping relationship based on the data analysis results, and performs bit transformation according to the bit rearrangement of the dataset to generate a new dataset. The transformed dataset is then fed into any XOR compression method for data compression. The decompression process follows any XOR compression method. Post-processing involves reversing the bit rearrangement relationship on the decompressed dataset to obtain the recovered data. A processing method for floating-point XOR compression of time series is proposed, as follows:
[0064] Specific Implementation Method 1: Combination Figure 6 This embodiment describes a method for XOR compression of floating-point numbers for time series data, characterized in that...
[0065] (a) Preprocessing stage:
[0066] S1. Data Acquisition and Preprocessing: Convert each value in the acquired time series data into binary format according to a preset standard to obtain floating-point numbers, forming a floating-point time series dataset. ;
[0067] Data acquisition and preprocessing are primarily determined by the data source, which can be obtained from industrial real-time sensors or from existing datasets. The acquired data needs to be cleaned of "dirty" data, ensuring all data points are valid floating-point numbers, and the values of the entire dataset are converted point-by-point to a binary format conforming to a preset standard. The acquired data is a floating-point time series dataset. The default standard can be the IEEE 754 standard.
[0068] In practical applications, the floating-point number is a single-precision floating-point number, a double-precision floating-point number, or a half-precision floating-point number.
[0069] S2, Data Analysis: Analysis of datasets Perform a cumulative XOR operation to obtain the cumulative XOR result sequence. For each cumulative XOR result sequence The frequency of occurrence of 0 elements in the bit index of each XOR result is sorted to obtain the sorted frequency distribution table T.
[0070] The sorted frequency distribution table T is indexed by bit position. and the frequency of 0 elements at that index position The structure is composed of elements, and sorted in descending order of the frequency of the element 0; where, For the first bit bit For integers; as an example, the frequency distribution of double-precision floating-point numbers before sorting is shown in Table 1;
[0071] Table 1 Frequency Distribution Before Sorting
[0072]
[0073] Table 1 contains 64 entries. ,in, It is a bit index. This represents the frequency of zero occurrences at that position, laying the data foundation for subsequent in-depth analysis and feature mining.
[0074] The purpose of this step is to determine the value distribution of the bit positions in the XOR result (e.g., 64 bits for a double-precision floating-point number). Since XOR-based compression algorithms rely on the number of leading and trailing zeros for encoding, leading zeros are consecutive zeros at the beginning of the binary representation, and trailing zeros are consecutive zeros at the end. By permuting the bit positions, moving more zeros to the beginning and end of the XOR result improves encoding efficiency.
[0075] S3. Construct the mapping relationship for bit rearrangement: Construct the mapping relationship for bit rearrangement based on the sorted frequency distribution table T.
[0076] The purpose of this step is to arrange the bit positions in descending order according to the frequency of occurrence of the 0 element, and to map the positions of the "0 element" with lower frequency (i.e., the "1 element" with higher frequency) to the middle part of the new arrangement of the floating-point number, and to map the positions of the "0 element" with higher frequency to the two ends (the front and the back parts) of the new arrangement of the floating-point number, and to generate a formal mapping relationship that records this correspondence for later substitution.
[0077] S4. Based on the bit rearrangement mapping relationship, process the dataset. The bits of each floating-point number in the dataset are rearranged to obtain the rearranged dataset. ;
[0078] The purpose of this step is to rearrange the bits of the underlying representation of the floating-point number. The middle position of the rearranged underlying representation is the region where "1" elements appear most frequently, and the positions before and after it are the regions where "0" elements appear most frequently. Since XOR-based floating-point compression algorithms encode by calculating the number of "0" elements before and after the underlying representation of the floating-point number, the rearranged floating-point number is easier for related compression algorithms to encode.
[0079] (II) Compression and Decompression Stage:
[0080] Use the XOR compression algorithm to process the dataset. Compress the file to obtain a compressed file;
[0081] After decompressing the compressed file using the XOR compression algorithm, the decompressed file is obtained;
[0082] In practical applications, the XOR compression algorithm can be an existing algorithm such as Gorilla, Chimp, or Elf.
[0083] (III) Post-processing stage:
[0084] The bit rearrangement mapping constructed in the inversion preprocessing stage restores the bit order of each floating-point number in the decompressed file, thus recovering the floating-point time series dataset. .
[0085] Furthermore, in step S2, the dataset is... Perform a cumulative XOR operation to obtain the cumulative XOR result sequence. The implementation methods include:
[0086] Preserving floating-point time series datasets The original value of the first floating-point number is obtained, and this original value is used as an XOR result; at the same time, for all floating-point numbers, the binary number of each bit in the current floating-point number is XORed with the binary number of the corresponding bit in the previous floating-point number to obtain the XOR result.
[0087] Arrange all XOR results in chronological order to obtain the cumulative XOR result sequence. .
[0088] Furthermore, in step S3, the implementation of the first mapping relationship for bit reordering based on the sorted frequency distribution table T includes:
[0089] Extract the data with the highest percentage from the sorted frequency distribution table T as the first sublist, and the remaining data as the second sublist; create a new permutation and fill the positions according to the following rules:
[0090] For the first sublist, extract the positions corresponding to the bit indices of the original permutation and the order of the 0 elements in the first sublist in descending order, and use these as the first batch of positions. Then, insert the first batch of positions extracted from the original permutation into the front and back positions of the new permutation in descending order of the frequency of the 0 elements, until the first batch of positions is exhausted.
[0091] For the second sublist, extract the positions corresponding to the bit indices of the original permutation and the order of the 0 elements in the second sublist in descending order, and use these as the second batch of positions. Then, alternately insert the extracted second batch of positions from the original permutation before and after the center position of the new permutation in order of the frequency of the 0 elements from low to high, until the second batch of positions is exhausted.
[0092] In this preferred embodiment, the mapping relationship generated by the first mapping method results in the highest frequency of "0 elements" in the mapped bit arrangement at both ends, gradually decreasing towards the middle. This mapping relationship increases the number of leading and trailing zeros in the XOR result, making the dataset adjusted by this relationship more... It is more suitable for XOR compression encoding, and the resulting compressed file saves more storage space.
[0093] Furthermore, the insertion direction for the first batch of alternating insertions is: alternating insertion from both ends towards the center. Two implementation methods are given below:
[0094] The first alternating insertion method for the first batch of positions is as follows: according to the order of the frequency of 0 elements from high to low, the position with the highest frequency of 0 elements is inserted into the frontmost position of the new arrangement, the position with the second highest frequency is inserted into the backmost position of the new arrangement, and so on; among them, the bit index of the frontmost position of the new arrangement is the smallest, and the bit index of the backmost position of the new arrangement is the largest.
[0095] The second alternating insertion method for the first batch of positions is as follows: according to the order of the frequency of 0 elements from high to low, the position with the highest frequency of 0 elements is inserted into the last position of the rear end of the new arrangement, the position with the second highest frequency is inserted into the first position of the front end of the new arrangement, and so on; among them, the bit index of the first position of the front end of the new arrangement is the smallest, and the bit index of the last position of the rear end of the new arrangement is the largest.
[0096] In this preferred embodiment, the alternating insertion method evenly distributes the bit indices with higher frequency of 0 elements across the front and back of the mapped binary representation, while simultaneously increasing the number of XOR leading and trailing zeros to prevent 0 elements from being unevenly piled up in one direction. Since the XOR compression method expects more leading and trailing zeros, the mapping relationship generated by the alternating insertion method will result in a larger number of leading and trailing zeros in the floating-point dataset. The logic of matching the XOR compression method.
[0097] Furthermore, in step S3, the implementation methods for constructing the second mapping relationship of bit reordering based on the sorted frequency distribution table T include:
[0098] Create a new permutation and fill the positions according to the following rules:
[0099] Extract the positions corresponding to the bit indices in the original permutation and the descending order of the frequency distribution table T after sorting. Then, alternately insert the extracted positions from the original permutation from both ends toward the center of the new permutation in descending order of the frequency of the 0 element, until exhaustion.
[0100] In this preferred embodiment, the second mapping relationship maps the bit positions with the highest frequency of 0 elements to the two ends of the adjusted arrangement, and the frequency of 0 elements decreases from the two ends to the center. This mapping relationship increases the number of leading and trailing zeros in the XOR result, making the dataset adjusted by this relationship more... It is more suitable for XOR compression encoding, and the resulting compressed file saves more storage space.
[0101] Specific Implementation Method Two: A data processing device includes a processor and a memory. The memory stores program instructions, which, when executed by the processor, implement the data preprocessing method for time-series floating-point XOR compression as described in Specific Implementation Method One.
[0102] Specific Implementation Method 3: A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the data preprocessing method for time-series floating-point XOR compression as described in Specific Implementation Method 1.
[0103] As an example, this invention provides a sorted frequency distribution table for the original dataset, and specifically provides a sorted frequency distribution table T for double-precision floating-point numbers, as shown in Table 2;
[0104] Table 2. Frequency Distribution After Sorting
[0105]
[0106] Table 2 also provides the mapping relationship of the bit rearrangement constructed according to the present invention under the frequency distribution table, and provides a comparison table of the bit change relationship between the old arrangement and the new arrangement before and after the mapping, as shown in Table 3;
[0107] Table 3. Relationship between bit changes before and after mapping
[0108]
[0109] While the invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that different dependent claims and features described herein can be combined in ways different from those described in the original claims. It is also understood that features described in conjunction with individual embodiments can be used in other described embodiments.
Claims
1. A processing method for XOR compression of floating-point numbers in time series, characterized in that, include: (a) Preprocessing stage: S1. Data Acquisition and Preprocessing: Convert each value in the acquired time series data into binary format according to a preset standard to obtain floating-point numbers, forming a floating-point time series dataset. ; S2, Data Analysis: Analysis of datasets Perform a cumulative XOR operation to obtain the cumulative XOR result sequence. For each cumulative XOR result sequence The frequency of occurrence of 0 elements in the bit index of each XOR result is sorted to obtain the sorted frequency distribution table T. The sorted frequency distribution table T is indexed by bit position. and the frequency of 0 elements at that index position The structure is composed of elements, and sorted in descending order of the frequency of the element 0; where, For the first bit bit It is an integer; S3. Construct a mapping relationship for bit rearrangement based on the sorted frequency distribution table T; S4. Based on the bit rearrangement mapping relationship, process the dataset. The bits of each floating-point number in the dataset are rearranged to obtain the rearranged dataset. ; (II) Compression and Decompression Stage: Use the XOR compression algorithm to process the dataset. Compress the file to obtain a compressed file; After decompressing the compressed file using the XOR compression algorithm, the decompressed file is obtained; (III) Post-processing stage: The bit rearrangement mapping constructed in the inversion preprocessing stage restores the bit order of each floating-point number in the decompressed file, thus recovering the floating-point time series dataset. .
2. The data preprocessing method for time-series floating-point XOR compression according to claim 1, characterized in that, In step S2, the dataset is processed. Perform a cumulative XOR operation to obtain the cumulative XOR result sequence. The implementation methods include: Preserving floating-point time series datasets The original value of the first floating-point number is obtained, and this original value is used as an XOR result; at the same time, for all floating-point numbers, the binary number of each bit in the current floating-point number is XORed with the binary number of the corresponding bit in the previous floating-point number to obtain the XOR result. Arrange all XOR results in chronological order to obtain the cumulative XOR result sequence. .
3. The data preprocessing method for time-series floating-point XOR compression according to claim 1, characterized in that, In step S3, the implementation methods for constructing the first mapping relationship of bit reordering based on the sorted frequency distribution table T include: Extract the data with the highest percentage from the sorted frequency distribution table T as the first sublist, and the remaining data as the second sublist; create a new permutation and fill the positions according to the following rules: For the first sublist, extract the positions corresponding to the bit indices of the original permutation and the order of the 0 elements in the first sublist in descending order, and use these as the first batch of positions. Then, insert the first batch of positions extracted from the original permutation into the front and back positions of the new permutation in descending order of the frequency of the 0 elements, until the first batch of positions is exhausted. For the second sublist, extract the positions corresponding to the bit indices of the original permutation and the descending order of the 0 elements in the second sublist, and use these as the second batch of positions. Then, alternately insert the extracted second batch of positions from the original permutation before and after the center position of the new permutation in order of the frequency of the 0 elements from low to high, until the second batch of positions is exhausted.
4. The data preprocessing method for time-series floating-point XOR compression according to claim 3, characterized in that, The insertion direction for the first batch of alternating insertions is: alternating insertion from both ends towards the center.
5. The data preprocessing method for time-series floating-point XOR compression according to claim 4, characterized in that, The first alternating insertion method for the first batch of positions is as follows: according to the order of the frequency of 0 elements from high to low, the position with the highest frequency of 0 elements is inserted into the frontmost position of the new arrangement, the position with the second highest frequency is inserted into the backmost position of the new arrangement, and so on; among them, the bit index of the frontmost position of the new arrangement is the smallest, and the bit index of the backmost position of the new arrangement is the largest. The second alternating insertion method for the first batch of positions is as follows: according to the order of the frequency of 0 elements from high to low, the position with the highest frequency of 0 elements is inserted into the last position of the rear end of the new arrangement, the position with the second highest frequency is inserted into the first position of the front end of the new arrangement, and so on; among them, the bit index of the first position of the front end of the new arrangement is the smallest, and the bit index of the last position of the rear end of the new arrangement is the largest.
6. The processing method for XOR compression of floating-point numbers for time series according to claim 1, characterized in that, In step S3, the implementation methods for constructing the second mapping relationship of bit reordering based on the sorted frequency distribution table T include: Create a new permutation and fill the positions according to the following rules: Extract the positions corresponding to the bit indices in the original permutation and the descending order of the frequency distribution table T after sorting. Then, alternately insert the extracted positions from the original permutation from both ends toward the center of the new permutation in descending order of the frequency of the 0 element, until exhaustion.
7. The data preprocessing method for time-series floating-point XOR compression according to claim 1, characterized in that, The floating-point number is a single-precision floating-point number, a double-precision floating-point number, or a half-precision floating-point number.
8. A data processing apparatus, characterized in that, It includes a processor and a memory, the memory storing program instructions, which, when executed by the processor, implement the data preprocessing method for time-series floating-point XOR compression as described in any one of claims 1 to 7.
9. A computer-readable storage medium, characterized in that, The system contains a computer program that, when executed by a processor, implements a data preprocessing method for time-series floating-point XOR compression as described in any one of claims 1 to 7.