An adaptive learning compression method and system for floating-point data incorporating outlier handling
By using adaptive rounding transformation and a lightweight prediction model to segment and compress floating-point data, the problem of low efficiency in floating-point data processing in existing technologies is solved, achieving high compression ratio and fast access capability, which is suitable for columnar storage and real-time analysis scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- EAST CHINA NORMAL UNIV
- Filing Date
- 2026-02-09
- Publication Date
- 2026-05-26
AI Technical Summary
Existing floating-point compression technologies cannot simultaneously achieve high compression ratios, fast random access, robustness to outliers, and the inability to utilize complex sequence correlations, resulting in low efficiency in floating-point data processing.
Floating-point numbers are converted to fixed-point integer representations through adaptive rounding transformation, outliers are identified and disguised, segmented compression is performed in conjunction with a lightweight prediction model, and bit packing technology is used to store the residual array, generating a storage structure that supports fast random access and range queries.
It achieves efficient compression of floating-point data, supports fast random access and range queries, and has the ability to handle high-precision values and outliers, thus improving storage efficiency and query performance.
Smart Images
Figure CN122092873A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of database and big data storage technology, and specifically relates to a method for efficient compression and querying of floating-point data for columnar storage, that is, an adaptive learning compression method for floating-point data that combines outlier handling. Background Technology
[0002] In columnar databases and big data analytics systems, data compression is one of the core technologies for improving storage efficiency and query performance. For integer data, a series of efficient, lightweight compression algorithms have been widely applied. For example, Run-Length Encoding (RLE) compresses highly repetitive data by recording consecutive repeating values and their frequency, but it is ineffective for high cardinality data. Frame Of Reference (FOR) maps a set of values to a smaller dynamic range by subtracting a common cardinality, such as a minimum value, and then performs bit packing, supporting efficient random access. Dictionary Encoding builds a mapping dictionary for each unique value and replaces the original value with a short integer codeword, making it particularly suitable for low cardinality columns. These methods are preferred for columnar storage due to their fast decompression speed, support for random access, and predicate pushdown.
[0003] However, the aforementioned integer compression methods cannot be directly applied to floating-point numbers in the IEEE 754 standard. The fundamental reason is that arithmetic operations on floating-point numbers, such as subtraction, introduce irreversible rounding errors. This makes it impossible for compression based on differences, such as Delta encoding, or base offsets, such as FOR encoding, to achieve lossless reconstruction, thus limiting the application of traditional lightweight algorithms on floating-point data types.
[0004] To address this, the industry has proposed compression techniques specifically for floating-point numbers. Early general-purpose block compressors, such as Zstd and LZ4, while offering high compression ratios, were "heavyweight" compressions, requiring the entire data block to be decompressed to access any value, thus failing to support the fast random access and query optimization required in database scenarios. In recent years, a series of "lightweight" floating-point-specific codes have been proposed, their core idea being to avoid arithmetic operations and instead use bit-level operations. For example, Gorilla coding effectively utilizes the continuity of time series by calculating the XOR of the current value with the previous value and storing the non-zero bit segment. Its improved scheme, Chimp / Chimp128, further enhances compression ratio and speed through finer pattern selection and history window lookup. The recently proposed ALP (Adaptive Lossless Floating-Point Compression) takes a different approach, adaptively converting floating-point vectors to integers based on the assumption that they may originate from decimal values, and then applying lightweight compression to the integers, achieving significant improvements in both compression ratio and speed.
[0005] Despite this, existing floating-point compression schemes still have limitations. XOR-based methods, such as Gorilla and Chimp, heavily rely on the bit similarity between values for compression efficiency. For data with complex numerical functional relationships but discontinuous bit patterns, their compression effect is limited. While ALP methods introduce integer transformations, their subsequent compression does not systematically explore and utilize the complex sequence correlations that may exist in the transformed integer sequence. On the other hand, in the field of integer compression, emerging learned compression frameworks, such as LeCo, demonstrate the potential to achieve ultra-high compression ratios by learning sequence patterns through machine learning models, such as linear / multinomial regression, using a "model prediction + residual storage" approach. However, directly applying this framework to raw floating-point numbers faces significant challenges because there is a gap between the binary representation of floating-point numbers and their numerical semantics, and outliers, such as extremely high-precision numbers, NaN, and Inf, can severely interfere with model training and prediction.
[0006] Therefore, there is a significant gap in the current technological landscape: the lack of an end-to-end floating-point compression scheme that can intelligently convert floating-point numbers into a representation suitable for learning models, while simultaneously achieving high compression ratios, fast random access, robustness to outliers, and balancing lossless and lossy requirements. Summary of the Invention
[0007] To overcome the technical problems of existing floating-point compression technologies that cannot simultaneously achieve high compression ratios, fast random access, robustness to outliers, and the inability to utilize complex sequence correlations, the purpose of this invention is to provide an adaptive learning compression method and system for floating-point data that combines outlier processing.
[0008] The specific technical solution for achieving the objective of this invention is as follows: An adaptive learning compression method for floating-point data that incorporates outlier handling includes the following steps: Step 1: Divide the input floating-point number sequence into data blocks. Perform adaptive rounding transformation on each data block to convert the floating-point number into a fixed-point integer representation. For outliers whose precision difference from other values in the block exceeds a preset threshold, a context-aware camouflage mechanism is used to identify outliers within the segment. At the corresponding outlier position, a pseudo-normal value similar in precision and order of magnitude is generated based on the distribution characteristics of normal values within the block for replacement. The original outlier and its position are recorded in an independent outlier table. The detected outliers also include: floating-point numbers that overflow due to excessive precision; NaN and Inf special values in the IEEE 754 standard. For NaN and Inf special values, the same context-aware camouflage mechanism is used for replacement processing and recording.
[0009] The adaptive rounding conversion specifically includes: for each data block, determining the decimal exponent e and scaling factor f by sampling; for each floating-point number v within the block, calculating round(v×10^(ef)) to convert it into an integer; and recording the conversion parameters e and f for subsequent recovery.
[0010] Step 2: Perform pattern analysis and segmentation on the converted integer sequence, and select a segmentation strategy according to the compression mode: In lossless compression mode, a segmentation algorithm with the goal of minimizing overall storage overhead is adopted; in lossy compression mode, a segmentation algorithm that covers all data points with the fewest segments within a preset error range is adopted; train a lightweight prediction model for each segment, wherein the lightweight prediction model is a multinomial regression model, and the model order is adaptively selected according to the data characteristics.
[0011] The segmentation strategy in the lossless compression mode includes: aiming to minimize the sum of "model parameter storage + segment length × maximum residual encoding bit width"; and using a dynamic programming algorithm for segment merging decisions. The segmentation strategy under the lossy compression mode includes: adopting an online segmentation algorithm based on error constraints; starting from the beginning of the sequence, data points that conform to the current model error range are sequentially included in the same segment until the error exceeds the limit and a new segmentation begins.
[0012] Step 3: Store the model parameters, segment boundary information, and residual array between the model predicted value and the actual integer value for each segment, and compress and encode the residual array; organize the model metadata, segment information, compressed residuals, global transformation parameters, and outlier table into a storage structure that supports fast random access and range queries; The residual array is compressed using bit packing technology, and the bit width is dynamically determined based on the maximum residual within the segment.
[0013] An adaptive learning compression system for floating-point data incorporating outlier handling, used to execute the method described herein, includes: The data conversion module is used to perform the adaptive conversion from floating-point numbers to integers and the handling of outliers and special values; The segmented modeling module is used to perform the sequence pattern learning and segmented modeling described above. An encoding and storage module is used to perform parameter and residual encoding and storage structure generation as described in claim 1 or 7.
[0014] A decompression method is provided for decompressing data compressed by the method of the present invention, comprising: receiving a decompression request and determining whether it is a single-point access or a range access; if it is a single-point access, locating the segment where the data point is located, reading the model parameters to calculate the predicted value, obtaining the corresponding residuals and adding them together, and then restoring it to a floating-point number according to the transformation parameters; if the point is recorded as an outlier or an anomaly, directly reading the original value from the independent outlier table for restoration; if it is a range access, locating the involved segments in batches, calculating the predicted value and residuals in parallel, restoring them to floating-point numbers in batches, and replacing the recorded outliers or anomalies within the range.
[0015] The process of restoring the floating-point number includes: reading the conversion parameters e and f of the corresponding data block; and restoring the integer to a floating-point number by calculating the integer value / 10^(ef).
[0016] This invention can efficiently and intelligently compress floating-point numbers, significantly reducing storage space while ensuring that compressed data supports efficient random access and range queries, and has the ability to handle high-precision values and outliers. Attached Figure Description
[0017] Figure 1 This is a flowchart of the compression process of the present invention; Figure 2 A schematic diagram of the sequence pattern learning and segmentation modeling module; Figure 3 This is a schematic diagram of the compression system structure of the present invention; Figure 4 This is a flowchart illustrating the decompression method. Detailed Implementation
[0018] The present invention will be further described below with reference to the accompanying drawings and embodiments.
[0019] like Figure 1 As shown, an adaptive learning compression method for floating-point data that incorporates outlier handling includes the following steps: Step 1: Data Blocking, Adaptive Transformation, and Outlier Unification: The input floating-point sequence is divided into data blocks of a fixed length, such as 1024 values. For each data block, an adaptive rounding precision parameter is determined, and the floating-point numbers within the block are converted to fixed-point integer representations accordingly. This process can be considered a form of lossy quantization. For outliers that cannot be effectively converted due to excessive precision, or those with values of NaN / Inf, a context-aware masquerading mechanism is used to replace them with a valid floating-point value with a precision and distribution similar to the data within the block, ensuring unified processing in subsequent steps.
[0020] Step 2, Sequence Pattern Learning and Segmented Modeling: For the integer sequence obtained in Step 1, perform pattern analysis and segmentation. First, analyze the numerical distribution, trends, and local stationarity of the sequence through sampling. Then, based on the analysis results, dynamically divide the long sequence into multiple continuous segments, ensuring that the numerical change patterns within each segment are relatively simple. A lightweight prediction model is constructed for each segment, preferably a multinomial regression model, which can predict the corresponding integer value using the data index as input.
[0021] Step 3: Parameter and Residual Encoding and Storage Structure Generation: This step stores the model parameters corresponding to each data segment, such as polynomial coefficients, segment boundary information, and the residual array between the original integer values and the model's predicted values. The residual array is compressed using entropy encoding or bit packing. Finally, all model metadata, segmentation information, and compressed residual data are organized into a self-describing storage format. This format is specifically designed to allow direct location of the data segment and its residual at any position through simple arithmetic calculations, thus natively supporting fast single-point random access and efficient range scanning.
[0022] An adaptive learning compression system for floating-point data incorporating outlier processing includes: a data conversion module, a segmentation modeling module, and an encoding and storage module, used to execute the steps of the above-mentioned floating-point number learning mode compression method incorporating outlier processing.
[0023] Specifically, the data conversion module includes: Fixed-length block partitioning: Divide the input floating-point number sequence into data blocks of a fixed length (e.g., 1024 values).
[0024] Adaptive rounding transformation: For each data block, the optimal rounding precision parameter `e` and scaling factor `f` are determined by sampling. The floating-point number `v` is rounded to an integer by calculating `v × 10^(ef)`, and the parameters `e` and `f` are stored. This transformation is, in most cases, a controlled lossy compression.
[0025] Outlier camouflage and marking: For outliers that cause conversion overflow or cannot be effectively converted due to excessive precision, as well as special values such as NaN and Inf, a context-aware camouflage mechanism is adopted. This mechanism analyzes the distribution characteristics of normal values within the data block, generates a pseudo-normal value that is similar to the context in terms of precision and order of magnitude, and replaces it. At the same time, the position and value of the original outlier are recorded in a separate outlier table to ensure consistent processing in subsequent compression steps.
[0026] The segmented modeling module specifically includes: Adaptive Segmenter: Employs a dual-mode segmentation strategy, intelligently selecting the segmentation algorithm based on the compression mode (lossless / lossy). In lossless mode, segmentation aims to minimize "model storage overhead + segment length × maximum residual bit width," ensuring optimal overall compression ratio. In lossy mode, within a preset error range, segmentation aims to cover all data points with the fewest possible line segments, minimizing model metadata storage. Intelligent conversion from lossless to lossy segmentation is supported, further reducing storage overhead by merging adjacent segments within acceptable error limits.
[0027] Regressor and Predictive Model: A lightweight predictive model is built for each data segment, with multinomial regression being the preferred choice. Model training employs the minimization of the maximum residual criterion and is accelerated using the SIMD instruction set to ensure high throughput.
[0028] Hyperparameter Advisor: By analyzing data characteristics through sampling, it automatically recommends optimal configurations: recommending the optimal rounding parameters `e` and `f` for floating-point conversion; determining whether to enable outlier masquerading; and recommending the most suitable segmentation strategy (fixed length / variable length) and regression model type. The encoding and storage module specifically includes: Compact encoding and bit packing: The model parameters for each segment, namely polynomial coefficients, segment boundary information, and residual array, are efficiently encoded. The residual array uses bit packing technology, dynamically determining the bit width based on the maximum value of the residual. The encoding process is optimized using SIMD instructions to achieve high-speed compression.
[0029] Metadata compression: For model parameters, i.e. floating-point numbers, a bit erasure compression technique similar to ELF is used to further reduce metadata overhead.
[0030] Outlier values are stored separately: The outliers identified in step 1 and their location information are compressed and stored separately to ensure the integrity of the original data.
[0031] like Figure 4As shown, a decompression method is used to decompress data compressed by the method of the present invention, comprising: receiving a decompression request and determining whether it is a single-point access or a range access; if it is a single-point access, locating the segment where the data point is located, reading the model parameters to calculate the predicted value, obtaining the corresponding residuals and adding them together, and then restoring it to a floating-point number according to the transformation parameters; if the point is recorded as an outlier or an anomaly, directly reading the original value from the independent outlier table for restoration; if it is a range access, locating the involved segments in batches, calculating the predicted value and residuals in parallel, restoring them to floating-point numbers in batches, and replacing the recorded outliers or anomalies within the range. Example
[0032] See Figure 1 The present invention provides an adaptive learning compression method for floating-point data that combines outlier processing, which is carried out according to the following steps: S101: Input a sequence of floating-point numbers Receives a raw floating-point sequence to be compressed, which typically originates from a database column, data file, or real-time data stream.
[0033] S102: Select an appropriate expansion factor based on the data characteristics to convert the block data into integers. The input sequence of floating-point numbers is divided into fixed-size blocks, such as 1024 values per block. For each block, its numerical range, precision distribution, and dynamic characteristics are analyzed through sampling to adaptively determine a set of optimal transformation parameters: the decimal exponent *ee* and the scaling factor *ff*. Then, for each floating-point number *v* within the block, a transformation *round(v×10(e−f))* is performed to convert it to an integer. This step achieves lossy transformation with controllable precision while preserving the main numerical characteristics, laying the foundation for subsequent integer compression.
[0034] S103: Determine if there are outliers During the conversion process, check for any values that cannot be converted normally, such as integer overflow due to excessive precision; special floating-point values, such as `NaN` and `Inf`, and outliers that differ too much from other values in the block. If there are abnormal values, execute S104; otherwise, jump to S105.
[0035] S104: Simulate a context-similar value at the original data location and record the original value. For detected outliers, a "pseudo-normal value" that is consistent with the context in terms of magnitude and precision is generated based on the distribution characteristics of normal values within the data block, such as mean, variance, and precision trends. This pseudo-normal value replaces the original value to maintain the continuity of the data pattern. The actual value of the original outlier and its index position are recorded in a separate outlier table to ensure the possibility of lossless reconstruction.
[0036] S105: Integrate the data into an integer sequence After completing the transformation and outlier handling, the integers output from all data blocks are integrated into a complete integer sequence in their original order, which is then used as input for subsequent compression.
[0037] S106: Determine if it is lossless compression Based on user presets or application scenario requirements, determine the target compression mode for this operation: if it is lossless compression, execute S107; if it is lossy compression, execute S108.
[0038] S107: Use a segmentation strategy based on minimizing storage overhead. In lossless compression mode, a segmentation algorithm is adopted with the goal of minimizing overall storage overhead. This algorithm dynamically evaluates the sum of "model size + segment length × maximum residual bit width" to determine whether to merge adjacent segments, and finally obtains the segmentation scheme with the smallest total compressed size while satisfying lossless requirements.
[0039] S108: Segmentation Algorithm Using O'Rourke's Online Algorithm In lossy compression mode, an online algorithm such as O'Rourke or its variants is used to greedily extend the current segment as much as possible within a preset error constraint, until adding the next point would cause the error to exceed the limit, thereby covering all data with the fewest segments and minimizing the amount of model metadata.
[0040] S109: Train a model for each segment of data based on the segmenter results. For each defined segment, a lightweight prediction model is trained using a regression method, such as multinomial fitting. The training objective is to minimize the maximum residual between the model's predicted values and the actual integers within that segment, thereby reducing the bit width of the residual encoding.
[0041] S110: Generate the final compressed data from the encoded data sequence and the residuals after model calculation. For each segment, its model parameters and segment boundary information are stored, and the residual array formed by the difference between the true integer and the model prediction is bit-packed and encoded. Finally, the metadata of all segments, compressed residuals, global transformation parameters (e, fe, f), and outlier table are encapsulated into a structured, self-describing compressed data file.
[0042] The above embodiments demonstrate the entire process of this invention from floating-point input to compressed data generation. This method achieves high compression ratios, supports fast random access, and possesses robust floating-point data compression through adaptive conversion, outlier handling, intelligent segmentation, and model prediction, making it particularly suitable for columnar storage and real-time analysis scenarios.
[0043] Example 2, Model Training Part See Figure 2 The model training of this invention includes the following steps: Select segmentation method based on compression mode Select the appropriate segmentation logic based on the compression requirements: if it is lossless compression, use lossless segmentation logic; if it is lossy compression, use lossy segmentation logic.
[0044] Lossless segmentation logic The lossless segmentation logic adopts a two-stage strategy of "segmentation-merging" with the goal of minimizing the overall storage overhead, that is, minimizing the sum of "model parameter storage + segment length × maximum residual encoding bit width".
[0045] Segmentation Stage: The integer sequence is initially divided into multiple smaller segments. Segmentation is determined based on the data's changing trend. Segmentation occurs when the numerical change between adjacent data points exceeds a preset threshold, or when a significant abrupt change occurs in the second-order difference of consecutive data points. For example, for the sequence [30, 31, 32, 29, 49, 50, 51, 100], an abrupt change from 29 to 49 can be detected, initially dividing it into two segments: [30, 31, 32, 29] and [49, 50, 51, 100].
[0046] Merging Phase: Calculate the change in overall storage overhead after merging adjacent segments. If the total overhead after merging is less than the sum of the overheads before merging, then the merge is performed. For example, consider segments [30,31,32,29] and [49,50,51,100], and evaluate whether merging them into a single segment [30,31,32,29,49,50,51,100] reduces the overall storage overhead. This process is implemented using a dynamic programming algorithm, ultimately yielding the segmentation scheme with the minimum total volume under lossless conditions.
[0047] Lossy segmentation logic The lossy segmentation logic aims to cover all data points with the fewest possible segments within a preset error range. An online segmentation algorithm based on error constraints, such as the O'Rourke algorithm or its improved variants, is employed.
[0048] The specific implementation is as follows: Starting from the beginning of the sequence, an initial polynomial model is fitted using the data points of the current segment. Then, subsequent data points are checked sequentially to determine whether they are within the allowable error range of the current model's prediction values. If so, the point is included in the current segment, and the model is refitted; otherwise, the current segment ends, and a new segment begins from that point. This process is repeated until all data points have been processed.
[0049] For example, for the sequence [30, 31, 32, 29, 49, 50, 51, 100], with an error range of ±3, the algorithm might merge the first four points [30, 31, 32, 29] into one segment because 29 is within the error range of the linear model prediction based on the first three points, while [49, 50, 51, 100] might need to be further split because 100 differs significantly from the other points.
[0050] The model is trained by multinomial fitting for each segment. For each finalized data segment, multinomial regression is used to train the model. The order of the multinomial is recommended based on the data characteristics by the hyperparameter advisor module, and is usually limited to a low order, such as 0 to 3, to ensure the model is lightweight.
[0051] During the fitting process, for lossless compression, the maximum residual criterion is adopted, which involves solving a constrained optimization problem to minimize the maximum absolute value of the difference between the predicted and actual values for all points within a segment. For lossy compression, under the premise of satisfying preset error constraints, the option to minimize the mean squared error or use the simplest model that satisfies the constraints can be selected.
[0052] The final model parameters obtained for each segment include: polynomial coefficients, segment start and end indices, and the maximum residual value within that segment. If it is lossy compression, there is no need to store the residual value.
[0053] Through the above segmentation and modeling process, the system can adaptively select the optimal data organization method and prediction model according to data characteristics and compression requirements, laying the foundation for efficient compression.
[0054] See Figure 3 The decompression method of the present invention is performed according to the following steps: S301: Start inputting the compressed data file, which contains compressed model metadata, residual data, transformation parameters, and an outlier table. S302: Determine if it is a single data decompression. Parse the user request and determine whether it is a request to decompress a single value at a specific location or a request to decompress a continuous range of data: if it is a request to decompress a single data value, execute S310; if it is a request to decompress a range of data, execute S303.
[0055] S303: Location Segment Metadata For range decompression requests, determine the contiguous data segments covered by the requested range. Locate all relevant data segments in batches using the index structure and obtain the metadata locations of these segments.
[0056] S304: Read the model parameters corresponding to the segment and calculate the predicted value. The model parameters of the involved segments are read in batches, and for each data point within the range, the predicted integer value is calculated in parallel using the model of the corresponding segment.
[0057] S305: Obtain the residual bit width and batch obtain the segmented residual values. Read the residual bit width of each segment, and extract the residual values corresponding to all data points within the range from the residual array in batches.
[0058] S306: Calculate integer data within the decompression range. The predicted values obtained in batches are added to the residual values to obtain the original integer values of all data points within the range.
[0059] S307: Batch recovery of floating-point numbers based on partitions Based on the data block to which each data point belongs, the corresponding transformation parameters (`e`, `f`) are read, and the calculation `integer value / 10^(ef)` is performed in parallel to batch restore all integer values to their original floating-point values. For outlier locations that may exist within the range, the original values are read in batches from the outlier table and replaced. S308: Output the decompressed floating-point sequence The final sequence of floating-point numbers is output in order of the requested range to complete the decompression process.
[0060] S309: Determine if it is a non-outlier value Based on the requested data location index, query the independently stored outlier table: if the location is marked as an outlier, execute S304; if the location is a normal value, execute S305.
[0061] S310: Output separately stored exception values The original floating-point value corresponding to that location is read directly from the outlier table and returned to the user. This step ensures that the outlier can be reconstructed without loss of quality.
[0062] S311: Locating metadata of a single data segment For normal values, the data segment to which the data point belongs can be quickly located using a pre-built index structure (such as a learned index or binary search), and the metadata storage location of that segment can be obtained.
[0063] S312: Read the model parameters corresponding to the segment and calculate the predicted value. The prediction model parameters (such as polynomial coefficients) corresponding to the segment are read from storage, and the predicted integer value is calculated by the model based on the relative position of the data point within the segment.
[0064] S313: Obtain the residual bit width and get the corresponding residual value. Read the residual bit width information stored in the segment, and extract the corresponding residual value from the compressed residual array according to the position index of the data point.
[0065] S314: Calculate the raw value of an integer Add the model predictions to the residuals to get the original integer values.
[0066] S315: Recovery data based on expansion factor Read the conversion parameters (decimal exponent `e` and scaling factor `f`) corresponding to the data block, and convert the integer back to the original floating-point value by calculating `integer value / 10^(ef)`, thus completing the decompression of a single data item.
[0067] Through the decompression process described above, this invention achieves efficient random access and fast range scanning of compressed data. Single data decompression avoids the overhead of traditional block compression algorithms that require decompressing the entire data block, while range decompression further improves throughput through batch processing and parallel computing, fully meeting the query performance requirements of modern analytical databases.
[0068] The above is merely a further description of the present invention and is not intended to limit the scope of this patent. Any equivalent implementation of the present invention should be included within the scope of the claims of this patent.
Claims
1. An adaptive learning compression method for floating-point data that incorporates outlier handling, comprising the following steps: Step 1: Divide the input floating-point number sequence into data blocks, perform adaptive rounding transformation on each data block to convert the floating-point number into a fixed-point integer representation; for outliers whose precision difference from other values in the block exceeds a preset threshold, use a context-aware masquerading mechanism to identify outliers in the segment and generate a pseudo-normal value that is similar in precision and order of magnitude at the corresponding outlier position, based on the distribution characteristics of normal values in the block, to replace it, and record the original outlier and its position in an independent outlier table; Step 2: Perform pattern analysis and segmentation on the converted integer sequence, and select a segmentation strategy according to the compression mode: In lossless compression mode, a segmentation algorithm with the goal of minimizing overall storage overhead is adopted. In lossy compression mode, a segmentation algorithm is adopted to cover all data points with the fewest segments within a preset error range; a lightweight prediction model is trained for each segment. Step 3: Store the model parameters, segment boundary information, and residual array between the model prediction value and the actual integer value for each segment, and compress and encode the residual array. The model metadata, segmentation information, compressed residuals, global transformation parameters, and outlier tables are organized into a storage structure that supports fast random access and range queries.
2. The adaptive learning compression method for floating-point data combined with outlier processing according to claim 1, characterized in that, The adaptive rounding transformation described in step 1 specifically includes: for each data block, determining the decimal exponent e and scaling factor f through sampling; for each floating-point number v within the block, calculating round(v×10^(ef)) to convert it into an integer; and recording the transformation parameters e and f for subsequent recovery.
3. The adaptive learning compression method for floating-point data combined with outlier processing according to claim 1, characterized in that, The outliers mentioned in step 1 also include: floating-point numbers that overflow due to excessive precision; special values such as NaN and Inf in the IEEE 754 standard; for special values such as NaN and Inf, the context-aware masquerading mechanism is also used to replace and record them.
4. The adaptive learning compression method for floating-point data combined with outlier processing according to claim 1, characterized in that, The segmentation strategy in the lossless compression mode described in step 2 includes: aiming to minimize the sum of "model parameter storage + segment length × maximum residual encoding bit width"; and using a dynamic programming algorithm to make segment merging decisions.
5. The adaptive learning compression method for floating-point data combined with outlier processing according to claim 1, characterized in that, The segmentation strategy in the lossy compression mode described in step 2 includes: using an online segmentation algorithm based on error constraints; starting from the beginning of the sequence, data points that conform to the current model error range are sequentially included in the same segment until the error exceeds the limit and a new segmentation begins.
6. The adaptive learning compression method for floating-point data combined with outlier processing according to claim 1, characterized in that, The lightweight prediction model described in step 2 is a multinomial regression model, and the model order is adaptively selected based on the data characteristics.
7. The adaptive learning compression method for floating-point data combined with outlier processing according to claim 1, characterized in that, The residual array described in step 3 is compressed using bit packing technology, and the bit width is dynamically determined based on the maximum residual within the segment.
8. A floating-point data adaptive learning compression system incorporating outlier processing, used to perform the method of claim 1, characterized in that, include: The data conversion module is used to perform adaptive conversion from floating-point numbers to integers and to handle outliers and special values. The segmented modeling module is used to perform sequence pattern learning and segmented modeling. The encoding and storage module is used to generate execution parameters and residual encoding and storage structures.
9. A decompression method for decompressing data compressed according to the methods described in claims 1-7, characterized in that, The decompression method includes: receiving a decompression request and determining whether it is a single-point access or a range access; if it is a single-point access, locating the segment where the data point is located, reading the model parameters to calculate the predicted value, obtaining the corresponding residuals and adding them together, and then restoring it to a floating-point number according to the transformation parameters; if the point is recorded as an outlier or an anomaly, directly reading the original value from the independent outlier table for restoration; if it is a range access, locating the involved segments in batches, calculating the predicted value and residuals in parallel, restoring them to floating-point numbers in batches, and replacing the recorded outliers or anomalies within the range.
10. The decompression method according to claim 9, characterized in that, The process of restoring the floating-point number includes: reading the conversion parameters e and f of the corresponding data block; and restoring the integer to a floating-point number by calculating the integer value / 10^(ef).