Ordered key learning compression parameter adaptive configuration method

Abstract

The invention relates to an adaptive configuration method for compression parameters of ordered key learning. The adaptive configuration method comprises the following steps: selecting a public data set containing ordered integer keys; calculating a gap value between two adjacent integer keys so as to obtain a gap value sequence set between all the integer keys; all change point positions of the sequence set are calculated by adopting a Bayesian online change point detection algorithm and mapped back to the original data set, and the original data set is divided into a plurality of partitions with similar statistical characteristics by change points; setting an initial error upper limit threshold value of each partition, and constructing a piecewise linear function of each partition and a corresponding bit-level space total cost function; solving an optimal error upper limit and a corresponding residual bit width in combination with the partition variance and the partition constraint; constructing a configuration portrait library containing optimal configuration; and traversing all partitions, generating a compression parameter adaptive configuration table, constructing a learning compressor according to the compression parameter adaptive configuration table, performing error controlled prediction and residual encoding on an original sequence, and outputting a final compression result.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data compression and database system compression, and particularly relates to an ordered key learning compression parameter adaptive configuration method. BACKGROUND

[0002] With the development of Internet applications and artificial intelligence, the scale of data is growing explosively, and how to efficiently store and retrieve large-scale data has become an important challenge in the field of database systems and information retrieval. In core applications such as vector databases and inverted indexes, the underlying usually relies on the storage and compression of ordered integer keys, such as the codebook index generated by Product Quantization (PQ) and the document identifier (docID) in search engines. However, just storing 20 billion 32-bit unsigned integer (uint32) ordered keys requires up to 80 GB of raw storage, and if the actual overhead of the index structure (such as B+ tree, hash table, etc.) is also considered, assuming that each key's index requires an additional 16 bytes (including pointers, etc.), the actual memory occupancy can reach 400 GB. Such a huge storage requirement not only puts extremely high demands on hardware resources, but also severely limits the scalability and real-time query performance of the system in large-scale data scenarios. In order to reduce storage overhead and improve access efficiency, the compression of ordered keys has become a key operation in data management and information retrieval.

[0003] Traditional general-purpose compression algorithms (such as Huffman encoding, LZ77 / LZ4 / LZMA, etc.) can reduce the data size to some extent, but they are usually based on statistical models and cannot fully utilize the distribution characteristics (such as monotonicity and local correlation) of ordered keys. In addition, compression methods specifically for ordered integer lists (such as Delta, PForDelta, Elias-Fano, VByte, etc.) are widely used in industry and academia, but their compression rates depend on the data distribution characteristics and lack adaptability, and the overall compression effect is limited.

[0004] In 2017, Tim Kraska et al. proposed the concept of learned index in the paper "The Case for Learned Index Structures", which uses machine learning models to replace traditional index structures (such as B+ tree) to significantly improve query performance. Inspired by this, researchers began to extend the idea of learned index to the field of data compression and proposed compression algorithms based on learning models. Learning compressors usually use the "model prediction + residual storage" approach to formalize the compression task as a function approximation problem, and use the prediction model obtained by training to replace the fixed compression rule, achieving higher adaptability and compression rate.

[0005] In the implementation of a learning compressor, the choice of model directly affects the compression effect and decoding efficiency. Although polynomial functions or even neural networks can theoretically be used to finely fit data sequences, such models often introduce high computational complexity and additional storage overhead, which is not conducive to efficient decoding in database systems. Therefore, in order to achieve a good balance between expressive power and inference efficiency, a piecewise linear model with simple structure and strong fitting ability is adopted as a feasible choice. The piecewise linear model fits the ordered key sequence under the constraint of the upper limit of error (∈) and combines residual storage to achieve a compression method that theoretically guarantees lossless reconstruction.

[0006] However, existing learning compression techniques still have the following problems and shortcomings:

[0007] 1. Lack of automated parameter configuration mechanism: Although the learning compression approach of "model prediction + residual storage" inspired by learning index can improve the compression rate, key hyperparameters (such as upper limit of error) are mostly selected through manual experience, grid search or trial and error, which is costly and lacks theoretical support.

[0008] 2. Inability to adapt to the non-stationarity and distribution drift of real data: For real key sequences, gap variance often fluctuates significantly with each segment, potentially influenced by factors such as vector cluster density, inverted index distribution, and temporal evolution. A single global error parameter cannot simultaneously achieve the optimal compression ratio for each segment; and existing learning compression methods lack the ability to detect and adaptively update changes in data distribution.

[0009] 3. Existing partitioning methods lack robustness and online capability: A few methods attempt to pre-partition sequences, but they generally rely on offline heuristic strategies and cannot respond in real time to online data streams, dynamic distributions, and local mutations; in addition, traditional partitioning methods are difficult to unify multi-scale changes and are susceptible to noise.

[0010] 4. Lack of engineered end-to-end solutions: Existing work is mostly at the algorithm prototype stage, and a unified automated toolchain has not been formed. It is impossible to achieve integration from data statistics, partitioning, parameter solving, model building to compression evaluation, which limits the deployment of learning compression technology in production systems. Summary of the Invention

[0011] To address the aforementioned problems in existing technologies, the technical problem this invention aims to solve is: how to improve the overall data compression rate in large-scale ordered key scenarios, while ensuring lossless data restoration and automated configuration of the optimal error upper limit at the partition level, and how to adapt to non-stationary distribution changes and system resource constraints.

[0012] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0013] An ordered key learning compression parameter adaptive configuration method includes the following steps:

[0014] S100: Select a public dataset D, which contains integer keys;

[0015] S200: Extract all integer keys from dataset D and sort all extracted integer keys in ascending order to obtain the original ordered key sequence set K = {k1,…,k...} i ,…,k N} and calculate the gap value between two adjacent integer keys. The gap value between the i-th integer key and the previous integer key is taken as the i-th gap value, denoted as gap. i The calculation formula is as follows:

[0016] gap i =k i -k i-1

[0017] Where, at this time, i takes the value [2,...,N], and k N This represents the Nth integer key, where j1≤k2≤…≤k N N represents the total length of the data;

[0018] All gap values ​​in K obtained by the above method constitute a gap sequence set;

[0019] S300: Preset probability threshold θ, use the Bayesian online change point detection algorithm BOCPD to calculate the posterior probability of the gap value changing at time step t for all gap values ​​in the gap sequence set;

[0020] The posterior probability of all calculated gap values ​​is compared with a preset probability threshold, and points that are greater than or equal to the preset probability threshold are designated as change points.

[0021] Once all the variable points are obtained, all variable point positions are mapped back to K, and K is divided into J partitions based on the variable point positions.

[0022] S400: Select the j-th partition from J partitions. j And set an initial upper limit threshold for error ∈ j Then targeting P j Construct a piecewise linear model ∈-PLA, and P j The j-th partition P is obtained by fitting the ordered key sequence input ∈-PLA. j piecewise linear function f j (i j ), f j (i j The expression for ) is as follows:

[0023]

[0024] Among them, i j P represents j The original ordered key index is contained within, and each partition corresponds to a piecewise function, α. l,j β represents the slope of the piecewise function in the l-th segment within the j-th partition. l,j s represents the intercept of the piecewise function in the l-th segment within the j-th partition. l Let f(·) represent the ordered key of the starting point of the l-th piecewise function, and L represent the total number of piecewise functions; the function f(·) needs to satisfy the constraint condition of the upper limit of error:

[0025] S500: Building about f j (i j The bit-space total cost function B) j (∈ j ), the B j (∈ j This includes the piecewise linear model overhead and the residual bit width overhead, calculated as follows:

[0026]

[0027] in, Let C represent the variance of the gap sequence set, and K represent the fitting constant. bit F represents the number of bits required to encode the integer key. bit The number of bits required to represent the slope and intercept; N j P represents j The total number of ordered keys on the array;

[0028] S600: P j When the gap sequence satisfies iid, make B j (∈ j Minimum f j (i j The optimal upper limit of error ∈ j * ,∈ j * The calculation formula is as follows:

[0029]

[0030] The iids are independent and identically distributed;

[0031] According to ∈ j * The residual bit width of partition j is calculated using the following formula:

[0032]

[0033] Among them, b j Indicates the residual bit width within the j-th partition;

[0034] S700: According to ∈ j * and b j Construct P j The partition bit width configuration;

[0035] S800: Traverse all J partitions, repeat S400-S700, and obtain the partition bit width configuration for each partition. All partition bit width configurations constitute the compression parameter adaptive configuration table list of K.

[0036] S900: The piecewise functions corresponding to each partition in J are aggregated to form a learning compressor Z with respect to K. The calculation expression is as follows:

[0037] Z = {f1,…,f j ,…,f J}

[0038] Among them, f j Represents the piecewise function for the j-th partition;

[0039] Input K into Z, and based on the compression parameter adaptive configuration table list, output the compression result K' of K.

[0040] Preferably, the S500 constructs information about f j (i j The bit-space total cost function B) j The steps for (∈) are as follows:

[0041] S510: According to the expected segment theory, satisfying ∈ j Sufficiently large, i.e., ∈ j >>σ j / μ j When, calculate f j (i j The expected number of keys to be covered The calculation formula is as follows:

[0042]

[0043] Where, μ j This represents the mean of all gaps. Denotes the variance of the gap sequence set, ∈ j f j (i j The maximum upper limit of error;

[0044] S520: Fitting length is N jGiven an ordered integer key sequence, the desired number of segments is calculated as follows:

[0045]

[0046] Right now

[0047]

[0048] Where O represents the calibration constant, N j P represents j The total number of ordered keys on the array;

[0049] S530: Based on the Expected Segment Count theory, we obtain information about f. j (i j The number of segments in the segment is L(∈ j The calculation formula is as follows:

[0050]

[0051] Where C represents the calibration constant;

[0052] S540: Using L(∈ j Calculate f j (i j The piecewise linear model cost B structure (∈ j The calculation formula is as follows:

[0053]

[0054] Among them, K bit F represents the number of bits occupied by the starting position of a piecewise function. bit K represents the slope intercept of the corresponding piecewise function as the number of bits occupied by the floating-point number. bit +2F bit This represents the total number of bits used to account for the cost of each piecewise function segment;

[0055] The expected number of pieces in a piecewise function is calculated using the following formula:

[0056] S550: Calculate residual bit width overhead B residual (∈ j The calculation formula is as follows:

[0057]

[0058] S560:B structure (∈ j ) and B residual (∈ j ) constitutes B(∈ j The calculation formula is as follows:

[0059] B j (∈ j ) = B residual (∈ j )+B structure (∈ j );

[0060] Preferably, in S600, the optimal upper limit of error is calculated ∈ j * The steps are as follows:

[0061] log2(2∈ j +1) is approximately a continuous function, i.e., log2(2∈ j +1)≈log2(2∈ j ), at this point, reconstruct B j (∈ j The function expression is as follows:

[0062]

[0063] For the reconstructed B j (∈ j ) for ∈ j Taking the derivative and setting it to zero yields the optimal upper limit of error ∈ j * The process is as follows:

[0064]

[0065] Preferably, the steps for calculating the change point position using BOCPD in S300 are as follows:

[0066] The BOCPD algorithm is used to recursively update the gap sequence set point by point. At time step t, the prior transition probability of the run-length is determined based on the hazard function, and combined with the observation likelihood model. Calculate the posterior distribution p(r) of the run-length. t =0|gap 1:t );

[0067] When the probability is greater than or equal to the preset probability threshold θ, time step t is marked as a change point; and the detected change point position is mapped to the corresponding position in the original key sequence K. K is divided into J partitions according to the corresponding position, so that the gap distribution within each partition remains relatively consistent.

[0068] Compared with the prior art, the present invention has at least the following advantages:

[0069] 1. This invention achieves automated parameter configuration, eliminates manual trial and error, and proposes a partition-level bit-level spatial cost function. Within a partition, the optimal upper limit value and bit width value can be directly obtained using a closed-form solution. It no longer relies on empirical parameter tuning or a large number of experimental trials. Based on known data statistics, an approximate optimal configuration can be given at once, significantly reducing engineering costs and effectively eliminating reliance on manual experience.

[0070] 2. This invention uses BOCPD to perform online change point detection on gap sequences, automatically dividing the original ordered key sequence into several partitions with similar statistical characteristics. Within each partition, an adaptive optimal upper limit value is independently calculated, and the residual bit width is calculated separately for each segment. For non-stationary scenarios where gap variance varies significantly across different segments, this invention can optimize the compression configuration of each partition based on local distribution characteristics, overcoming the performance bottleneck of a single global parameter in heterogeneous scenarios and effectively improving the overall compression rate.

[0071] 3. The partitioning scheme proposed in this invention supports online distribution drift monitoring and automatic maintenance of the partition structure. In scenarios with long-term operation or continuous incremental data, this invention can continuously monitor distribution drift using BOCPD output and gap statistical features. When a change in local statistical properties is detected, it automatically splits the corresponding partition or merges adjacent partitions, while updating the adaptive configuration table list of compression parameters and the learning compressor Z. This achieves dynamic adaptive maintenance of the model structure and parameters, avoiding frequent manual intervention.

[0072] 4. Introducing a configuration profile library to enable the migration and reuse of historical configurations. While generating configuration tables by traversing each partition, the invention constructs a configuration profile library, recording the statistical characteristics of each partition and its corresponding optimal parameters. When encountering new data with similar statistical characteristics, the profile library can be used for rapid matching and configuration migration without re-executing the complete parameter solving process, thereby significantly shortening system cold start time and reducing computational overhead.

[0073] 5. Parameter solving is linked to system resource constraints, allowing for adaptive adjustment based on budget. This invention, when constructing the cost function and solving for the optimal error upper limit, can take into account system resource constraints such as the overall compressed bit count, memory budget, or decoding latency, and adjust these constraints by pruning or normalizing ∈ j * With b j This approach achieves a controllable balance between compression performance and resource consumption, making it more suitable for deployment in real-world industrial systems.

[0074] 6. The constructed model is concise and the data can be lossless and reversible. It adopts the ∈-PLA piecewise linear model and stores the piecewise parameter table + residual array + bit width configuration. It avoids the inference and storage overhead caused by high-complexity models (polynomial / neural network). It effectively constrains the upper limit of error and achieves fixed-width encoding through residuals. It ensures that the original key sequence can be completely recovered after compression. The decoding formula is also concise and reliable.

[0075] 7. This invention employs BOCPD to perform point-by-point recursive posterior updates on the gap sequence, enabling real-time determination of local statistical mutations without rescanning historical data when a new gap arrives. Compared to PELT, which requires processing the entire dataset at once, has a fixed partitioning structure, and cannot be incrementally maintained, BOCPD possesses online detection capabilities, updating the posterior distribution in a run-length probability recursive manner. This allows for real-time detection of change points in streaming data, continuously appended data, and long-running systems, without requiring repetitive overall calculations. BOCPD also supports dynamic partition maintenance. When the gap distribution within a partition suddenly changes, BOCPD immediately increases the probability of the change point, triggering partition splitting. When the distributions of two adjacent partitions converge, the system can automatically merge them, reducing model redundancy. This capability allows the partitioning structure to adapt to the statistical characteristics of the data in real-time, significantly improving the applicability of this invention in real-world environments. PELT cannot achieve such dynamic partitioning correction. BOCPD is also more adaptable to complex distributions. The replaceable observation likelihood model allows it to adapt to common abrupt changes, skewed distributions, heavy-tailed noise, and weak multimodal structures in real-world data, resulting in more stable and accurate partitioning. In contrast, PELT's cost function typically implies a Gaussian assumption, making it prone to false positives or false negatives when encountering complex distributions. Furthermore, PELT's results are highly dependent on penalty selection; slight differences in penalty settings can lead to significant variations in the number of partitions. BOCPD, on the other hand, automatically determines the segmentation strength through a hazard function and a likelihood model, eliminating the need for manual parameter tuning and significantly improving automation. Attached Figure Description

[0076] Figure 1 This is a schematic diagram of the ordered key learning compressor (left) and the coordinate diagram of the error boundary piecewise linear model (right) in this invention.

[0077] Figure 2 This is a schematic diagram of the overall framework of the present invention.

[0078] Figure 3 Compression ratio curves for each dataset at different residual bits: (a) SIFT dataset, (b) AUDIO dataset, (c) Deep1B dataset.

[0079] Figure 4This is a schematic diagram of the user interface of the present invention in actual use, including data exploration, partitioning, parameter optimization and benchmark evaluation panels. Detailed Implementation

[0080] The present invention will now be described in further detail.

[0081] See Figures 1-4 An ordered key learning compression parameter adaptive configuration method includes the following steps:

[0082] S100: Select a public dataset D, which contains an integer key; the integer key is a 32 / 64-bit unsigned integer, and the data source can be a vector database codebook index, inverted index docID, or other sortable datasets containing one-dimensional integer keys;

[0083] S200: Extract all integer keys from dataset D and sort all extracted integer keys in ascending order to obtain the original ordered key sequence set K = {k1,…,k...} i ,…,k N} and calculate the gap value between two adjacent integer keys. The gap value between the i-th integer key and the previous integer key is taken as the i-th gap value, denoted as gap. i The calculation formula is as follows:

[0084] gap i =k i -k i-1

[0085] Where, at this time, i takes the value [2,...,N], and k N Let k1 represent the Nth integer key, and k1≤k2≤…≤k N N represents the total length of the data;

[0086] All gap values ​​in K obtained by the above method constitute a gap sequence set;

[0087] S300: Preset probability threshold θ, use the Bayesian online change point detection algorithm BOCPD to calculate the posterior probability of the gap value changing at time step t for all gap values ​​in the gap sequence set;

[0088] The posterior probability of all calculated gap values ​​is compared with a preset probability threshold, and points that are greater than or equal to the preset probability threshold are designated as change points.

[0089] Once all the variable points are obtained, all variable point positions are mapped back to K, and K is divided into J partitions based on the variable point positions.

[0090] The steps for calculating variable points using BOCPD in S300 for partitioning are as follows:

[0091] First, taking the gap sequence set as the input, the BOCPD algorithm is used to perform point-by-point recursive update on the gap sequence set. At time step t, the prior transition probability of the run-length is determined based on the hazard function, where r t represents the length of the samples continuously observed since the most recent change point at time step t (corresponding to the t-th gap); for each time step t, based on the prior probability of the change point represented by the prior hazard function h(·) and the observation likelihood model based on the Gaussian distribution, the posterior distribution formula is recursively updated as follows:

[0092]

[0093] where are the parameters of the gap distribution within the interval (the mean and variance of the Gaussian distribution), and p(r t |r t-1 ) is given by the hazard function and is used to control the prior probability of the change point; when p(r t =0|gap 1:t )>θ, t is marked as a change point, where θ is the preset change point probability threshold; both the prior hazard function and the observation likelihood model are prior arts;

[0094] where are the statistical parameters of the gap distribution within the interval, including the mean μ and the variance σ 2 , and the observation likelihood adopts Gaussian likelihood, which is expressed as: The transition probability of the run-length is given by the hazard function, and the hazard function h(·) represents the prior probability of a change point occurring at any time step. For ease of engineering implementation, the present invention selects a commonly used fixed constant hazard, that is, it is assumed that there is the same prior probability h of a change point at any time step. Based on this hazard, the transition probability of the run-length can be obtained: p(r t =0|r t-1 ) = h indicates that a new change point occurs at time step t; and: p(r t =r t-1 +1|r t-1 ) = 1 - h indicates that the current section continues to extend by one sample. Here, 0 < h < 1 is a preset constant used to control the prior intensity of the change point. According to the above recursive update, the posterior probability of run-length = 0 (i.e., a change point occurs) can be obtained: p(r t =0|gap 1:t ). When the condition p(r t =0|gap 1:t )≥θ is satisfied, the position t is marked as a change point, where θ is the preset change point probability threshold

[0095] Through the recursive update described above, the locations where abrupt changes in the statistical properties of the gap sequence can be identified online and efficiently while minimizing the global cost. The system applies BOCPD, taking the gap sequence as input and returning a set of change point indices {τ1, τ2, ..., τ...}. m}, and map these variable point indices back to the original key sequence K, dividing the entire key sequence into m+1 intervals: the gap distribution within each interval is statistically relatively uniform, which is more suitable for fitting local ∈-PLA models separately, thus maintaining a good compression ratio.

[0096] Furthermore, in online or incremental write scenarios, BOCPD operates continuously in a streaming manner: when a new batch of gap samples is added to the sequence, the system updates the posterior p(r) based on the incremental update. t |gap 1:t Reassess the probability of change points in the new tail interval; if the probability of a change point at a certain position exceeds the threshold θ, then generate a new change point at that position, and partition the original P. j Automatically split into two sub-partitions P j,1 ,P j,2 And re-execute S400–S700 on both sub-partitions; simultaneously, for adjacent partition P j P j+1 If its gap distribution statistical characteristics (mean μ) j ,μ j+1 With variance )satisfy This can trigger partition merging, and P j With P j+1 Merge into a new partition And recalculate the corresponding optimal upper limit of error based on the merged statistics. With bit width b j This allows for control over the number of partitions and management overhead while ensuring modeling accuracy.

[0097] S400: Select the j-th partition from J partitions. j And set an initial upper limit threshold for error ∈ j Then targeting P j Construct a piecewise linear model ∈-PLA, and P j The j-th partition P is obtained by fitting the ordered key sequence input ∈-PLA. j piecewise linear function f j (i j ), f j (i j The expression for ) is as follows:

[0098]

[0099] Among them, i j P represents j The original ordered key index is contained within, and each partition corresponds to a piecewise function, α. l,j β represents the slope of the piecewise function in the l-th segment within the j-th partition. l,j s represents the intercept of the piecewise function in the l-th segment within the j-th partition. l Let f(·) represent the ordered key of the starting point of the l-th piecewise function, and L represent the total number of piecewise functions; the function f(·) needs to satisfy the constraint condition of the upper limit of error: By recording the model parameters (α) for each segment l,j ,β l,j and the residuals at each location The original sequence can then be restored; since the residual is restricted to the interval [-∈ j ,+∈ j Therefore, 2∈ j +1 possible values, encoding the number of bits b of the residual at each position. residual for:

[0100] S500: Building about f j (i j The bit-space total cost function B) j (∈ j ), the B j (∈ j This includes the piecewise linear model overhead and the residual bit width overhead, calculated as follows:

[0101]

[0102] in, Let C represent the variance of the gap sequence set, and K represent the fitting constant. bit F represents the number of bits required to encode the integer key. bit The number of bits required to represent the slope and intercept; N j P represents j The total number of ordered keys on B; in constructing B j (∈ j In addition, system resource constraints can be introduced according to different application scenarios. This is a major function of the BOCPD partitioning algorithm. It can add appropriate conditions to constrain different scenarios without affecting the operation of the overall model. It is mainly used to characterize the total compression bit budget, memory limit or decoding latency constraints, so as to achieve a balance between compression ratio and resource consumption.

[0103] The S500 constructs about f j (i j The bit-space total cost function B)j The steps for (∈) are as follows:

[0104] S510: According to the expected segment number theory, which is an existing technology, it satisfies ∈ j Sufficiently large, i.e., ∈ j >>σ j / μ j When, calculate f j (i j The expected number of keys to be covered The calculation formula is as follows:

[0105]

[0106] Where, μ j This represents the mean of all gaps. Denotes the variance of the gap sequence set, ∈ j f j (i j The maximum upper limit of error;

[0107] S520: Fitting length is N j Given an ordered integer key sequence, the desired number of segments is calculated as follows:

[0108]

[0109] Right now:

[0110]

[0111] Where O represents the calibration constant, N j P represents j The total number of ordered keys on the array;

[0112] S530: Based on the Expected Segment Count theory, we obtain information about f. j (i j The number of segments in the segment is L(∈ j The calculation formula is as follows:

[0113]

[0114] Where C represents the calibration constant;

[0115] S540: Using L(∈ j Calculate f j (i j The piecewise linear model cost B structure (∈ j The calculation formula is as follows:

[0116]

[0117] Among them, K bit F represents the number of bits occupied by the starting position of a piecewise function. bit K represents the slope intercept of the corresponding piecewise function as the number of bits occupied by the floating-point number. bit +2F bit This represents the total number of bits used to account for the cost of each piecewise function segment;

[0118] The expected number of pieces in a piecewise function is calculated using the following formula:

[0119] S550: Calculate residual bit width overhead b residual (∈ j The calculation formula is as follows:

[0120]

[0121] S560:B structure (∈ j ) and B residual (∈ j ) constitutes B(∈ j The calculation formula is as follows:

[0122] B j (∈ j ) = B residual (∈ j )+B structure (∈ j ).

[0123] For the constant C, this invention fits a simple and effective linear regression model on the collected dataset, the process of which is as follows: collect 100 sets of sorting keys with different specifications and gaps, and then change ∈ from {2 1 ,2 2 ,…,2 10 To physically construct the ∈-PLA model, for each ∈, the actual piecewise count L is recorded and fitted to the equation. Through extensive testing, this invention was ultimately adopted. As an estimate of C, it achieves consistently high accuracy across different datasets; in practical applications, it can also be used to estimate B. j (∈ j Explicitly add system resource-related penalty or constraint terms to control the overall number of compressed bits or peak memory usage.

[0124] In resource-constrained scenarios, this invention further introduces system resource constraints to extend the partition-level objective function, so that the upper limit of error ∈ jThe choice of optimization method balances compression ratio and availability of engineering resources. In resource-constrained application scenarios, this invention further introduces a system resource constraint term based on the original bit-level objective function to construct an extended partition-level optimization objective. Specifically, by adjusting the bit-width objective function B of the partition... j (∈ j By adding a resource penalty term to the equation, we obtain the new comprehensive objective function as follows:

[0125]

[0126] Where λ represents a coefficient that adjusts the trade-off between compression ratio and resource consumption, and R j (∈ j ) represents a penalty function related to system resource constraints, which can simultaneously consider the partition's upper limit of the candidate error ∈ j Factors such as peak memory usage, decoding latency or throughput limits during decompression, and disk or network bandwidth budgets must be considered. In practical implementation, R can be flexibly designed according to different business needs. j (∈ j The present invention adopts a form that achieves an engineering compromise between "optimal compression efficiency" and "controllable resource overhead." Based on this, the present invention extends the objective function... As an optimization criterion, by solving ∈ j * This allows us to obtain the optimal upper limit of partition error that simultaneously satisfies bit-level compression performance and system resource constraints.

[0127] In the basic embodiment that does not consider resource constraints, B j (∈ j ) as the optimization objective; in embodiments considering resource constraints, with Replace B j (∈ j As the optimization objective, the derivation process is similar and will not be repeated here.

[0128] S600: P j When the gap sequence satisfies iid, make B j (∈ j Minimum f j (i j The optimal upper limit of error ∈ j * ,∈ j * The calculation formula is as follows:

[0129]

[0130] The iid is independent and identically distributed; the independent and identically distributed iid is existing technology.

[0131] According to ∈ j * The residual bit width of partition j is calculated using the following formula:

[0132]

[0133] Among them, b j Indicates the residual bit width within the j-th partition;

[0134] In S600, the optimal upper limit of error is calculated. j * The steps are as follows:

[0135] log2(2∈ j +1) is approximately a continuous function, i.e., log2(2∈ j +1)≈log2(2∈ j ), at this point, reconstruct B j (∈ j The function expression is as follows:

[0136]

[0137] For the reconstructed B j (∈ j ) for ∈ j Taking the derivative and setting it to zero yields the optimal upper limit of error ∈ j * The process is as follows:

[0138]

[0139] Bit-level space cost function B j (∈ j This function exhibits a typical U-shaped structure, meaning the total cost of compression is within the range of ∈ j The error is relatively high whether the error is small or large, meaning there exists an optimal error ∈ j * This minimizes the overall compression cost. To obtain this optimal error ∈ j * This allows for the compression of the cost function B. j (∈ j ∈ j Find the minimum point by taking the derivative;

[0140] S700: According to ∈ j * and b j Construct P j The partition bit width configuration;

[0141] S800: Traverse all J partitions, repeating S400-S700 to obtain the partition bit width configuration for each partition. All partition bit width configurations constitute the compression parameter adaptive configuration table list of K. Simultaneously, the statistical characteristics N of each partition can be optionally included. j ,μ j , and the corresponding ∈ j * ,b j Store the configuration profile in the database; this will be used for rapid matching and parameter migration when processing new data with similar statistical characteristics, reducing the cost of re-solving.

[0142] S900: The piecewise functions corresponding to each partition in J are aggregated to form a learning compressor Z with respect to K. The calculation expression is as follows:

[0143] Z = {f1,…,f j ,…,f J}

[0144] Among them, f j Represents the piecewise function for the j-th partition;

[0145] Input K into Z, and based on the compression parameter adaptive configuration table list, output the compression result K' of K. For online or incremental data scenarios, distribution drift can be monitored based on the continuous output of BOCPD and changes in gap statistical characteristics. When a significant change in the statistical properties within a certain partition is detected, the partition is automatically split or adjacent partitions are merged, and the partition structure, configuration table list, and configuration profile library are updated accordingly, realizing the dynamic adaptive maintenance of the learning compressor Z.

[0146] Z is a set of several error-controlled piecewise linear functions, defining the prediction mapping structure for K; K is input to the corresponding partition's f j Make predictions and record the residuals: Further expansion generates the final compressed result K'={(f j ,Δ j )|j=1,…,J}, where the learning compressor Z is a set of error-controlled piecewise linear functions, and the compression result K′ is the output of the compressor after it is applied to the original key sequence, which includes model parameters and residual information.

[0147] This invention uses an automatic partition parameter configuration method to independently determine the optimal upper limit of error for each partition. j * To minimize the bit-level space cost function ∈ j * The optimal upper limit of error ∈ j *The piecewise linear model ∈-PLA is used as input parameters to fit the ordered key sequence within the partition; during the compression phase, based on the optimal ∈ j * Calculate the corresponding residual bit width b j And construct a piecewise linear function f within each partition. j (i j The piecewise functions of all partitions and the residuals together form a learning compressor, which is subject to error constraints. Theoretically, lossless reconstruction can be achieved.

[0148] During the verification process, a learning compressor is constructed based on the segmentation parameters and residual bit width, and the compression result is output. This compressor uses the optimal upper limit of error ∈ j * piecewise linear function f j (i j This ensures lossless restoration of the original data. The output compression result includes a segmentation parameter table, a residual bit configuration table, and a residual array. A "residual bit width - compression ratio" curve is plotted, and the optimal bit width is marked for visualization. The compression ratio is calculated using the following formula:

[0149]

[0150]

[0151] The output can be decoded using the following decoding formula:

[0152]

[0153] Using the decoding formula is equivalent to knowing the residual and the fitted piecewise function value at each position, and then our true value is... Calculated, Δ i Let f(i) represent the residual of the i-th integer key, and let f(i) represent the piecewise fitting function to which the i-th integer key belongs.

[0154] In a further embodiment of the present invention, to adapt to the common scenarios of online appending, batch importing, and long-term operation in practical databases and vector retrieval systems, the present invention constructs an online incremental compression mechanism for streaming data. When the system receives a new key sequence based on the existing compressed result K′, it can perform incremental compression updates at the partition level without reprocessing all historical data, thereby improving the system's scalability and engineering feasibility.

[0155] For the new key k appended to the end of the sequence new The system first determines the statistical interval to which the key belongs based on the BOCPD online change point detection method described in step S300, and obtains its corresponding partition P. jIf the newly added key belongs to an existing partition P. j The system queries the configuration profile database and extracts the optimal parameters stored historically for that partition. Reuse the corresponding piecewise linear function f j And based on this, calculate the residual of the newly added bond: The residual corresponding to the newly added key is then directly appended to the compressed result K′, achieving seamless splicing of the new data stream without the need for recompression of historical data.

[0156] To address the distribution drift that is prevalent in real-world business scenarios, this invention continuously maintains the statistical characteristics of each partition (including the gap mean μ) during system operation. j ,variance (residual distribution, etc.) and resource function R j (∈ j ), where R j (∈ j ) is about the upper limit of error ∈ j The resource cost estimation function is also used as a resource penalty term in the extended objective function to measure the comprehensive resource cost of the partition under the current configuration, including the number of model segments, residual bit width, memory usage, decoding latency, or bandwidth consumption. If a partition's latest statistical characteristics deviate significantly from the historical statistics recorded in the profile database, or if the resource index R... j (∈ j If the threshold is exceeded, the system will automatically trigger a local reconfiguration process and re-execute steps S300–S700 corresponding to that partition.

[0157] During local reconfiguration, if a partition exhibits a significant multimodal structure or local statistical abrupt changes, the system automatically performs partition splitting, decomposing the original partition into multiple sub-partitions and solving for the local optimal parameters in each sub-partition. b j And piecewise functions; if the statistical characteristics of two adjacent partitions are highly consistent, partition merging is performed to reduce the number of model segments and overall overhead. After local reconfiguration is completed, the new partition parameters will cover the corresponding segments in the compression result K′, and the learning compressor set Z will be updated, thereby realizing dynamic adaptive maintenance of model structure and compression parameters.

[0158] Furthermore, in resource-constrained scenarios, this invention introduces system resource budget constraints when solving for the optimal error upper limit to balance compression ratio and engineering constraints. For each candidate error upper limit ∈ j All systems can be based on bit-level space cost B j (∈ j Derive the corresponding resource overhead R j (∈ j And construct an extended optimization objective: Where λ is the tradeoff coefficient, R j (∈ j ) is the resource penalty function. The system uses Solve for the optimal upper limit of error for the optimization object. This ensures a high compression ratio while meeting engineering constraints such as memory budget, maximum number of segments, and real-time decoding. Through this mechanism, the present invention can achieve adaptive configuration and dynamic maintenance of ordered key learning compression parameters in various scenarios, including online services, large-scale databases, and memory-constrained devices, thereby improving the long-term stability and resource controllability of the system.

[0159] Evaluation and Visualization

[0160] Compression ratio and compression / decompression time were evaluated and compared with baseline methods, including LZ4, LZMA, and Huffman. On the PQ codebook, the original 3.8 GiB was compressed to 0.88 GiB.

[0161] Systematic implementation of devices / modules

[0162] This invention constructs an end-to-end automated learning compression device, covering the entire process from data exploration to compression evaluation. The system first preprocesses the input integer key sequence through a data exploration module, including key value sorting, gap sequence calculation, and its statistical characteristics and visualization analysis. Then, an online partitioning module performs change point detection on the gap sequence based on the BOCPD method, automatically identifying segment boundaries and outputting a partition structure with local stationarity. Based on this, a residual bit allocation module automatically calculates the upper limit of error within each partition according to the closed-form optimal solution proposed in this invention. With the corresponding residual bit width b j It generates a "residual bit width - compression ratio" relationship curve and marks the optimal bit width configuration; then the compressor construction module uses the piecewise linear function f for each partition. j The residual array forms the overall structure Z of the learned compressor. Finally, the evaluation and visualization module measures metrics such as compression ratio, decompression speed, and resource consumption, and compares them with baseline algorithms such as LZ4 and LZMA. It also provides visualization results for partition structure, variable point posterior, and budget constraint optimization. These modules together constitute a fully adaptive compression system that can run online, be incrementally updated, and be budget-constrained, giving the invention high engineering usability and scalability. Example: Application on vector database codebook data.

[0163] To verify the effectiveness and universality of the method of this invention, this embodiment was used in a typical vector database codebook scenario. The public dataset SIFT1B was selected as the data source. This data source is widely used in vector retrieval and nearest neighbor search research. Its code text files consist of a large number of 32-bit unsigned integer keys. The key values ​​have strict monotonicity and local stationarity, making them suitable for compression using an error-controlled piecewise linear model.

[0164] (1) Data preprocessing

[0165] First, the original codebook key sequence is sorted in ascending order, and the adjacent key difference (gap) sequence is calculated. By visualizing the gap distribution, it can be observed that the sequence has significant variance differences in different segments, indicating that the data distribution is non-stationary.

[0166] (2) Partitioning and parameter configuration

[0167] To address the non-stationarity of gap sequences, this embodiment employs the BOCPD (Bayesian Online Change Point Detection) method for change point detection. BOCPD recursively updates the run-length posterior distribution, simultaneously handling offline data partitioning and online data appending, and then automatically outputs several partitions with relatively consistent variance.

[0168] The gap variance was then calculated within each partition. And based on the bit-level space cost function proposed in this invention:

[0169]

[0170] Find the upper bound of the local optimum error:

[0171]

[0172] This automatically generates a residual bit width configuration table for each partition, and sets N... j ,μ j , ∈ j * ,b j Write it into the image database.

[0173] (3) Compression and decompression verification

[0174] Within each partition, with the optimal upper limit of error. Fitting error controlled piecewise linear function f j (i j This forms a set of learning compressors:

[0175] Z = {f1, f2, ..., f J}

[0176] Record residuals:

[0177]

[0178] The compression result is expressed as:

[0179] K′={(f j ,Δ j )|j=1,…,J}

[0180] By decoding the formula:

[0181]

[0182] It can achieve lossless recovery of the original sequence in experiments.

[0183] (4) Results and Analysis (Offline Mode)

[0184] The method of this invention is implemented on SIFT1B codebook data (original 3.8 GiB):

[0185] After compression, it occupies only 0.88 GiB, with a compression ratio of approximately 4.32×.

[0186] Compared to common compression algorithms such as LZ4 and LZMA, the compression ratio is improved by approximately 37%–62%;

[0187] The decompression time is improved by about 3.1 times compared to LZMA, and is basically close to that of LZ4;

[0188] The theoretically calculated optimal residual bit width matches the peak value of the actual compression ratio curve, verifying the correctness of the bit-level space cost model and the closed-form optimal solution. This invention can significantly reduce ordered key storage overhead while maintaining lossless recovery, exhibiting strong adaptability and engineering practicality.

[0189] (5) Example of online incremental update and distribution drift

[0190] To verify the online adaptive capability of this invention, a simulation was conducted where the vector retrieval system continuously appends the PQ codebook index:

[0191] Calculate the newly added gap sequence and input it into the BOCPD online posterior update module; if the newly added key belongs to an existing partition P j The system directly reuses images stored in the image database. Only the newly added residuals are calculated and appended to K′; if a significant distribution shift is detected, partitioning is automatically performed and the new partitions are solved. b j,a ,f j,aIf adjacent segments have the same statistical height, the system will automatically merge the partitions to reduce structural overhead. This mechanism allows the system to seamlessly expand new data without recompressing historical data.

[0192] (6) Example of a resource budget constraint scenario

[0193] In certain engineering scenarios (such as online vector retrieval, storage engines, and embedded databases), the system not only needs to achieve a high compression ratio but also must meet strict resource constraints, such as decoding latency budget, maximum tolerable number of model segments, memory usage, or encoding byte count budget. To adapt to these practical deployment conditions, this invention further introduces resource constraint terms into the original bit-level space cost function, constructing an extended optimization objective function: Where λ is the tradeoff coefficient, R j (∈ j ) is the resource penalty function, and different forms are selected according to different system requirements. The system uses Solve for the optimal upper limit of error for the optimization object. This ensures a high compression rate while meeting engineering constraints such as memory budget, maximum number of segments, and real-time decoding.

[0194] Taking the real-time constraint of decoding as an example, the decoding delay mainly consists of two parts: "segment lookup + residual decoding", which can be modeled as: Delay j (∈ j )=α·L j (∈ j )+β·b j (∈ j ). Among them: L j (∈ j ) indicates the number of segments; the more segments, the slower the model lookup. j (∈ j Let be the residual bit width. The larger the residual bit width, the slower the decoding bit operation. Assuming the system requires a latency ≤ 0.9μs, then define: Final goal solution: Achieving an engineering compromise between compression ratio and real-time performance.

[0195] Experimental content and results

[0196] 1. Practical application effect

[0197] 1) The compressed configuration can be proven to be optimal, and the results are visually verifiable.

[0198] The theory gives ∈ * With the optimal residual bit width mapped therefrom The results closely match the measured optimal points: the system plotted "residual bit width b versus compression ratio" curves on multiple real datasets, with the marked values... The results are largely consistent with the observed peak values, proving the correctness of the cost model and automatic configuration strategy.

[0199] 2) Achieve higher compression ratios on large-scale data with minimal efficiency loss.

[0200] The system actually builds and learns the compressor in the "Benchmark" test and reports the compression ratio and compression / decompression time; it supports comparison with baselines such as LZ4 / LZMA / Huffman. Experimental results indicate that BitTuner achieves the highest compression ratio with minimal efficiency loss on various integer compression tasks.

[0201] 3) Typical application example: Significantly reducing the size of the PQ codebook for vector databases.

[0202] In the SIFT1B PQ codebook scenario, the system compressed the original 3.8 GiB codebook to 0.88 GiB, demonstrating the solution's capacity to reduce the size of real business data.

[0203] 4) Applicable to multiple scenarios, covering vector libraries and inverted indexes

[0204] The system incorporates built-in PQ codebook data such as SIFT1B, Deep1B, and AUDIO, as well as inverted lists of document IDs from CCNews and Clueweb, and provides two types of synthetic data (Uniform / Normal). The automatic configuration strategy applies equally to these differently distributed datasets, enhancing the method's generalizability.

[0205] 5) Online incremental updates are stable and reliable, significantly reducing recalculation costs.

[0206] In online append scenarios, this invention utilizes BOCPD (Online Bayesian Variable Point Detection) to identify partition stability. By reusing historical parameters through a profile cache, reconfiguration is performed only on a small range of affected partitions. With continuous online incremental appending, the compression ratio remains consistent without significant fluctuations; decompression latency is stable, and there is no performance degradation caused by distribution drift.

[0207] 6) Adjusting solutions based on user needs in resource budget-constrained scenarios.

[0208] When introducing resource constraints (such as delayed budgets) max In scenarios where ), this invention solves the extended objective function.

[0209]

[0210] Experiments show that under a delay budget, the system automatically selects a larger error upper limit, reducing the number of segments and lowering the decoding latency. Compared to the unconstrained case, although the compression ratio decreases, the system stability is significantly improved. This proves that the present invention can balance "optimal compression ratio" and "controllable system resources".

[0211] 2. Experiment Content

[0212] 1) Data and task settings.

[0213] This invention has been experimentally validated on multiple real and synthetic datasets, covering typical scenarios of vector databases and inverted indexes, ensuring the method's broad applicability and generalization ability. Vector databases include PQ codebooks such as SIFT1B, Deep1B, and AUDIO, which have large data volumes; inverted document ID lists such as CCNews and Clueweb. Synthetic data uses a Uniform / Normal distribution to verify optimality and robustness under ideal distributions. Furthermore, the system supports user-uploaded custom datasets, ensuring the method's scalability and portability.

[0214] 2) Evaluation Indicators

[0215] To comprehensively evaluate the performance of the method of this invention, the system adopts the following indicators:

[0216] Table 1 Evaluation metrics for compression algorithms

[0217]

[0218] 3) Experimental Procedure

[0219] The experimental workflow is managed uniformly by the system and is mainly divided into four modules: Data Exploration, Partition, Allocation, and Benchmark. These four modules can be used together to complete a complete compression process, or they can be used independently, allowing users to operate as needed.

[0220] The detailed steps are as follows:

[0221] Data Exploration module: When users select built-in datasets or upload custom data, the system will automatically sort the data and generate sorting charts and gap distribution charts to help users understand the overall structure of the data;

[0222] The Partition module is primarily used to segment the data. It automatically divides the data into segments using the BOCPD algorithm, generating the mean and variance for each segment. Considering that directly running BOCPD on a dataset of billions of keys is still costly, the system employs a sampling strategy in its engineering implementation: by default, only a 10% subset of the original key set K is sampled to perform BOCPD, balancing the estimation accuracy of the partitioning results with operational efficiency. The data is then mapped back to the original data. This ratio is an empirical value that significantly reduces partitioning overhead while maintaining high accuracy. Users can, of course, customize and adjust this ratio.

[0223] The Allocation module automatically calculates the optimal residual bit width for each segment based on the variance of each partition and the theoretical formulas introduced earlier. It also outputs a graph showing the correspondence between the compression ratio and the residual bits and writes the results to the configuration profile library.

[0224] Benchmark module: The system actually builds an ∈-PLA learning compressor for each partition, reports the compression ratio, compression time, and decompression time, and allows users to select LZ4 / LZMA / Huffman as a baseline for comparison.

[0225] 3. Representative Results

[0226] Visual consistency: On SIFT1B, AUDIO, Deep1B, and other datasets, the system estimates... The result is close to the measured optimal bit width, verifying the effectiveness of the cost model and the closed-form solution.

[0227] Compression performance: In vector database scenarios, the SIFT1B codebook size decreased from 3.8 GiB to 0.88 GiB, significantly reducing storage. In inverted list scenarios (CCNews, Clueweb), the system also provides partition adaptive configuration and baseline comparison, allowing observation of the compression advantages under different distributions.

[0228] In summary, by using a provable bit-level cost model and closed-form optimal solutions, this invention transforms the hyperparameter configuration of learning compression from empirical tuning into a computable and reproducible automated process; at the same time, BOCPD partitioning + partition-level bit width allocation effectively addresses the distribution drift of real data.

[0229] The system implements an end-to-end toolchain using the Remi framework, integrating data exploration, partitioning, residual bit allocation, and benchmark comparison. Users can complete "visible parameter tuning, verification, and comparison" in a browser, significantly reducing the barrier to entry.

[0230] In real-world tasks such as vector databases and inverted indexes, this invention demonstrates significant compression ratio advantages with minimal efficiency loss; taking the SIFT1B codebook as an example, the original 3.8 GiB is reduced to 0.88 GiB.

[0231] Therefore, this invention not only has theoretical optimization guarantee, but also proves its engineering practicality and universal value in large-scale ordered integer key scenarios such as vector database codebook compression and inverted index compression through end-to-end implementation and experimental verification.

[0232] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for adaptive configuration of ordered key learning compression parameters, characterized in that: Includes the following steps: S100: Select a public dataset D, which contains integer keys; S200: Extract all integer keys from dataset D and sort all extracted integer keys in ascending order to obtain the original ordered key sequence set K = {k1,…,k...} i ,…,k N } and calculate the gap value between two adjacent integer keys. The gap value between the i-th integer key and the previous integer key is taken as the i-th gap value, denoted as gap. i The calculation formula is as follows: gap i =k i -k i-1 Where, at this time, i takes the value [2,...,N], and k N Let k1 represent the Nth integer key, and k1≤k2≤…≤k N N represents the total length of the data; All gap values ​​in K obtained by the above method constitute a gap sequence set; S300: Preset probability threshold θ, use the Bayesian online change point detection algorithm BOCPD to calculate the posterior probability of the gap value changing at time step t for all gap values ​​in the gap sequence set; The posterior probability of all calculated gap values ​​is compared with a preset probability threshold, and points that are greater than or equal to the preset probability threshold are designated as change points. Once all the variable points are obtained, all variable point positions are mapped back to K, and K is divided into J partitions based on the variable point positions. S400: Select the j-th partition from J partitions. j And set an initial upper limit threshold ε for error. j Then targeting P j Construct a piecewise linear model ε-PLA, and use P j The j-th partition P is obtained by fitting the ordered key sequence into the ε-PLA. j piecewise linear function f j (i j ), f j (i j The expression for ) is as follows: Among them, i j P represents j The original ordered key index is contained within, and each partition corresponds to a piecewise function, α. l,j β represents the slope of the piecewise function in the l-th segment within the j-th partition. l,j s represents the intercept of the piecewise function in the l-th segment within the j-th partition. l Let f(·) represent the ordered key of the starting point of the l-th piecewise function, and L represent the total number of piecewise functions; the function f(·) needs to satisfy the constraint condition of the upper limit of error: S500: Building about f j (i j The bit-space total cost function B) j (ε j ), the B j (ε j This includes the piecewise linear model overhead and the residual bit width overhead, calculated as follows: in, Let C represent the variance of the gap sequence set, and K represent the fitting constant. bit F represents the number of bits required to encode the integer key. bit The number of bits required to represent the slope and intercept; N j P represents j The total number of ordered keys on the array; S600: P j When the gap sequence satisfies iid, make B j (ε j Minimum f j (i j The optimal upper limit of error ε j * , ε j * The calculation formula is as follows: The iid represents independent and identically distributed; According to ε j * The residual bit width of partition j is calculated using the following formula: Among them, b j Indicates the residual bit width within the j-th partition; S700: According to ε j * and b j Construct P j The partition bit width configuration; S800: Traverse all J partitions, repeat S400-S700, and obtain the partition bit width configuration for each partition. All partition bit width configurations constitute the compression parameter adaptive configuration table list of K. S900: The piecewise functions corresponding to each partition in J are aggregated to form a learning compressor Z with respect to K. The calculation expression is as follows: Z={f1,…,f j ,…,f J } Among them, f j Represents the piecewise function for the j-th partition; Input K into Z, and based on the compression parameter adaptive configuration table list, output the compression result K' of K.

2. The ordered key learning compression parameter adaptive configuration method as described in claim 1, characterized in that: The S500 constructs about f j (i j The bit-space total cost function B) j (ε j The steps are as follows: S510: According to the expected number of segments theory, ε is satisfied. j Sufficiently large, that is, ε j >>σ j / μ j When, calculate f j (i j The expected number of keys to be covered The calculation formula is as follows: Where, μ j This represents the mean of all gaps. ε represents the variance of the gap sequence set. j f j (i j The maximum upper limit of error; S520: Fitting length is N j Given an ordered integer key sequence, the desired number of segments is calculated as follows: Where O represents the calibration constant, N j P represents j The total number of ordered keys on the array; S530: Based on the Expected Segment Count theory, we obtain information about f. j (i j The number of segments in the segmented data is L(ε). j The calculation formula is as follows: Where C represents the calibration constant; S540: Utilizing L(ε) j Calculate f j (i j The piecewise linear model cost B structure (ε j The calculation formula is as follows: Among them, K bit F represents the number of bits occupied by the starting position of a piecewise function. bit K represents the slope intercept of the corresponding piecewise function as the number of bits occupied by the floating-point number. bit +2F bit This represents the total number of bits used to account for the cost of each piecewise function segment; The expected number of pieces in a piecewise function is calculated using the following formula: S550: Calculate residual bit width overhead B residual (ε j The calculation formula is as follows: S560:B structure (ε j ) and B residual (ε j ) constitutes B(ε j The calculation formula is as follows: B j (e j )=B residual (e j )+B structure (e j )。 3. The ordered key learning compression parameter adaptive configuration method as described in claim 2, characterized in that: The optimal upper limit of error ε is calculated in S600. j * The steps are as follows: log2(2ε) j +1) is approximately a continuous function, i.e., log2(2ε) j +1)≈log2(2ε j ), at this point, reconstruct B j (ε j The function expression is as follows: For the reconstructed B j (ε j ) for ε j By taking the derivative and setting it to zero, we can obtain the optimal upper limit of error ε. j * The process is as follows:

4. The ordered key learning compression parameter adaptive configuration method as described in claim 3, characterized in that: The steps for calculating the change point position using BOCPD in S300 are as follows: The BOCPD algorithm is used to recursively update the gap sequence set point by point. At time step t, the prior transition probability of the run-length is determined based on the hazard function, and combined with the observation likelihood model. Calculate the posterior distribution p(r) of the run-length. t =0|gap 1:t );r t This represents the length of the sample that has been continuously observed since the most recent change point at time step t; When the probability is greater than or equal to the preset probability threshold θ, time step t is marked as a change point; and the detected change point position is mapped to the corresponding position in the original key sequence K. K is divided into J partitions according to the corresponding position, so that the gap distribution within each partition remains relatively consistent.

Images