Self-adaptive coding compression method and system for column database

Abstract

The invention relates to an adaptive coding compression method and system for a column database, belongs to the technical field of data compression, and solves the problems of low compression rate and low query performance caused by curing of a compression algorithm and strong manual dependence in the prior art. Comprising the following steps: partitioning collected original data, and extracting a multi-dimensional feature vector from each dynamic column in each data block; inputting the multi-dimensional feature vector of each dynamic column into a trained prediction model to obtain a prediction result, the prediction result comprising a compression algorithm selection probability, a compression ratio prediction value and a decompression overhead level; based on the prediction result of each dynamic column, selecting an optimal compression algorithm for the dynamic columns through a multi-objective decision-making method; and further executing a compression operation. And the compression speed and precision of mass data are improved.

Description

Technical Field

[0001] This invention relates to the field of data compression technology, and in particular to an adaptive encoding compression method and system for columnar databases. Background Technology

[0002] Columnar databases store and organize data by columns rather than rows, which can significantly reduce I / O overhead and improve query efficiency in scenarios such as data analysis and batch queries. Therefore, they are widely used in data warehouses, real-time analysis and other systems. However, massive data still faces the core problems of "high storage costs and performance impact from compression / decompression latency".

[0003] In existing technologies, columnar database encoding compression schemes mainly adopt the following two types of approaches: one is a static fixed strategy, which specifies a uniform encoding method for specific data types or the entire table (for example, Parquet uses SNAPPY universal compression for all columns by default, and ClickHouse uses Delta interpolation encoding for numeric columns); the other is to manually pre-define the encoding method based on the analysis of data characteristics by the operation and maintenance personnel.

[0004] However, existing technical solutions have the following drawbacks, making it difficult to achieve the optimal balance between storage efficiency and query performance in complex and ever-changing data scenarios: The inability to precisely adapt to the data characteristics of different columns (such as the high cardinality and discrete characteristics of the "User ID" column, and the continuous time-series characteristics of the "Electricity Voltage" column) leads to low compression rates or excessive decompression computation overhead. Faced with massive tables and thousands of columns, relying on operations personnel to manually analyze data characteristics and select encoding strategies is not only inefficient but also highly susceptible to inappropriate strategy selection due to lack of experience. Furthermore, when data characteristics change with business development or over time, static strategies cannot detect and automatically adjust, causing previously effective compression strategies to significantly degrade in compression rates or query performance in later stages. Summary of the Invention

[0005] Based on the above analysis, the embodiments of the present invention aim to provide an adaptive encoding compression method and system for columnar databases, in order to solve the problems of low compression ratio and query performance caused by the rigidity of compression algorithms and strong reliance on manual intervention in existing systems.

[0006] On one hand, embodiments of the present invention provide an adaptive encoding compression method for columnar databases, comprising the following steps: The collected raw data is divided into blocks, and a multidimensional feature vector is extracted from each dynamic column in each data block. The multidimensional feature vector of each dynamic column is input into the trained prediction model to obtain the prediction results, which include: compression algorithm selection probability, compression ratio prediction value and decompression overhead level. Based on the prediction results of each dynamic column, the optimal compression algorithm is selected for the dynamic column using a multi-objective decision-making method; then the compression operation is performed.

[0007] Based on further improvements to the above method, the collected raw data is divided into blocks, including: The original data is divided into blocks based on a preset time window, entity identifier, and quantity threshold, forming a data block containing a single entity and a sequence number within a single time window. The sequence number is generated by dividing the data blocks sequentially according to the quantity threshold. Generate a globally unique block identifier for each data block.

[0008] Based on further improvements to the above method, the data block includes: the attributes and values ​​of static columns, and the attributes and value sequences of dynamic columns; the extracted multidimensional feature vectors include: statistical features and semantic features.

[0009] Based on further improvements to the above methods, statistical features are obtained by calculating the proportion of unique values, monotonicity score, difference variance, and average run length from the value sequence of dynamic columns; semantic features are obtained by converting the values ​​of multiple static columns into normalized scalars through a predefined mapping table.

[0010] Based on further improvements to the above method, the prediction model sequentially includes: a feature extraction module, a gated routing module, a multi-branch prediction module, and a weighted fusion module. The feature extraction module maps the multi-dimensional feature vector of each dynamic column into a high-dimensional feature vector, which is then passed to the gated routing module and the multi-branch prediction module. The gated routing module calculates the routing weight of each branch in the multi-branch prediction module based on the high-dimensional feature vector and passes it to the weighted fusion module. In the multi-branch prediction module, each branch performs multi-task prediction in parallel based on the same high-dimensional feature vector, generates its own initial prediction result, and then passes it to the weighted fusion module. The weighted fusion module performs weighted fusion of the initial prediction results of each branch in the multi-branch prediction module based on the routing weight of each branch, and generates the final prediction result.

[0011] Based on the above method, the prediction model adopts a multi-task learning framework and is obtained by training by minimizing the weighted multi-task loss function. The loss function includes: compression algorithm classification loss, compression ratio prediction loss, decompression overhead classification loss, and route distribution regularization loss.

[0012] Based on the above method, a further improvement is made to select the optimal compression algorithm for the dynamic column using a multi-objective decision-making method, including: Based on the compression algorithm selection probability output by the prediction model, multiple candidate algorithms with the highest probabilities are selected. Candidate algorithms are filtered according to preset business constraints; If there are no candidate algorithms after filtering, the default compression algorithm is selected as the optimal compression algorithm; if there is only one candidate algorithm after filtering, it is directly selected as the optimal compression algorithm; otherwise, the comprehensive score is calculated based on the compression ratio prediction value and decompression overhead level of the candidate algorithms after filtering, and the candidate algorithm with the highest comprehensive score is selected as the optimal compression algorithm for this dynamic column.

[0013] Based on further improvements to the above method, the comprehensive score is calculated using the following formula, according to the predicted compression ratio and decompression overhead level of the filtered candidate algorithms: , in, Indicates the filtered first 10 candidate algorithms This represents the predicted compression ratio. This indicates the estimated time taken for the decompression overhead level mapping. and These represent the historical maximum compression ratio and decompression time, respectively, used for normalization; and These represent the compression efficiency weight and query performance weight, respectively, both of which are greater than 0.

[0014] Based on further improvements to the above method, a compression operation is performed, including: Based on the optimal compression algorithm for each dynamic column, generate corresponding table creation statements for each data block, where the compression algorithm attribute used is explicitly specified for each dynamic column, and the storage mode of the table is explicitly specified as columnar. A columnar storage table is created based on the table creation statement. Then, according to the optimal compression algorithm for each dynamic column, the corresponding encoding compression library is called to compress the value sequence of each dynamic column before importing it into the columnar storage table to complete the storage.

[0015] On the other hand, embodiments of the present invention provide a columnar database adaptive encoding compression system, comprising: The data feature extraction module is used to divide the collected raw data into blocks and extract multi-dimensional feature vectors for each dynamic column in each data block. The compression algorithm prediction module is used to input the multi-dimensional feature vector of each dynamic column into the trained prediction model to obtain the prediction results, which include: compression algorithm selection probability, compression ratio prediction value, and decompression overhead level. The data compression and storage module is used to select the optimal compression algorithm for each dynamic column based on the prediction results of each dynamic column through a multi-objective decision-making method, and then perform the compression operation.

[0016] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects: 1. Adaptive block partitioning ensures the homogeneity of data unit features, laying the foundation for accurate prediction; through automated multi-dimensional feature extraction and prediction models, it realizes automatic analysis of the inherent patterns and business context of each dynamic column, and quantitatively predicts the most suitable compression algorithm, eliminating the strong dependence on human experience and improving the compression speed and accuracy of processing massive amounts of data.

[0017] 2. The constructed prediction model adopts a gated routing mechanism and a multi-branch prediction structure, which overcomes the limitation that a single model cannot master all data patterns. Moreover, for the feature vector of each input column, a set of exclusive routing weights is dynamically generated to adaptively integrate the judgments of different branches, realizing column-level adaptive capability. This makes the model exhibit far higher accuracy and robustness than static rules or ordinary models when facing data with different forms in different scenarios.

[0018] 3. For each data column, three closely related quantitative prediction values ​​are output in parallel. The quantitative decision-making process incorporates multi-objective optimization and business constraints, so that the final selected compression algorithm is not only optimal in terms of data characteristics, but also most suitable for the business scenario.

[0019] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0020] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. Figure 1 This is a flowchart of an adaptive encoding compression method for columnar databases according to Embodiment 1 of the present invention. Detailed Implementation

[0021] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0022] Example 1 A specific embodiment of the present invention discloses an adaptive encoding compression method for columnar databases, such as... Figure 1 As shown, it includes steps S1-S3.

[0023] S1. Divide the collected raw data into blocks, and extract multidimensional feature vectors for each dynamic column in each data block.

[0024] This step involves structurally dividing the original data into blocks and extracting multidimensional vectors that can comprehensively and accurately characterize its compressibility and performance impact.

[0025] It should be noted that columnar databases typically store data column-wise, but during the data ingestion process, data is still physically stored in blocks organized by row or batch. Traditional block partitioning strategies are usually based on a fixed number of rows or a fixed size (e.g., the default 128MB row group in Parquet), ignoring the inherent business logic and access patterns of the data, resulting in mixed data characteristics within blocks, which is not conducive to the accurate adaptation of encoding strategies.

[0026] This embodiment divides the collected raw data into blocks, including: The original data is divided into blocks based on a preset time window, entity identifier, and quantity threshold, forming a data block containing a single entity and a sequence number within a single time window. The sequence number is generated by dividing the data blocks sequentially according to the quantity threshold. Generate a globally unique block identifier for each data block.

[0027] It should be noted that by combining time windows and entity identifiers, data is organized into data blocks that are business-coherent and feature-converging. Furthermore, to ensure the size of each physical storage block is controllable and facilitates parallel processing and memory loading, new sub-blocks are generated by incrementing the sequence number for data blocks exceeding a predetermined threshold, based on the initial sequence number. Ultimately, the globally unique block identifier generated for each data block is uniquely determined by the time window, entity identifier, and sequence number.

[0028] It should be noted that the quantity threshold is a preset maximum number of rows. The basis for setting it is to ensure that the storage size of a single data block does not exceed the preset maximum storage limit (8 MB) under typical data width, thereby avoiding excessive pressure on memory and I / O.

[0029] For example, the block identifier (block_id) is: blk_{entity_key}_{YYYYMMDD_HH}_{NNN}. Where: blk indicates a prefix; {entity_key} represents the entity identifier, such as meter10001, which is a device ID in the device table; {YYYYMMDD_HH} represents year, month, day, and hour, corresponding to the start time of the time window; for example, the length of the time window is 1 hour, and 20251218_14 represents 14:00 to 15:00 on December 18, 2025; {NNN} represents a three-digit sub-block number starting from 000, used to handle situations where the data volume is too large under a single time window and device combination, ensuring that the size of each physical block is controllable (e.g., not exceeding 1GB), facilitating parallel processing and memory loading.

[0030] For example, blk_merter10001_20251218_14_000 represents the first data block of device 10001 at 14:00 on December 18, 2025; if the data written during this period exceeds the quantity threshold, such as the maximum number of rows 50,000, then subsequent data will be allocated a new block identified as blk_merter10001_20251218_14_001.

[0031] It should be noted that each data block corresponding to block_id contains all relevant data of the corresponding entity within that time window, including: the attributes and values ​​of static columns, and the attributes and value sequences of dynamic columns.

[0032] Static columns are attribute columns that do not change or change very infrequently within a time window, such as device ID, device model, installation region, and department. These columns typically have the same value across all rows within the block, or only a few discrete values.

[0033] Dynamic columns refer to data columns that are continuously generated over time through sampling or business events. They are mainly time-series indicators or status records, such as collection timestamps, voltage values, current values, power consumption, temperature, and equipment status codes.

[0034] To achieve efficient, reliable, and resource-controlled asynchronous feature extraction and prediction, this embodiment stores the information of each data block obtained from the original data segmentation according to a static / dynamic separation structure. Each data block corresponds to a directory in the storage system based on its block identifier, including a static metadata file and a dynamic column data file. That is, the static metadata file and the dynamic data file are associated through a shared block identifier.

[0035] Specifically, the static metadata file stores the static information of the data block in a lightweight JSON format, including: block identifier, entity identifier, time window range, total number of rows in the data block, and static column attributes and their data; the dynamic column data file stores the data of all dynamic columns in the data block in the Apache Parquet columnar storage format, including: dynamic column attribute name, dynamic column type, and dynamic column data sequence.

[0036] Furthermore, from both statistical regularity and semantic context perspectives, multidimensional feature vectors are extracted from each dynamic column in each data block, including statistical features and semantic features.

[0037] The statistical features are obtained by calculating the proportion of unique values, monotonicity score, difference variance, and average run length from the value sequence of the dynamic columns; the semantic features are obtained by converting the values ​​of multiple static columns into normalized scalars through a predefined mapping table. In other words, all dynamic columns in the same data block have the same semantic features.

[0038] ① The uniqueness ratio is obtained by calculating the proportion of different values ​​in the value sequence of a dynamic column to the total number of rows, and is used to reflect the degree of data duplication.

[0039] If the uniqueness ratio is close to 1, it means that the values ​​in this column in the data block are basically unique, with extremely low repetition, and is suitable for general compression (LZ4) or difference encoding (Delta, FOR) methods; if the uniqueness ratio is close to 0, it means that there are a large number of duplicate values ​​in this column in the data block, with extremely high repetition, and is suitable for data dictionary encoding (Dictionary) or run-length encoding (RLE).

[0040] For example, a list of device status codes [normal, normal, fault, normal, fault] has a uniqueness ratio of 2 / 5 = 0.4, indicating that there is some duplication.

[0041] ② The monotonicity score is calculated by taking the proportion of adjacent element pairs in the value sequence of a dynamic column that satisfy the non-decreasing condition. It is used to identify increasing or approximately increasing sequences, such as timestamps and auto-incrementing IDs.

[0042] The closer the monotonicity score is to 1, the better the data is suited for difference coding (Delta), because the differences between continuous values ​​are small and stable.

[0043] For example, if adjacent data in a timestamp value sequence all increase, then the monotonicity score is 1.

[0044] ③ Difference variance is obtained by first calculating the difference between adjacent elements in the value sequence of the dynamic column, and then calculating the variance of all differences. It is used to measure the stability of local changes in the sequence.

[0045] The smaller the variance, the smaller the fluctuation of the difference, which is suitable for Delta or FOR encoding; the larger the variance, the more drastic the numerical jump, which is more suitable for general compression algorithms.

[0046] ④ Average run length is calculated by dividing the value sequence of a dynamic column into several "runs" based on consecutive identical values, and calculating the average number of elements contained in each run. This is used to evaluate the potential efficiency of run-length encoding (RLE).

[0047] The larger the average run length, the more times the same value appears consecutively, and the higher the RLE compression ratio.

[0048] For example, the value sequence [A, A, A, B, B, C, C, C, C] has 3 runs, a total length of 9, and an average run length of (3+3+3) / 3 = 3.0.

[0049] The semantic features extracted in this embodiment include entity type encoding and region encoding, both based on corresponding static columns. Their values ​​are converted into a normalized scalar through a predefined mapping table to introduce business context. This approach is suitable for scenarios where the dynamic data models of entities of different types or regions may exhibit systematic differences.

[0050] It is understandable that the extraction of semantic features is not limited to entity type encoding and region encoding; the attributes of the corresponding static columns can be predetermined according to the business scenario.

[0051] It should be noted that the extracted multidimensional data features are normalized or standardized to map the values ​​to the [0,1] interval, thus eliminating the influence of dimensions and accelerating model training. Ultimately, each dynamic column in the data block corresponds to a 6-dimensional feature vector.

[0052] The feature extraction in this step is column-level and block-level independent, which naturally supports column-level parallel processing in massive data scenarios, greatly improving efficiency.

[0053] S2. Input the multidimensional feature vector of each dynamic column into the trained prediction model to obtain the prediction results, which include: compression algorithm selection probability, compression ratio prediction value, and decompression overhead level.

[0054] The prediction model in this embodiment is a multi-task neural network model that introduces column-level adaptive routing. Its core is to treat the feature vector of each dynamic column as an independent prediction sample, and to dynamically and independently combine the outputs of multiple branches for each sample to obtain the prediction result, thereby achieving a highly refined "one strategy per column".

[0055] It should be noted that the prediction model includes, in sequence: a feature extraction module, a gated routing module, a multi-branch prediction module, and a weighted fusion module.

[0056] Specifically, the feature extraction module is a lightweight feedforward neural network (such as two fully connected layers) used to map the multidimensional feature vector of each dynamic column into a higher-dimensional, more discriminative shared feature space. After obtaining the high-dimensional feature vector of each dynamic column, it is fed into the gated routing module and the multi-branch prediction module.

[0057] The gated routing module includes a linear layer and a softmax activation function. It calculates a set of routing weights based on the high-dimensional feature vectors of dynamic columns, corresponding to each branch in the subsequent multi-branch prediction module. This weights characterize the contribution of each branch to making the correct decision under the specific data characteristics of the current column. For example, even within the same data block, a timestamp column with high monotonicity and a state column with high repetition will receive completely different routing weight assignments, thus forming different combinations of branch results.

[0058] In the multi-branch prediction module, each branch performs parallel multi-task predictions based on a high-dimensional feature vector from the same dynamic column of the feature extraction module. Each branch is a neural network with identical structure but no parameter sharing, consisting of a multilayer perceptron and a multi-task output head. The multilayer perceptron is used to further extract deep features from the high-dimensional feature vector that are highly correlated with the patterns that the branch excels at (such as "temporally continuous patterns" or "discrete repetitive patterns"), and then passes these features to the parallel multi-task output head. The multi-task output head generates a complete set of initial prediction results, which are then passed to the weighted fusion module.

[0059] Specifically, the multi-task output header includes: a compression algorithm selection header, a compression ratio prediction header, and a decompression overhead level header.

[0060] The compression algorithm selection head is a multi-classifier, consisting of a single fully connected layer followed by a Softmax activation function. Its output is a K-dimensional probability vector, representing the probability that the branch believes the current column should use each predefined compression algorithm. K represents the number of compression algorithms; in this embodiment, K=5, corresponding to the following compression algorithms: Dictionary (data dictionary encoding), RLE (Run-Length Encoding), Delta (Differential Encoding), FOR (Frame Of Reference), and LZ4.

[0061] The compression ratio prediction head is a linear regressor, a single fully connected layer, whose output is a positive scalar value, which is the natural logarithm of the compression ratio prediction. After performing an exponential operation on it, the compression ratio prediction value (i.e., the ratio of the original size to the compressed size) is obtained.

[0062] The decompression overhead level head is also a multi-classifier, with a structure similar to the compression algorithm selection head, but the output is a 3-dimensional probability vector, corresponding to the probabilities of the three levels: "low", "medium" and "high".

[0063] The weighted fusion module performs weighted fusion of the initial prediction results of each branch in the multi-branch prediction module according to their routing weights, and generates the final prediction result, including: compression algorithm selection probability, compression ratio prediction value, and decompression overhead level.

[0064] Furthermore, following the method in step S1, the multidimensional feature vector of each dynamic column in the historical data is obtained as an independent training sample, and the true label is obtained and added to the sample set. The true label includes: the true compression algorithm (one-hot encoding), the natural logarithm of the true compression ratio (numerical value), and the true decompression overhead level (one-hot encoding).

[0065] It should be noted that the actual decompression overhead level label is obtained by dividing the actual decompression time of the historical dynamic column into three categories: "low", "medium" and "high" using a clustering algorithm.

[0066] The prediction model in this embodiment is based on a sample set and adopts a multi-task learning framework. It is trained by minimizing the weighted multi-task loss function.

[0067] loss function Includes: Classification loss in compression algorithms Compression ratio prediction loss Decompression overhead classification loss and routing distribution regularization loss The calculation formula is as follows: , in, , , and These represent the weights of the compression algorithm classification loss, compression ratio prediction loss, decompression overhead classification loss, and routing distribution regularization loss, respectively.

[0068] Specifically, the classification loss of the compression algorithm uses the cross-entropy loss function, as shown in the following formula: , in, Indicates batch size, Indicates the first The true compression algorithm for each sample Indicates the first The predicted number of samples The probability of a compression algorithm, This indicates the number of categories of compression algorithms. This represents the logarithmic function.

[0069] The decompression overhead classification loss also uses the cross-entropy loss function, as shown in the following formula: , in, Indicates the first The actual decompression overhead level of each sample. Indicates the first The predicted number of samples The probability of each decompression overhead level. This indicates the number of decompression overhead levels. .

[0070] The compression ratio prediction loss uses the mean squared error function to supervise the model's prediction accuracy for the compression ratio. To match the long-tailed distribution characteristics of compression ratio data (i.e., most values ​​are concentrated in a small range, with a few extremely large values), and to make the optimization process focus more on relative error rather than absolute error, this embodiment calculates the loss in logarithmic space, as shown in the following formula: , in, Indicates the first Compression ratio of each sample The natural logarithm prediction, Indicates the first The true compression ratio of each sample The natural logarithm of .

[0071] The route distribution regularization loss is used to ensure that each branch in the multi-branch prediction module receives a minimum amount of training data, preventing undertraining of any branch due to excessive concentration of route weights; the formula is shown below: , in, This indicates the number of branches in the multi-branch prediction module of the prediction model, in this embodiment. ; This represents the minimum usage threshold, such as 0.05. Indicates the first The branch pairs with the first The routing weight of each sample.

[0072] As can be seen from the formula, this embodiment does not force the utilization rate of each branch in the prediction model to be close. Instead, it adaptively determines the routing weight based on data characteristics, and only when the batch average utilization rate of a certain branch is lower than the minimum utilization rate threshold... The penalty is applied only when necessary to prompt the gated routing module to adjust the distribution of routing weights; when the utilization rate of all branches is higher than the minimum utilization rate threshold, this item is zero and does not affect training.

[0073] It should be noted that introducing route distribution regularization loss into the loss function not only prevents a certain branch used for prediction from failing due to insufficient training, but also maintains the adaptability of the prediction model to different data patterns, enhances the robustness of the prediction model, and prevents overfitting to specific data patterns.

[0074] During implementation, the multidimensional feature vector of each dynamic column to be predicted is input into the trained prediction model to obtain a set of prediction results: compression algorithm selection probability, compression ratio prediction value, and decompression overhead level.

[0075] S3. Based on the prediction results of each dynamic column, the optimal compression algorithm is selected for the dynamic column through a multi-objective decision-making method; then the compression operation is performed.

[0076] This step uses a multi-objective decision-making method to weigh the prediction results of each dynamic column obtained in step S2 among multiple objectives (high compression ratio, low latency), and transforms them into a deterministic optimal compression strategy that meets business constraints.

[0077] Specifically, it includes the following steps: ① Based on the compression algorithm selection probability output by the prediction model, select the multiple candidate algorithms with the highest probability.

[0078] After sorting the compression algorithms by probability in descending order, the top S compression algorithms with the highest probabilities are selected as candidate algorithms to balance prediction confidence and optimization space. For example, S=3.

[0079] ②Filter candidate algorithms according to preset business constraints.

[0080] It should be noted that the preset business constraints are based on a preset decompression overhead level for filtering. For example, if the preset business constraint is "low CPU" mode, then candidate algorithms that predict a "high" decompression overhead level will be eliminated.

[0081] ③ If there are no candidate algorithms after filtering, the default compression algorithm is selected as the optimal compression algorithm; if there is only one candidate algorithm after filtering, it is directly selected as the optimal compression algorithm; otherwise, the comprehensive score is calculated based on the compression ratio prediction value and decompression overhead level of the candidate algorithms after filtering, and the candidate algorithm with the highest comprehensive score is selected as the optimal compression algorithm for this dynamic column.

[0082] It should be noted that the comprehensive score is calculated based on the predicted compression ratio and decompression overhead level of the filtered candidate algorithms using the following formula: , in, Indicates the filtered first 10 candidate algorithms This represents the predicted compression ratio. This indicates the estimated time taken for the decompression overhead level mapping. and These represent the historical maximum compression ratio and decompression time, respectively, used for normalization; and These represent the compression efficiency weight and query performance weight, respectively, both of which are greater than 0.

[0083] It should be noted that the estimated time is obtained based on the preset mapping relationship between decompression overhead level and decompression time. For example, the estimated time is 1ms for a "low" level, 5ms for a "medium" level, and 20ms for a "high" level. (Weight) and These weights are configured based on business needs, representing the degree of importance placed on storage savings and the sensitivity to decompression speed, respectively. For example, in an offline analytics scenario, where storage efficiency is prioritized, the weights are set as follows: In real-time query scenarios, prioritizing query performance is crucial, so the following settings should be configured: .

[0084] Compared with existing technologies, this step does not simply select the compression algorithm with the highest probability, but incorporates a quantitative decision-making process that integrates multi-objective optimization and business constraints, so that the final selected compression algorithm is not only optimal in terms of data characteristics, but also most suitable for the business scenario.

[0085] Furthermore, based on the prediction results, physical tables with specific column-level compression attributes are dynamically created, and the value sequences of dynamic columns in the data blocks are compressed using the optimal compression algorithm before being imported. The specific process is as follows: Based on the optimal compression algorithm for each dynamic column, generate corresponding table creation statements (DDL) for each data block, where the compression algorithm attribute used is explicitly specified for each dynamic column, and the storage mode of the table is explicitly specified as columnar. A columnar storage table is created based on DDL statements. Then, according to the optimal compression algorithm for each dynamic column, the corresponding encoding compression library is called to compress the value sequence of each dynamic column before importing it into the columnar storage table to complete the storage.

[0086] Specifically, based on the unique block identifier `block_id` of the current data block and the optimal compression algorithm for all its dynamic columns, a DDL statement for creating a columnar storage table is dynamically generated. The core feature of this DDL statement is that, when defining each dynamic column, its compression algorithm is explicitly specified through an extended syntax; for example, `voltage FLOATCOMPRESSION DELTA` explicitly specifies the Delta compression algorithm for voltage; and `WITH(orientation = COLUMN)` explicitly specifies the table storage mode as columnar. This transforms the abstract compression algorithm into physical storage instructions that the database kernel can recognize and execute.

[0087] After creating a columnar storage table using the table creation statement, perform data compression and data insertion operations, including: The database extracts the value sequence of each dynamic column from the data block. For each column, the corresponding encoding and compression library is called to compress the value sequence. The compressed column data is then efficiently populated into the created columnar storage table through the database's batch import interface. The database storage engine organizes the compressed data into internal columnar storage data blocks (CUs) based on the COMPRESSION attribute in the table creation statement.

[0088] It should be noted that in step S1 of this embodiment, static metadata files and dynamic data files have been stored by data blocks. The value sequence of each dynamic column can be obtained by reading the dynamic data file in the data block.

[0089] Upon successful data entry, the header of each columnar data block (CU) stored within the database records the compression algorithm used, the original data size, and the compressed size.

[0090] Compared with existing technologies, this embodiment uses dynamic DDL generation to seamlessly integrate the prediction results into the physical storage layer of the database, thus realizing the implementation of adaptive encoding compression.

[0091] Preferably, in order to learn from the execution results and continuously improve, the actual compression effect is also collected during the actual application process, compared with the prediction results, and the difference is used to optimize the prediction model, so that its prediction becomes more and more accurate and its decision-making becomes better and better.

[0092] Specifically, a compression feedback table is constructed, with each record including key fields such as: block identifier, column name, prediction algorithm, actual algorithm, predicted compression ratio, actual compression ratio, actual decompression time / level, and timestamp. During compression operations and decompression queries, relevant information is recorded in the compression feedback table for subsequent model iterations and strategy optimization.

[0093] The method in this embodiment is also used to periodically analyze the compressed feedback table, calculate the prediction error, and filter out high-error samples (e.g., actual errors exceeding the error threshold) or prediction failure samples (e.g., the prediction optimal algorithm performs poorly in practice). These samples are placed in a priority replay buffer pool for model updates.

[0094] Once the buffer pool has accumulated a sufficient number of samples, the prediction model is incrementally fine-tuned by performing a small number of training iterations with new samples, while maintaining the original model structure. The learning rate is set low. The entire prediction model is then retrained periodically (weekly or monthly) using all historical high-quality feedback samples and the original training data.

[0095] In addition, each new prediction model obtained from training is saved by version so that if a problem occurs, it can be quickly rolled back to the previous stable version with one click, minimizing the impact on business.

[0096] Compared with existing technologies, this embodiment provides a columnar database adaptive encoding compression method that ensures the homogeneity of data unit features through adaptive block partitioning, laying the foundation for accurate prediction. Through automated multi-dimensional feature extraction and prediction models, it automatically analyzes the inherent patterns and business context of each dynamic column and quantitatively predicts the most suitable compression algorithm, eliminating strong reliance on human experience and improving the speed and accuracy of decision-making in processing massive datasets. The constructed prediction model employs a gated routing mechanism and a multi-branch prediction structure, overcoming the limitation of a single model being unable to master all data patterns. Furthermore, for the feature vector of each input column, a set of dedicated routing weights is dynamically generated to adaptively integrate the judgments of different branches, achieving column-level adaptive capability. This allows the model to exhibit significantly higher accuracy and robustness than static rules or ordinary models when facing data with diverse forms in different scenarios. Three closely related quantitative prediction values ​​are output in parallel for each data column, incorporating a multi-objective optimization and business constraint-based quantitative decision-making process. This ensures that the final selected compression algorithm is not only optimal in terms of data features but also most suitable for the business scenario.

[0097] Example 2 Another embodiment of the present invention discloses a columnar database adaptive encoding compression system, thereby implementing the columnar database adaptive encoding compression method in Embodiment 1. The specific implementation of each module is described in the corresponding description in Embodiment 1. The system includes: The data feature extraction module is used to divide the collected raw data into blocks and extract multi-dimensional feature vectors for each dynamic column in each data block. The compression algorithm prediction module is used to input the multi-dimensional feature vector of each dynamic column into the trained prediction model to obtain the prediction results, which include: compression algorithm selection probability, compression ratio prediction value, and decompression overhead level. The data compression and storage module is used to select the optimal compression algorithm for each dynamic column based on the prediction results of each dynamic column through a multi-objective decision-making method, and then perform the compression operation.

[0098] Since the columnar database adaptive encoding compression system described in this embodiment and the aforementioned columnar database adaptive encoding compression method are related and can be mutually referenced, this description is redundant and will not be repeated here. Because this system embodiment shares the same principle as the aforementioned method embodiment, it also possesses the corresponding technical effects of the aforementioned method embodiment.

[0099] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0100] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. An adaptive encoding compression method for columnar databases, characterized in that, Includes the following steps: The collected raw data is divided into blocks, and a multidimensional feature vector is extracted from each dynamic column in each data block. The multidimensional feature vector of each dynamic column is input into the trained prediction model to obtain the prediction result, which includes: compression algorithm selection probability, compression ratio prediction value and decompression overhead level. Based on the prediction results of each dynamic column, the optimal compression algorithm is selected for the dynamic column using a multi-objective decision-making method; and then the compression operation is performed.

2. The columnar database adaptive encoding compression method according to claim 1, characterized in that, The process of dividing the collected raw data into blocks includes: The original data is divided into blocks based on a preset time window, entity identifier, and quantity threshold, forming a data block containing a single entity and a sequence number within a single time window. The sequence number is generated by dividing the data blocks sequentially according to the quantity threshold. Generate a globally unique block identifier for each data block.

3. The columnar database adaptive encoding compression method according to claim 2, characterized in that, The data block includes: attributes and values ​​of static columns, and a sequence of attributes and values ​​of dynamic columns; the multidimensional feature vector includes: statistical features and semantic features.

4. The columnar database adaptive encoding compression method according to claim 3, characterized in that, The statistical features are obtained by calculating the proportion of unique values, monotonicity score, difference variance, and average run length from the value sequence of the dynamic column; the semantic features are obtained by converting the values ​​of multiple static columns into normalized scalars through a predefined mapping table.

5. The columnar database adaptive encoding compression method according to claim 1, characterized in that, The prediction model sequentially includes: a feature extraction module, a gated routing module, a multi-branch prediction module, and a weighted fusion module. The feature extraction module maps the multi-dimensional feature vector of each dynamic column into a high-dimensional feature vector, which is then passed to the gated routing module and the multi-branch prediction module. The gated routing module calculates the routing weight of each branch in the multi-branch prediction module based on the high-dimensional feature vector and passes it to the weighted fusion module. In the multi-branch prediction module, each branch performs multi-task prediction in parallel based on the same high-dimensional feature vector, generates its own initial prediction result, and then passes it to the weighted fusion module. The weighted fusion module performs weighted fusion of the initial prediction results of each branch in the multi-branch prediction module based on the routing weight of each branch to generate the final prediction result.

6. The columnar database adaptive encoding compression method according to claim 1 or 5, characterized in that, The prediction model adopts a multi-task learning framework and is obtained by training by minimizing a weighted multi-task loss function. The loss function includes: compression algorithm classification loss, compression ratio prediction loss, decompression overhead classification loss, and route distribution regularization loss.

7. The columnar database adaptive encoding compression method according to claim 1, characterized in that, The step of selecting the optimal compression algorithm for the dynamic column using a multi-objective decision-making method includes: Based on the compression algorithm selection probability output by the prediction model, multiple candidate algorithms with the highest probabilities are selected. The candidate algorithms are filtered according to preset business constraints; If there are no candidate algorithms after filtering, the default compression algorithm is selected as the optimal compression algorithm; if there is only one candidate algorithm after filtering, it is directly selected as the optimal compression algorithm; otherwise, the comprehensive score is calculated based on the compression ratio prediction value and decompression overhead level of the candidate algorithms after filtering, and the candidate algorithm with the highest comprehensive score is selected as the optimal compression algorithm for the dynamic column.

8. The columnar database adaptive encoding compression method according to claim 7, characterized in that, Based on the predicted compression ratio and decompression overhead levels of the filtered candidate algorithms, the overall score is calculated using the following formula: ， in, Indicates the filtered first 10 candidate algorithms This represents the predicted compression ratio. This indicates the estimated time taken for the decompression overhead level mapping. and These represent the historical maximum compression ratio and decompression time, respectively, used for normalization; and These represent the compression efficiency weight and query performance weight, respectively, both of which are greater than 0.

9. The columnar database adaptive encoding compression method according to claim 3, characterized in that, The compression operation includes: Based on the optimal compression algorithm for each dynamic column, generate corresponding table creation statements for each data block, where the compression algorithm attribute used is explicitly specified for each dynamic column, and the storage mode of the table is explicitly specified as columnar. Based on the table creation statement, a columnar storage table is created. Then, according to the optimal compression algorithm for each dynamic column, the corresponding encoding compression library is called to compress the value sequence of each dynamic column, and the result is imported into the columnar storage table to complete the storage.

10. A columnar database adaptive encoding compression system, characterized in that, include: The data feature extraction module is used to divide the collected raw data into blocks and extract multi-dimensional feature vectors for each dynamic column in each data block. The compression algorithm prediction module is used to input the multi-dimensional feature vector of each dynamic column into the trained prediction model to obtain the prediction result, which includes: compression algorithm selection probability, compression ratio prediction value and decompression overhead level; The data compression and storage module is used to select the optimal compression algorithm for each dynamic column based on the prediction results of each dynamic column using a multi-objective decision-making method, and then perform the compression operation.

Images