Data compression method for a database and related product

Abstract

The application provides a data compression method of a database and a related product. The method comprises the following steps: obtaining an original data page to be compressed, performing a difference operation on a plurality of data tuples in the original data page to obtain a difference data page, determining a data matrix corresponding to the difference data page, performing a transposition operation on the data matrix in the process of performing the difference operation, obtaining a data page to be processed from the transposed data matrix, and performing a compression operation on the data page to be processed according to a preset compression level. According to the scheme, the transposition operation is performed on the data matrix in the process of performing the difference operation, the difference data stored in a row order is converted into difference data stored in a column order, the bytes in the same column in the data matrix are arranged continuously in physical storage, the matching efficiency of the compression algorithm for the continuous repeated data is enhanced, and the compression rate is improved.

Description

Technical Field

[0001] This invention relates to database technology, and in particular to a data compression method and related products for databases. Background Technology

[0002] In database systems, data is organized and managed in fixed-size pages, which are compressed by a compression engine. The compression engine uses a general compression algorithm to compress the data pages, which contain a header, row pointer arrays, and tuple data. This general compression algorithm treats the data pages as an unstructured byte stream, failing to utilize the unique characteristics of the different components within the data page, thus limiting the compression ratio. Summary of the Invention

[0003] One objective of this invention is to improve the compression ratio of database compression engines.

[0004] A further objective of this invention is to expand the compression engine's adaptability to different business scenarios.

[0005] Another further objective of this invention is to maintain the processing efficiency of the compression process while increasing the compression ratio.

[0006] Specifically, the present invention provides a data compression method for a database, comprising: Obtain the original data page to be compressed, perform a differential operation on multiple data tuples in the original data page to obtain a differential data page, and store the differential data between adjacent data tuples in the differential data page; Determine the data matrix corresponding to the differential data page, and perform a transpose operation on the data matrix during the differential operation to obtain the data page to be processed from the transposed data matrix; The data pages to be processed are compressed according to the preset compression level.

[0007] Optionally, performing a difference operation on multiple data tuples in the original data page to obtain a differential data page includes: Using the last row of the original data page as the starting row of the differential scan, the scan proceeds back row by row in descending order of row number to determine the differential data between the current row data tuple and the previous row data tuple. The differential data is then stored in ascending order of row number to generate the differential data page. The first row data tuple of the original data page is used as the base data, and the base data and the differential data page together form the compressed data storage structure.

[0008] Optionally, determining the data matrix corresponding to the differential data page includes: Each differential data row in the differential data page is used as a row vector, and the bytes in each differential data row that are at the same byte offset position are used as column vectors to construct the data matrix of the differential data page.

[0009] Optionally, the bytes at the same byte offset position in each of the differential data rows are used as a column vector, including: The length of each differential data row is calculated based on the byte length after binary conversion. The minimum reference byte length among all the byte lengths is selected as the consistent column number of the data matrix; For target differential data rows whose byte length is greater than the base byte length, the original data of the target differential data rows is retained and not used for differential operations; The bytes at the same byte offset position in each of the differential data rows are extracted as column vectors of the data matrix, wherein the number of column vectors is equal to the length of the base byte, and each column vector is composed of the bytes at the same byte offset position in the differential data row arranged in row order.

[0010] Optionally, during the execution of the difference operation, performing a transpose operation on the data matrix includes: In response to all differential data rows in the differential data page being written into the data matrix, bytes at the same byte offset position of each differential data row in the data matrix are read sequentially along the column direction. The read bytes are sorted sequentially according to the row number in ascending direction to generate a byte sequence stored continuously in column order, so as to perform the transpose operation.

[0011] Optionally, performing compression operations on the data pages to be processed according to a preset compression level includes: The sliding window capacity is determined based on a preset compression level, and the sliding window capacity is positively correlated with the compression level. Using the current data segment to be compressed in the data page to be processed as the target data, query whether there is target historical data that matches the target data within the historical data range determined by the sliding window capacity; If so, a distance-length data pair is constructed using the distance between the target historical data and the starting position of the target data, and the matching length between the target historical data and the target data, and the distance-length data pair is used as the compression code of the target data; If not, the original bytes of the target data shall be used as the compressed encoding of the target data; The compressed codes of each target data are concatenated sequentially according to the order in which the target data is arranged in the data page to be processed, to generate a compressed data stream of the data page to be processed.

[0012] Optionally, querying whether there is target historical data matching the target data within the historical data range determined by the sliding window capacity includes: Determine the hash value of the target data; Within the historical data range determined by the sliding window capacity, if a historical data entry matching the hash value is found, the historical data entry is identified as candidate historical data. The target data is compared byte by byte with the candidate historical data to determine the matching length between the target data and the candidate historical data. In response to the matching length being greater than a preset length threshold, it is determined that there is target historical data that matches the target data, and the candidate historical data is used as the target historical data; In response to the matching length being less than or equal to a preset length threshold, it is determined that there is no target historical data that matches the target data.

[0013] According to another aspect of the present invention, a computer-readable storage medium is also provided, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the steps of the data compression method for any of the above-described databases.

[0014] According to another aspect of the present invention, a computer program product is also provided, comprising a computer program that, when executed by a processor, implements the steps of any of the above-described database data compression methods.

[0015] According to another aspect of the present invention, a computer device is also provided, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of any database data compression method.

[0016] The database data compression method of this invention performs a differential operation on multiple data tuples in the original data page, replacing the original data values ​​with the differential data between adjacent data tuples. This eliminates data redundancy caused by close numerical values ​​between data rows, making the data distribution in the differential data page more concentrated and improving the compression ratio. By performing a transpose operation on the data matrix during the differential operation, the row-order stored differential data is converted to column-order stored, ensuring that bytes in the same column of the data matrix are arranged consecutively in physical storage. This enhances the matching efficiency of the compression algorithm for consecutive and repetitive data, further improving the compression ratio. By performing compression operations on the data page to be processed using a preset compression level, the compression engine can select the corresponding compression level according to different business scenarios, achieving dynamic adaptation between compression strategies and business requirements.

[0017] Furthermore, the database data compression method of the present invention ensures that the transpose operation can be performed on the complete matrix structure by statistically analyzing the byte length of each differential data row after binary conversion and obtaining the smallest reference byte length from each byte length as the consistent column number of the data matrix. This avoids transpose errors caused by inconsistent row lengths. For differential data rows whose byte length exceeds the reference byte length, the excess bytes are offset sequentially and appended to the end of the data matrix in row order to form additional column vectors. Bytes at the same offset position in each differential data row are extracted as column vectors of the data matrix, so that bytes originally scattered in the same position in each row are arranged continuously in physical storage. When the compression algorithm scans the byte stream, the continuous arrangement of bytes in the same column has higher repetition, and can match more repetitive bytes with a shorter scan path, thereby improving the compression ratio. The data matrix is ​​constructed based on the minimum byte length, ensuring that all bytes of all differential data rows are included in the matrix structure for transpose processing, avoiding information loss due to data truncation, and ensuring that the data can be completely recovered during decompression.

[0018] Furthermore, the database data compression method of the present invention determines the sliding window capacity according to a preset compression level and makes the sliding window capacity positively correlated with the compression level. This allows for the selection of the corresponding compression level based on different business scenario requirements for compression ratio and read / write performance, achieving dynamic adaptation of the compression engine to different business scenarios. By querying target historical data matching the target data within the historical data range determined by the sliding window capacity, and replacing the original bytes of the target data with distance-length data pairs when a match is found, duplicate data only needs to be represented by position and length information, eliminating the need to repeatedly store the complete data content and reducing the storage space occupied by the data. When no matching target historical data is found, the original bytes of the target data are used as the compression code, and the compression codes of each target data are sequentially concatenated according to the arrangement order of the target data in the data page to be processed. This ensures that the compressed data stream retains the complete information and arrangement order of the original data while guaranteeing lossless recovery of the original data during decompression.

[0019] The above and other objects, advantages and features of the present invention will become more apparent to those skilled in the art from the following detailed description of specific embodiments of the invention in conjunction with the accompanying drawings. Attached Figure Description

[0020] The following sections will describe some specific embodiments of the invention in detail by way of example and not limitation, with reference to the accompanying drawings. The same reference numerals in the drawings denote the same or similar parts or portions. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings: Figure 1 This is a schematic flowchart of a database data compression method according to an embodiment of the present invention; Figure 2 This is a schematic flowchart illustrating the determination of column vectors in a database data compression method according to an embodiment of the present invention; Figure 3 This is a schematic flowchart of the compression operation in a database data compression method according to an embodiment of the present invention; Figure 4 This is a schematic flowchart illustrating the query of target historical data in a database data compression method according to an embodiment of the present invention; Figure 5 This is a schematic diagram of a computer program product according to an embodiment of the present invention; Figure 6 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present invention; Figure 7 This is a schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0021] In database systems, data is organized and managed in fixed-size pages, which are compressed by a compression engine. The compression engine uses a general compression algorithm to compress the data pages, which contain a header, row pointer arrays, and tuple data. This general compression algorithm treats the data pages as an unstructured byte stream, failing to utilize the unique characteristics of the different components within the data page, thus limiting the compression ratio.

[0022] As database runtime increases and data volume accumulates, tuples in the same column of a data page tend to be similar in content, and the numerical differences in the same column within adjacent data rows are small. Physically, these are represented in binary form, resulting in a large number of repetitive or highly similar byte sequences within the data page. However, common compression algorithms treat the entire data page as an unstructured byte stream, and their matching and encoding strategies are designed for general data, making it difficult to perceive the numerical proximity and structural similarity between specific columns within a database data page. This leads to these potential repetitive patterns not being fully utilized, and the compression ratio cannot be optimized.

[0023] The trade-off between compression ratio and read / write performance varies across different business scenarios. In archiving scenarios, data is rarely accessed after being written, making storage space cost the primary consideration, requiring the highest possible compression ratio to save disk space. In OLTP (Online Transaction Processing) scenarios, data is accessed frequently, making read / write performance a priority, requiring the highest possible I / O response speed. However, general-purpose compression algorithms use a single, fixed compression strategy, making it difficult to dynamically adjust the compression intensity according to the different weighting requirements of compression ratio and read / write performance in these different business scenarios. This limits the adaptability of the compression engine to various business scenarios, making it impossible to achieve a balance between storage space saving and access performance.

[0024] This embodiment provides a database data compression scheme. By fully utilizing the similarity and structural characteristics of column data within a data page, it achieves improved compression ratio and dynamic adaptation to business scenarios.

[0025] Figure 1 This is a schematic flowchart of a database data compression method according to an embodiment of the present invention. Figure 1 As shown, the data compression method for this database can generally include: Step S101: Obtain the original data page to be compressed, perform a differential operation on multiple data tuples in the original data page to obtain a differential data page, and store the differential data between adjacent data tuples in the differential data page; Step S102: Determine the data matrix corresponding to the differential data page, and perform a transpose operation on the data matrix during the differential operation to obtain the data page to be processed from the transposed data matrix; Step S103: Perform compression operation on the data pages to be processed according to the preset compression level.

[0026] In this embodiment, the database system reads the complete original data page to be compressed from the disk cache or memory buffer through the storage interface. The original data page contains multiple data tuples arranged in order. One of the data tuples is retained as the base data. For each data tuple other than the base data, a reference data tuple adjacent to the base data is determined in the original data page. The difference between the base data tuple and the reference data tuple is calculated to obtain the differential data corresponding to the base data tuple. The base data and each differential data are integrated according to the arrangement order of the multiple data tuples in the original data page to generate a differential data page.

[0027] In this embodiment, the data matrix uses each difference data point as a row vector and the sub-units within each difference data point, divided according to their corresponding data sequence positions, as column vectors. During the generation of difference data row by row in the difference operation, each generated difference data point is written into the data matrix as a row. After all difference operations are completed and all difference data rows have been written into the data matrix, a transpose operation is performed on the data matrix, converting it from row-major order storage to column-major order storage. This ensures that sub-units at the same data sequence position are arranged consecutively in storage, resulting in the data page to be processed.

[0028] In this embodiment, the retrieval range is determined according to the compression level. The current data segment is retrieved sequentially from the data page to be processed. Historical data matching the current data segment is queried within the retrieval range. If matching historical data is found, the reference information of the historical data is used as the compression code of the current data segment. If no matching historical data is found, the original data of the current data segment is used as its compression code. The above process is repeated until all data segments in the data page to be processed are processed. The compression codes of each data segment are combined into a compressed data stream according to the order of each data segment in the data page to be processed.

[0029] In some embodiments, the compression engine has multiple pre-configured compression algorithms, including ZSTD, ZLIB, and PGLZ. The appropriate compression algorithm is selected based on the preset compression level and / or data type. For example, ZSTD is selected in scenarios requiring a balance between compression ratio and performance; ZLIB is selected in archiving scenarios requiring extreme compression ratios; and PGLZ is selected in database-embedded compression scenarios.

[0030] Based on the above steps, by performing a differential operation on multiple data tuples in the original data page, the original data values ​​are replaced with the differential data between adjacent data tuples. This eliminates data redundancy caused by close numerical values ​​between data rows, making the data distribution in the differential data page more concentrated and improving the compression ratio. By performing a transpose operation on the data matrix during the differential operation, the row-order stored differential data is converted to column-order stored data, ensuring that bytes in the same column of the data matrix are arranged consecutively in physical storage. This enhances the matching efficiency of the compression algorithm for consecutive and repetitive data, further improving the compression ratio. By performing compression operations on the data pages to be processed using preset compression levels, the compression engine can select the appropriate compression level according to different business scenarios, achieving dynamic adaptation between compression strategies and business requirements.

[0031] In the implementation of step S101, performing a differential operation on multiple data tuples in the original data page to obtain a differential data page includes: The last row of the original data page is used as the starting row for differential scanning. The data is backtracked row by row in descending order of row number to determine the difference between the current row data tuple and the previous row data tuple. The difference data is stored in ascending order of row number to generate a differential data page. The first row data tuple of the original data page is used as the base data. The base data and the differential data page are used together as the compressed data storage structure.

[0032] In this embodiment, the original data page contains N rows of data tuples R1, R2, …, R N For the data tuple in the i-th row (i = N, N-1, …, 2), calculate D_i = R_i - R_{i-1}, where D_i is the inter-row difference between the i-th row and the (i-1)-th row. Store the calculated difference values ​​D_i in ascending order of row number; that is, D_2 is stored in the first row of the difference data page, D_3 in the second row, …, D_N in the (N-1)-th row, thus generating the difference data page. Simultaneously, the first data tuple R1 of the original data page is independently saved as the base data, which is used as the initial value for cumulative restoration during the decompression stage. The base data and the difference data page together constitute the compressed data storage structure.

[0033] Based on the above implementation, the difference operation utilizes the temporal or spatial correlation between adjacent data tuples to transform the original data value into a difference value between adjacent rows. This can narrow the range of data amplitudes, reduce the information entropy of the data, and provide better data distribution characteristics for subsequent compression processing. For data sequences exhibiting monotonicity, stationarity, or slow variation, the amplitude of the differenced data is significantly smaller than the amplitude of the original data, thereby improving coding efficiency.

[0034] In the implementation of step S102, determining the data matrix corresponding to the differential data page includes: The data matrix of the differential data page is constructed by using each differential data row as a row vector and the bytes in each differential data row that are at the same byte offset as column vectors.

[0035] In this embodiment, the differential data page contains M differential data rows (M = N-1, where N is the number of rows in the original data page), and each row contains L bytes. Each row is used as a row vector of the data matrix to construct a data matrix of dimension M×L. Each row of this data matrix corresponds to a differential data row, and each column corresponds to the byte at the same byte offset position in each differential data row. The element A[j][k] of the data matrix represents the value at the offset position of the k-th byte in the j-th differential data row, where j = 1, 2, …, M; k = 1, 2, …, L. It should be noted that the elements of the data matrix are organized at the byte level and do not depend on specific data types or semantics. The number of columns in the data matrix is ​​equal to the byte length of a single differential data row, and the number of rows is equal to the total number of differential data rows in the differential data page.

[0036] Based on the above implementation method, by constructing a data matrix by using bytes at the same byte offset position in each differential data row as column vectors, the bytes at the same byte offset position scattered in each differential data row are arranged continuously in physical storage. When the compression algorithm processes the byte stream, the continuously arranged bytes have higher repetition and can match more repetitive bytes with a shorter scan path, thereby improving the compression rate.

[0037] Figure 2 This is a schematic flowchart illustrating the determination of column vectors in a database data compression method according to an embodiment of the present invention. Figure 2 As shown, bytes at the same byte offset in each differential data row are used as column vectors, including: Step S201: Calculate the byte length of each differential data row based on the binary conversion; Step S202: Obtain the minimum value of the reference byte length from the byte lengths, and use it as the consistent column number of the data matrix; Step S203: For target differential data rows whose byte length is greater than the base byte length, retain the original data of the target differential data rows and do not participate in the differential operation; Step S204: Extract the bytes at the same byte offset position in each differential data row into column vectors of the data matrix. The number of column vectors is equal to the length of the base byte. Each column vector is composed of bytes at the same byte offset position in the differential data row arranged in row order.

[0038] In this embodiment, for each differential data row in the differential data page, the byte length of the differential data row represented in binary form is obtained. The byte length of the i-th differential data row is L_i (i = 1, 2, …, M, where M is the number of rows in the differential data page), where, for the case of a negative differential value, the two's complement form of the negative value is used as its binary representation, and the byte length L_i is the number of bytes required to store the two's complement.

[0039] In this embodiment, the byte length with the smallest value among all byte lengths is selected as the reference byte length, and this reference byte length is determined as the consistent number of columns in the data matrix. For example, if the reference byte length is L_min = min (L_1,L_2, …, L_M), then the number of columns in the data matrix is ​​L_min. The reference byte length ensures that all valid data within the first L_min byte range in all differential data rows can be included in the data matrix to avoid data loss.

[0040] In this embodiment, for target difference data rows whose byte length is greater than the base byte length, the original data of the target difference data row is retained and does not participate in the difference operation. For example, if the byte length L_i of the i-th difference data row is greater than the base byte length L_min, then the original (L_i-L_min) bytes of data are retained at the end of the difference data row. After padding, the total length of the difference data row is consistent with the original length. When transposing the matrix, transpose according to the common length L_min first; the remaining parts with inconsistent lengths are transposed byte by byte, thus not adding extra data.

[0041] In this embodiment, bytes at the same byte offset position in each difference data row are extracted as column vectors of the data matrix. The number of column vectors is equal to the base byte length L_min, with the remaining bytes transposed separately. Each column vector consists of bytes at the same byte offset position in each difference data row arranged in ascending order of row number. For the k-th column vector (k = 1, 2, …, L_min), it contains M bytes, which are the byte values ​​at byte offset position k in each difference data row. All column vectors are arranged in ascending order of k, together forming the complete set of column vectors of the data matrix.

[0042] Based on the above steps, the byte length of each differential data row after binary conversion is statistically analyzed, and the smallest reference byte length is selected as the consistent column number of the data matrix. This ensures that the transpose operation can be performed on the complete matrix structure, avoiding transpose errors caused by inconsistent row lengths. For differential data rows whose byte length exceeds the reference byte length, the excess bytes are offset sequentially and appended to the end of the data matrix in row order, forming additional column vectors. Bytes at the same offset position in each differential data row are extracted as column vectors of the data matrix, so that bytes originally scattered in the same position in each row are arranged continuously in physical storage. When the compression algorithm scans the byte stream, bytes arranged continuously in the same column have higher repetition, and can match more repetitive bytes with a shorter scan path, thereby improving the compression ratio. The data matrix is ​​constructed based on the minimum byte length, ensuring that all bytes of all differential data rows are included in the matrix structure for transpose processing, avoiding information loss due to data truncation, and ensuring that the data can be completely recovered during decompression.

[0043] In the embodiment of step S102, performing a transpose operation on the data matrix during the difference operation includes: In response to all differential data rows in the differential data page being written to the data matrix, the bytes at the same byte offset position of each differential data row in the data matrix are read sequentially along the column direction. The read bytes are sorted in ascending order of row number to generate a byte sequence stored continuously in column order for transpose operation.

[0044] In this embodiment, a differential data row is generated in response to the differential operation, and this differential data row is written into the data matrix as a row of the data matrix. The above steps are repeated until all differential data rows of the differential data page have been written into the data matrix. After all differential data rows of the differential data page have been written into the data matrix, bytes at the same byte offset position of each differential data row in the data matrix are read sequentially along the column direction. The read bytes are sorted in ascending order of row number to generate a byte sequence stored contiguously in column order. This byte sequence is the transpose of the data matrix. The contiguously stored byte sequence in column order is used as the data page to be processed.

[0045] Based on the above implementation method, by uniformly performing a transpose operation on the data matrix after the differential operation is completed, the data layout is changed from row-major order to column-major order, so that bytes with the same byte offset position in each differential data row are arranged continuously in storage, which is beneficial for the compression algorithm to discover data redundancy within the column, thereby improving compression efficiency.

[0046] Figure 3 This is a schematic flowchart of the compression operation in a database data compression method according to an embodiment of the present invention. Figure 3As shown, the compression operation performed on the data pages to be processed according to the preset compression level includes: Step S301: Determine the sliding window capacity based on the preset compression level. The sliding window capacity is positively correlated with the compression level. Step S302: Using the current data segment to be compressed in the data page to be processed as the target data, query within the historical data range determined by the sliding window capacity whether there is target historical data that matches the target data; Step S303: If yes, construct a distance-length data pair using the distance between the target historical data and the starting position of the target data, and the matching length between the target historical data and the target data, and use the distance-length data pair as the compressed encoding of the target data. Step S304: If not, use the original bytes of the target data as the compression encoding of the target data; Step S305: The compressed codes of each target data are concatenated sequentially according to the order of the target data in the data page to be processed, to generate a compressed data stream of the data page to be processed.

[0047] In this embodiment, a preset compression level parameter, Level, is obtained. The compression level is a pre-set parameter used to control the degree and efficiency of compression, and its value range is a finite discrete level (such as an integer between 1 and 7). The sliding window capacity W is determined according to the compression level. The sliding window capacity W increases as the compression level Level increases. That is, the higher the compression level, the larger the sliding window capacity, and the wider the range of historical data that can be retrieved; the lower the compression level, the smaller the sliding window capacity, and the narrower the range of historical data that can be retrieved.

[0048] The sliding window capacity W is used to limit the upper limit of the amount of historical data that can be retrieved backward from the current data segment to be compressed during the compression encoding process. For the starting position pos of the current data segment to be compressed in the data page to be processed, the range of historical data covered by the sliding window is [max(1, pos - W), pos - 1].

[0049] In this embodiment, compression encoding is performed segment by segment along the data arrangement direction of the data page to be processed. The data segment starting from the current encoding position is taken as the target data Target. The length of the target data Target is a preset maximum matching length (such as 64 bytes, 256 bytes, etc.) or the remaining length to the end of the data page. The target data is the data unit to be encoded, and its starting position pos increases with the encoding progress.

[0050] Within the historical data range [max(1, pos - W), pos - 1] defined by the sliding window capacity W, query whether there exists target historical data that matches the target data Target. The historical data range consists of processed data located before the current target data and at a distance not exceeding W, and belongs to the data set that can be retrieved for matching encoding.

[0051] The query operation can use hash matching, string matching, or other similarity retrieval methods to determine whether there is a data segment in the historical data range that is the same as the target data. If so, the matching historical data segment is used as the target historical data.

[0052] In this embodiment, if target historical data exists that matches the target data, the relative positional relationship and matching length between the target historical data and the target data are obtained. Distance (Dist) is the byte offset between the starting position of the target historical data and the starting position of the target data, and its value is a positive integer. Matching length (MatchLen) is the number of consecutive identical bytes in the target historical data and the target data starting from their respective starting positions, and its value is a positive integer.

[0053] The distance Dist and the matching length MatchLen are combined into a (distance, length) tuple, i.e., a distance-length data pair. This data pair points to an existing data segment in the historical data. By recording the backtracking distance and the read length, the original bytes storing the target data are replaced, thereby achieving compression of the target data.

[0054] Distance-length data pairs are a form of compression encoding where the data size (usually several bytes) is much smaller than the original number of bytes in the target data, thus achieving data compression.

[0055] In this embodiment, if there is no target historical data that matches the target data, the target data is not replaced with encoding. Instead, the original bytes of the target data are kept unchanged and directly output as the compressed encoding of the target data.

[0056] The raw bytes are the original binary representation of the target data in the data page to be processed, without any compression or transformation. Although this encoding method does not produce a compression effect, it ensures that all data can be preserved intact in a lossless manner and provides a data foundation for possible matching of subsequent data segments.

[0057] In this embodiment, each target data is compressed and encoded sequentially according to the order from the first byte to the last byte in the data page to be processed. The compressed encoding (distance-length data pair or original bytes) corresponding to each target data is then concatenated end to end in sequence to form a continuous binary byte sequence, which serves as the compressed data stream of the data page to be processed.

[0058] The concatenation order strictly matches the position order of the target data within the data pages to be processed, ensuring that during decompression, each target data point can be restored sequentially in the same order, recovering the original data pages. The compressed data stream is the final compressed result of the data pages to be processed, which can be used for storage or transmission.

[0059] Based on the above steps, by determining the sliding window capacity according to the preset compression level and making the sliding window capacity positively correlated with the compression level, the appropriate compression level can be selected according to different business scenarios' requirements for compression ratio and read / write performance, achieving dynamic adaptation of the compression engine to different business scenarios. By querying target historical data matching the target data within the historical data range determined by the sliding window capacity, when a match is successful, the original bytes of the target data are replaced with distance-length data pairs. This ensures that duplicate data only needs to be represented by position and length information, eliminating the need to repeatedly store the complete data content and reducing the storage space occupied by the data. When no matching target historical data is found, the original bytes of the target data are used as the compression code, and the compression codes of each target data are concatenated sequentially according to the arrangement order of the target data in the data page to be processed. This ensures that the compressed data stream retains the complete information and arrangement order of the original data while guaranteeing lossless recovery of the original data during decompression.

[0060] Figure 4 This is a schematic flowchart illustrating the querying of target historical data in a database data compression method according to an embodiment of the present invention. Figure 4 As shown, querying whether there is target historical data matching the target data within the historical data range determined by the sliding window capacity includes: Step S401: Determine the hash value of the target data; Step S402: Within the historical data range determined by the sliding window capacity, if a historical data entry matching the hash value is found, the historical data entry is identified as candidate historical data. Step S403: Compare the target data with the candidate historical data byte by byte to determine the matching length between the target data and the candidate historical data; Step S404: In response to the matching length being greater than a preset length threshold, it is determined that there is target historical data that matches the target data, and candidate historical data is used as target historical data. Step S405: In response to the matching length being less than or equal to a preset length threshold, it is determined that there is no target historical data that matches the target data.

[0061] In this embodiment, the current data segment to be compressed is obtained as the target data. A preset hash function is applied to the target data to calculate the hash value corresponding to the target data. The hash function is a hash function that maps binary data of arbitrary length to a fixed-length value, and its output value is the hash value. The hash value is used to quickly locate historical data entries that may have a matching relationship with the target data within the historical data range. By performing matching retrieval through hash value, the time complexity of matching search can be reduced from O(W) to O(1), thereby significantly improving the compression processing efficiency.

[0062] In this embodiment, within the historical data range covered by the sliding window capacity, each historical data entry is traversed, its hash value is calculated, and compared with the hash value of the target data. If a historical data entry with the same hash value exists, it is marked as a candidate historical data entry. A historical data entry is a data segment in the processed data that has the same or similar length to the target data, based on the current processing position. The hash value of each historical data entry can be pre-calculated and stored during historical data processing for subsequent queries to avoid redundant calculations. If no historical data entry matching the hash value of the target data is found, no candidate historical data exists.

[0063] In this embodiment, starting from the first byte of the target data, it is compared byte by byte with the first byte of the candidate historical data, until the first unequal byte is found or the preset maximum matching length boundary is reached. The number of consecutive equal bytes is recorded as the matching length. When there are multiple candidate historical data, each candidate historical data is compared byte by byte to determine the matching length between each candidate historical data and the target data, and the candidate historical data with the largest matching length is selected.

[0064] In this embodiment, the length threshold is a preset positive integer value, and its value affects the matching sensitivity of compression. If the length threshold is set too small, short-length reference codes are allowed to pass. When the storage space occupied by the reference code (including distance and length information) is greater than or equal to the original data being replaced, the reference code cannot achieve effective compression benefits, resulting in a decrease in compression ratio. If the length threshold is set too large, the conditions for successful matching are too strict, which may miss target historical data that can be effectively compressed, also resulting in the compression ratio not reaching the optimal level. It should be noted that the length threshold should be adjusted according to the requirements of compression ratio and compression speed in the actual application scenario.

[0065] If the matching length is greater than the length threshold, it is determined that there is valid target historical data matching the target data within the current search range, and this candidate historical data is identified as the target historical data. If the matching length is less than or equal to the length threshold, it is determined that there is no valid target historical data matching the target data within the current search range. In this case, the target data will not be encoded in reference form, but will be output as its original byte sequence using compressed encoding.

[0066] Based on the above steps, the length threshold determination mechanism avoids mismatches caused by hash collisions and invalid references introduced due to excessively short matching lengths, thus ensuring the effectiveness of compressed encoding and the quality of compressed data.

[0067] The flowchart provided in this embodiment is not intended to indicate that the operations of the method will be performed in any particular order, or that all operations of the method are included in every case. Furthermore, the method may include additional operations. Within the scope of the technical concept provided by the method in this embodiment, additional variations can be made to the above method.

[0068] It should be understood that in some embodiments, the components may be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods may be implemented using software or firmware stored in memory and executed by a suitable instruction execution system.

[0069] This embodiment also provides a computer program product 10, a computer-readable storage medium 20, and a computer device 30. Figure 5 This is a schematic diagram of a computer program product 10 according to an embodiment of the present invention. Figure 6 This is a schematic diagram of a computer-readable storage medium 20 according to an embodiment of the present invention. Figure 7 This is a schematic diagram of a computer device 30 according to an embodiment of the present invention. Figure 5 As shown, the computer program product 10 includes a computer program 11, which, when executed by the processor 32, implements the steps of any of the above-described database data compression methods. For example... Figure 6 As shown, a computer-readable storage medium 20 stores the aforementioned computer program 11, which, when executed by the processor 32, implements the steps of the database data compression method of any of the above embodiments. Figure 7 As shown, the computer device 30 may include a memory 31, a processor 32, and a computer program 11 stored on the memory 31 and running on the processor 32.

[0070] The computer program 11 used to perform the operations of this invention may be assembly instructions, Instruction Set Architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages ​​and procedural programming languages. The computer program 11 may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer. In some embodiments, to perform aspects of this invention, electronic circuits, including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute computer-readable program instructions to personalize the electronic circuits by utilizing status information of the computer-readable program instructions.

[0071] For the purposes of this embodiment, computer program product 10 is a related product that includes computer program 11.

[0072] For the purposes of this embodiment, computer-readable storage medium 20 is a tangible device capable of holding and storing a computer program 11. It can be any device capable of containing, storing, communicating, propagating, or transmitting the program 11 for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable storage medium 20 include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital versatile disc (DVD), memory stick, floppy disk, mechanical encoding device, and any suitable combination thereof.

[0073] Computer device 30 can be, for example, a server, desktop computer, laptop computer, tablet computer, or smartphone. In some examples, computer device 30 can be a cloud computing node. Computer device 30 can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., that perform specific tasks or implement specific abstract data types. Computer device 30 can be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can reside on local or remote computing system storage media, including storage devices.

[0074] Computer device 30 may include a processor 32 adapted to execute stored instructions and a memory 31 that provides temporary storage space for the operation of said instructions during operation. The processor 32 may be a single-core processor, a multi-core processor, a computing cluster, or any other configuration. The memory 31 may include random access memory (RAM), read-only memory, flash memory, or any other suitable storage system.

[0075] Computer device 30 may also include a network adapter / interface and an input / output (I / O) interface. The I / O interface allows external devices that can be connected to the computer device to input and output data. The network adapter / interface provides communication between the computer device and a network, typically represented as a communication network.

[0076] Therefore, those skilled in the art should recognize that although numerous exemplary embodiments of the present invention have been shown and described in detail herein, many other variations or modifications conforming to the principles of the present invention can be directly determined or derived from the disclosure of the present invention without departing from the spirit and scope of the invention. Thus, the scope of the present invention should be understood and construed as covering all such other variations or modifications.

Claims

1. A data compression method for a database, characterized in that, include: Obtain the original data page to be compressed, perform a differential operation on multiple data tuples in the original data page to obtain a differential data page, and store the differential data between adjacent data tuples in the differential data page; Determine the data matrix corresponding to the differential data page, and perform a transpose operation on the data matrix during the differential operation to obtain the data page to be processed from the transposed data matrix; The data pages to be processed are compressed according to the preset compression level.

2. The database data compression method according to claim 1, characterized in that, Performing a difference operation on multiple data tuples in the original data page to obtain a differential data page includes: Using the last row of the original data page as the starting row of the differential scan, the scan proceeds back row by row in descending order of row number to determine the differential data between the current row data tuple and the previous row data tuple. The differential data is then stored in ascending order of row number to generate the differential data page. The first row data tuple of the original data page is used as the base data, and the base data and the differential data page together form the compressed data storage structure.

3. The database data compression method according to claim 1, characterized in that, Determining the data matrix corresponding to the differential data page includes: Each differential data row in the differential data page is used as a row vector, and the bytes in each differential data row that are at the same byte offset position are used as column vectors to construct the data matrix of the differential data page.

4. The database data compression method according to claim 3, characterized in that, The bytes at the same byte offset position in each of the differential data rows are included as a column vector: The length of each differential data row is calculated based on the byte length after binary conversion. The minimum reference byte length among all the byte lengths is selected as the consistent column number of the data matrix; For target differential data rows whose byte length is greater than the base byte length, the original data of the target differential data rows is retained and not used for differential operations; The bytes at the same byte offset position in each of the differential data rows are extracted as column vectors of the data matrix, wherein the number of column vectors is equal to the length of the base byte, and each column vector is composed of the bytes at the same byte offset position in the differential data row arranged in row order.

5. The database data compression method according to claim 1, characterized in that, During the execution of the difference operation, performing a transpose operation on the data matrix includes: In response to all differential data rows in the differential data page being written into the data matrix, bytes at the same byte offset position of each differential data row in the data matrix are read sequentially along the column direction. The read bytes are sorted sequentially according to the row number in ascending direction to generate a byte sequence stored continuously in column order, so as to perform the transpose operation.

6. The database data compression method according to claim 1, characterized in that, The compression operation performed on the data page to be processed according to the preset compression level includes: The sliding window capacity is determined based on a preset compression level, and the sliding window capacity is positively correlated with the compression level. Using the current data segment to be compressed in the data page to be processed as the target data, query whether there is target historical data that matches the target data within the historical data range determined by the sliding window capacity; If so, a distance-length data pair is constructed using the distance between the target historical data and the starting position of the target data, and the matching length between the target historical data and the target data, and the distance-length data pair is used as the compression code of the target data; If not, the original bytes of the target data shall be used as the compressed encoding of the target data; The compressed codes of each target data are concatenated sequentially according to the order in which the target data is arranged in the data page to be processed, to generate a compressed data stream of the data page to be processed.

7. The database data compression method according to claim 6, characterized in that, Querying whether there is target historical data matching the target data within the historical data range determined by the sliding window capacity includes: Determine the hash value of the target data; Within the historical data range determined by the sliding window capacity, if a historical data entry matching the hash value is found, the historical data entry is identified as candidate historical data. The target data and the candidate historical data are compared byte by byte to determine the matching length between the target data and the candidate historical data. In response to the matching length being greater than a preset length threshold, it is determined that there is target historical data that matches the target data, and the candidate historical data is used as the target historical data; In response to the matching length being less than or equal to a preset length threshold, it is determined that there is no target historical data that matches the target data.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the data compression method for the database according to any one of claims 1 to 7.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the data compression method for the database as described in any one of claims 1 to 7.

10. A computer device, characterized in that, The system includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the data compression method for the database according to any one of claims 1 to 7.

Images