Compression method of sparse data
By optimizing the memory layout and ORC encoding of UnsafeRow, the problems of large memory consumption and storage waste of sparse data in Apache Spark are solved, achieving more efficient data processing and storage.
Patent Information
- Application Number
- CN202511642986.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-11
- Publication Date
- 2026-02-10
AI Technical Summary
In existing technologies, sparse data in Apache Spark suffers from problems such as high memory consumption, low memory utilization, high GC pressure, high network I/O and storage I/O consumption, and wasted ORC storage space.
By optimizing the memory layout of UnsafeRow, compressing the memory usage of sparse data, reducing network and disk overhead, and using ORC encoding optimization, data storage space is reduced.
Significantly reduces memory footprint for sparse data, reduces GC pressure, lowers network and storage I/O, optimizes ORC storage space, and improves data processing efficiency for Spark SQL.
Smart Images
Figure CN121508545A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application belongs to the field of big data and real-time computing, and particularly relates to a compression method for sparse data. BACKGROUND
[0002] Apache Spark is a popular distributed computing framework, and its data is mainly stored, exchanged and processed through UnsafeRow. UnsafeRow is deeply integrated with the Tungsten execution engine, and has the advantages of low object overhead, high serialization efficiency and high performance.
[0003] Defects or shortcomings of prior art:
[0004] 1. Large memory occupation and low memory utilization in UnsafeRow: the existing technology needs to occupy 8 bytes in the fixed-length area even if the value of a field is empty, but it will not be used in access. For sparse data (data with many fields but few fields with values), there is a problem of large memory occupation and low memory utilization. For example, if there are 1024 fields and only 100 fields have values, nearly 8K of memory will be wasted, and when 1 million data are managed in memory, 8G of memory will be wasted.
[0005] 2. Frequent memory allocation and release during writing, and large GC pressure: the memory allocated by UnsafeRow each time is fixed (64 bytes in the variable-length area), and when the variable-length area is full, a new buffer needs to be applied and the data written in the old buffer needs to be copied to the new buffer. When multiple fields are written, the memory will be allocated and copied multiple times, resulting in large GC pressure and affecting performance.
[0006] 3. Large network I / O occupation: Apache SparkSQL often needs to transmit data between multiple processes (such as shuffle) when processing a large amount of data. The data is serialized according to UnsafeRow and transmitted between processes. In the presence of a large amount of sparse data, a large amount of network I / O is occupied for transmission, and the transmission efficiency is low.
[0007] 4. Large storage I / O occupation: in the sorting overflow and shuffle overflow scenarios, UnsafeRow needs to be written to the disk, and then read from the disk in subsequent processing. In the scenario of sparse data, a large amount of storage I / O is wasted by empty fields.
[0008] 5. ORC null disk occupancy is high: ORC stores statistics information even if a column in a stripe is all null, and the same applies to columns that are all null in a file. In the case of sparse data, there are a large number of all-null columns, resulting in a large amount of invalid data occupying storage space and wasting storage resources.
[0009] 6. Statistics information of all-null columns wastes computing resources: Both stripe and file level column statistics information writing and reading require protobuf encoding and decoding, and compression and decompression. A large amount of statistics information of all-null column data brings a large amount of computing overhead, wasting computing resources. SUMMARY
[0010] The technical problem to be solved by the present application is to provide a sparse data compression method to optimize the memory layout of UnsafeRow, compress the memory usage of sparse data, improve the memory usage efficiency of data, reduce the overhead of network and disk, and improve the data processing efficiency of Spark SQL. At the same time, the storage space of data storage is reduced by optimizing the encoding of ORC.
[0011] The present application adopts the following technical solutions to solve the above technical problems:
[0012] A sparse data compression method, specifically comprising the following steps:
[0013] Step 1, UnsafeRow data memory layout optimization: the memory layout of UnsafeRow is optimized, and the optimized memory layout includes field space bitmap, fixed-length area and variable-length area; wherein UnsafeRow is a row data encoding format in Apache Spark, and is a main data transmission and processing format in Spark;
[0014] Step 2, UnsafeRow data encoding;
[0015] Step 3, UnsafeRow data decoding;
[0016] Step 4, serialization and deserialization of compressed UnsafeRow;
[0017] Step 5, ORC write optimization; wherein ORC is a high-efficiency and high-performance columnar storage format;
[0018] Step 6, ORC read optimization.
[0019] As a further preferred scheme of the present application, in step 1, UnsafeRow data memory layout optimization specifically comprises:
[0020] Step 1.1: Empty plots and variable-length regions are compatible with native formats;
[0021] Step 1.2: Reduce the memory footprint of sparse data by optimizing the memory layout of fixed-length regions and the encoding / decoding methods.
[0022] Fixed-length areas only store the values, offsets, or lengths of non-empty fields;
[0023] The system utilizes encoding and decoding methods to provide interface-compatible read and write capabilities.
[0024] Step 1.3: Record the number of empty fields to calculate the offset and length of the fixed-length area.
[0025] As a further preferred embodiment of the invention of a sparse data compression method, in step 2, the UnsafeRow data encoding is specifically as follows:
[0026] Step 2.1: Allocate Writers from the memory pool according to the total number of fields in the data to avoid repeatedly allocating and releasing memory.
[0027] When writing a row of data, a writer is requested from the pool. If the memory required for the row of data is less than the memory already in the writer, the memory in the writer is reused; if the memory required for the data is greater than the memory already in the writer, new memory is requested, and this memory can be reused for subsequent writes.
[0028] If multiple data fields in the same table are identical, multiple rows of data can share the memory in the writer, avoiding multiple allocations and releases;
[0029] Step 2.2, write the value of the field:
[0030] Field value is empty: Set the corresponding field in the empty bitmap to empty and increment the number of empty fields by 1;
[0031] Field value NOT null: Sets the corresponding field in the empty bitmap to NOT null.
[0032] Fixed-length values: When a field is a fixed-length value such as bool, int, or long, the value of the corresponding field is written into the corresponding field in the fixed-length area;
[0033] Non-fixed-length values: When the field value is a non-fixed-length value such as string or decimal, the data is first written to the variable-length area, and then the offset and length are written to the fixed-length area.
[0034] Step 2.3, Generate Compressed UnsafeRow: After all fields have been written, compressed data is generated using the following algorithm:
[0035] Calculate the compressed memory size: Memory size = data length in Writer - empty field data * 8; Allocate memory for compressed data, and allocate memory for the data in line a) as a data buffer.
[0036] Encode the data as follows: copy the empty bitmap from the writer to the buffer;
[0037] Writing to the fixed-length area: First, set the offset of the fixed-length area to 0, and then process it according to the field order as follows: If the value is not non-empty, write the 8-byte value of the field to the memory corresponding to the offset, and increment the offset by 8; if the value is empty, do not process it; copy the data of the writer's variable-length area to the variable-length buffer.
[0038] Step 2.4, Release Writer: Return the writer to the pool for subsequent writing;
[0039] Step 2.5: Return the compressed UnsafeRow.
[0040] As a further preferred embodiment of the invention of a sparse data compression method, in step 3, the decoding of the UnsafeRow data specifically includes:
[0041] Step 3.1, Calculate the fixed-length area offset by index: Since the compressed UnsafeRow only stores the values of non-null fields in the fixed-length area, the data in the fixed-length area cannot be accessed directly using column subscripts. The following correction is required, and the method is as follows:
[0042] The number of all non-empty fields preceding the given field is calculated using the empty bitmap and used as the index of the fixed-length region.
[0043] For scenarios involving sequential access of multiple fields: read empty spaces from the previously read field, and calculate the number of non-empty fields before that field as the index of the fixed-length area;
[0044] Step 3.2: If the data type of the column being read is a non-variable length value, read the value of the data using the subscript corrected in 1), and return it;
[0045] Step 3.3: For variable-length data types, read the offset and length of the data using the index corrected in Step 3.1;
[0046] Step 3.4, Calculate the variable-length offset: Since the offset is recorded in an uncompressed format, it needs to be extracted from the offset after compression.
[0047] Subtract the length of the compressed memory; after modification, offset = offset - 8 * number of empty fields;
[0048] Step 3.5, Read Data: Read data from the variable-length area using the corrected offset and return it.
[0049] As a further preferred embodiment of the invention of a sparse data compression method, in step 4, the serialization and deserialization of the compressed UnsafeRow are specifically as follows:
[0050] Among them, (a) serialization:
[0051] Number of bytes written: Total number of bytes in the bitmap area, compressed fixed-length area, and variable-length area;
[0052] Write data: Includes all data in the bitmap area, compressed fixed-length area, and variable-length area;
[0053] Write compression identifier: This identifier is used to distinguish between compressed and uncompressed scenarios to ensure compatibility.
[0054] Number of empty fields written;
[0055] (b) Deserialization:
[0056] Read the total number of bytes and allocate memory based on that number of bytes;
[0057] Read data: including all data in the bitmap area, compressed fixed-length area, and variable-length area;
[0058] Read compression flag: When this flag is true, decoding needs to be processed according to the compression format;
[0059] Reads the number of empty fields, used for decoding compressed data.
[0060] As a further preferred embodiment of the invention of a sparse data compression method, in step 5, the ORC write optimization specifically includes the following steps;
[0061] Step 5.1, Writing Data: When writing data, the encoding and compression of the data in memory are completed according to the ORC process; when the memory size reaches the Stripe threshold, the memory data is written to the file as a whole according to the stripe. The optimized Stripe writing process is as follows:
[0062] Determine if a column is completely empty based on the data in each column's statistics. If the column is completely empty, delete the rowgroup statistics for that column and record the column ID of the empty column in the StripeFooter.
[0063] When writing to stripeIndex, only data in non-fully empty columns is written; data in fully empty columns and statistical information are not written to disk.
[0064] After completing the writing of stripeIndex and RowData, update the non-empty column data and the length of the index in the footer, and write to StripeFooter;
[0065] Step 5.2, when closing a file, it is necessary to write the file's metadata and file footer. The optimized steps are as follows:
[0066] Based on file-level statistics, determine if the data in a column is completely empty. If it is, delete the Stripe statistics for that column and record the ID of the empty column in the Footer file.
[0067] Add statistics for non-fully empty columns to the metadata and write them to a file;
[0068] Write to FileFooter.
[0069] As a further preferred embodiment of the invention of a sparse data compression method, in step 6, the ORC read optimization specifically includes the following steps;
[0070] Step 6.1, Filter Stripe by Predicate Conditions: First, read stripe statistics from the metadata, then insert null value column statistics into the corresponding column based on the null value column ID in the footer, with multiple columns sharing one null value column statistics; then perform stripe filtering based on predicate conditions.
[0071] Step 6.2, filter rowgroups by predicate condition: First, determine whether the predicate contains a completely empty column by checking the null value column ID in the stripe footer. For completely empty columns, construct completely empty statistics in memory as the rowgroup statistics for that column. For non-empty columns, read the column statistics from the file and then perform predicate filtering.
[0072] Step 6.3, when reading data: First, check if the corresponding column is completely empty by checking the Stripe footer. If the column is completely empty in the stripe, use the empty column reader. This reader does not read the file, directly constructs an empty value and returns the data. After returning the data, the column will no longer access the file. For columns that are not completely empty, read the data from the file and return it.
[0073] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:
[0074] 1) Reduce data memory usage: UnsafeRow reduces data memory usage by compressing empty fields. This is especially effective for sparse data scenarios, significantly reducing memory consumption.
[0075] 2) Reduce memory allocation and deallocation to lower GC pressure: Reuse UnsafeRow Writer and memory for multiple rows to avoid allocating and deallocating memory multiple times for each row of data, thus reducing GC pressure;
[0076] 3) Reduce network and storage I / O: By using UnsafeRow data compression, the amount of data transmitted over the network and written to and read from the disk is reduced, thereby reducing network and storage I / O consumption;
[0077] 4) ORC optimization reduces storage space: When writing with ORC, statistics on completely empty columns are filtered at both the Stripe and file levels, and are no longer written to the file, reducing the amount of data written to the file and thus reducing storage space;
[0078] 5) Reduce ORC read / write computational overhead: Writing and reading fully empty columns no longer processes the statistical information of fully empty columns, which reduces the computational overhead of protobuf encoding and decoding, as well as data compression and decompression. Reading data from fully empty columns no longer involves reading the file, decompression, and decoding, further reducing the computational overhead of data reading;
[0079] 6) Seamless Integration: This invention is fully compatible with UnsafeRow's write, read, serialization, and deserialization methods, enabling seamless integration with Apache Spark. The ORC optimization is fully compatible with existing ORC interfaces and does not affect integration with other software. Attached Figure Description
[0080] Figure 1 This is a flowchart of the UnsafeRow data encoding method;
[0081] Figure 2 This is a flowchart of the UnsafeRow data decoding method;
[0082] Figure 3 Here is a flowchart of ORC write optimization methods;
[0083] Figure 4 This is a flowchart of data memory sorting and writing to the document. Detailed Implementation
[0084] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings:
[0085] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention. The present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments. The purpose and effects of the present invention will become clearer. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.
[0086] This invention optimizes the memory layout of data while maintaining compatibility with Spark SQL's native UnsafeRow interface, and provides data encoding, decoding, and serialization schemes based on the optimized layout. Simultaneously, it optimizes the compression and reading methods for ORC for sparse data.
[0087] Sparse data refers to data with a large number of fields, but each row of data has a high number of value fields. For example, some network protocol data may have thousands of fields, but only 100 of them have values.
[0088] UnsafeRow: A row data encoding format in Apache Spark, and a primary data transmission and processing format in Spark.
[0089] ORC is a high-efficiency, high-performance columnar storage format designed specifically for big data analytics scenarios and widely used in the Hadoop ecosystem.
[0090] 1. Optimization methods for UnsafeRow data memory layout
[0091] To address the characteristics of sparse data with many fields and few fields with values, the optimized memory layout of this solution is shown in Table 1:
[0092] Table 1
[0093]
[0094] a) Empty plots and variable-length areas are compatible with native formats;
[0095] b) Reduce the memory footprint of sparse data by optimizing the memory layout of fixed-length regions and encoding / decoding methods:
[0096] Fixed-length areas only store the values of non-empty fields or offset / length;
[0097] The system utilizes encoding and decoding methods to provide interface-compatible read and write capabilities.
[0098] c) Record the number of empty fields to calculate the offset and length of the fixed-length area;
[0099] 2. Data encoding method for UnsafeRow:
[0100] For the memory layout of this scheme, the encoding method during data writing is as follows: Figure 1 As shown:
[0101] 1) Allocate Writers from the pool based on the total number of fields in the data to avoid repeatedly allocating and releasing memory:
[0102] When writing a row of data, a writer is requested from the pool. If the memory required for the row is less than the memory already in the writer, the memory in the writer is reused. If the memory required for the row is greater than the memory already in the writer, new memory is requested, and this memory can be reused for subsequent writes.
[0103] If multiple data fields in the same table are identical, multiple rows of data can share memory in the writer, avoiding multiple allocations and releases.
[0104] 2) Write the value of the field:
[0105] Field value is empty: Set the corresponding field in the empty bitmap to empty and increment the number of empty fields by 1;
[0106] Field value not null: Set the corresponding field in the empty bitmap to not null, and:
[0107] Fixed-length values: When the field is a fixed-length value such as bool, int, or long, the value of the corresponding field will be written to the fixed-length value.
[0108] The corresponding field in the area.
[0109] Non-fixed-length values: When the field value is a non-fixed-length value such as string or decimal, the data is first written to the variable-length area.
[0110] Then write the offset and length into the fixed-length area.
[0111] 3) Generate Compressed UnsafeRow: After all fields have been written, compressed data is generated using the following algorithm:
[0112] a) Calculate the compressed memory size: Memory size = Data length in Writer - Empty field data * 8
[0113] b) Allocate memory for compressed data. Based on the memory size calculated in a), allocate memory for this row of data.
[0114] buffer.
[0115] c) Encode the data using the following steps:
[0116] The empty bitmap in the copy writer is copied to the buffer.
[0117] Writing to the fixed-length area: First, set the offset of the fixed-length area to 0, and then process it according to the field order as follows: If the value is not non-empty, write the 8-byte value of the field to the memory corresponding to the offset, and increment the offset by 8; if the value is empty, do not process it; copy the data of the writer's variable-length area to the variable-length buffer.
[0118] 4) Release Writer: Return the writer to the pool for subsequent writing;
[0119] 5) Return the compressed UnsafeRow;
[0120] 3. Decoding methods for UnsafeRow data
[0121] Because data in the fixed-length region is no longer stored in field order, it's impossible to directly access data by column index. An algorithm is needed to calculate the position of non-empty data within the fixed-length region. Simultaneously, due to memory compression in the fixed-length region, the offset of variable-length data also needs to be corrected using an algorithm. The following process is required during data decoding: Figure 2 As shown:
[0122] UnsafeRow's data reading interface uses methods such as getInt(int index), getLong(int index), and getUTF8String(int index), and requires reading data according to its data type. The parameter is the column index of the field.
[0123] 1) Calculate the fixed-length range offset by index: Since the compressed UnsafeRow only stores the values of non-null fields in the fixed-length range, it is not possible to directly access the data in the fixed-length range using column subscripts. The following correction is required, and the method is as follows:
[0124] The number of all non-empty fields preceding the given field is calculated using the empty bitmap and used as the index of the fixed-length region.
[0125] For scenarios involving sequential access of multiple fields: read empty spaces from the previously read field, and calculate the number of non-empty fields before that field as the index of the fixed-length region.
[0126] 2) If the data type of the column being read is a non-variable value, read the data value using the subscript corrected in 1) and return it.
[0127] 3) For variable-length data types, use the corrected index from 1) to read the data's offset and length.
[0128] 4) Calculate the variable-length offset: Since the offset is recorded in an uncompressed format, the length of the compressed memory needs to be subtracted from the offset after compression.
[0129] After modification Number of empty fields
[0130] 5) Read data: Read data from the variable-length area using the corrected offset and return it.
[0131] 4. Serialization and deserialization methods for compressing UnsafeRow
[0132] a) Serialization:
[0133] i. Number of bytes written: Total number of bytes in the bitmap area, compressed fixed-length area, and variable-length area
[0134] ii. Write data: Includes all data in the bitmap area, compressed fixed-length area, and variable-length area.
[0135] iii. Write compression identifier: This identifier is used to distinguish between compressed and uncompressed scenarios to ensure compatibility.
[0136] iv. Number of empty fields written
[0137] b) Deserialization:
[0138] i. Read the total number of bytes and allocate memory based on the number of bytes.
[0139] ii. Read data: including all data in the bitmap area, compressed fixed-length area, and variable-length area.
[0140] iii. Read the compression flag: When this flag is true, decoding needs to be processed according to the compression format.
[0141] iv. Count the number of empty fields for decoding compressed data.
[0142] 5. ORC write optimization methods, such as... Figure 3 As shown,
[0143] 1) Writing Data: During data writing, the encoding and compression of in-memory data are completed according to the ORC process. When the memory size reaches the Stripe threshold, the in-memory data is written to the file as a whole, using the Stripe process. The optimized Stripe write process is as follows:
[0144] Determine if a column is entirely empty based on the data in each column's statistics. If a column is entirely empty, delete the rowgroup statistics for that column. Record the column ID of the empty column in the StripeFooter.
[0145] When writing to stripeIndex, only data in non-fully empty columns is written; data in fully empty columns and statistics are not written to disk.
[0146] After completing the writing of stripeIndex and RowData, update the non-empty column data and the length of the index in the footer, and write them to StripeFooter.
[0147] 2) When closing a file, it is necessary to write the file's metadata and file footer. The optimized steps are as follows:
[0148] Based on file-level statistics, determine if the data in a column is entirely empty; if so, delete the stripe for that column.
[0149] Statistical information, and record the ID of the empty column in the Footer file;
[0150] Add statistics for non-fully empty columns to the metadata and write them to a file.
[0151] Write to FileFooter;
[0152] Through the above optimizations, data and column statistics are no longer written to completely empty columns within the Stripe, and statistics for completely empty columns are no longer written to the file metadata, thus achieving data compression. Simultaneously, it also reduces the computational overhead of protobuf encoding and data compression.
[0153] 6. ORC read optimization methods
[0154] To optimize the process of no longer writing all null value columns and column statistics to files, the following optimizations are needed in the read process:
[0155] 1) Stripe filtering by predicate conditions: First, stripe statistics are read from the metadata. Then, based on the null value column ID in the footer, null value column statistics are inserted into the corresponding column. Multiple columns share the same null value column statistics. Then, stripe filtering is performed based on predicate conditions.
[0156] 2) Filter rowgroup by predicate condition: First, determine whether the predicate contains a completely empty column by checking the null value column ID in the stripe footer. For completely empty columns, construct completely empty statistics in memory as the rowgroup statistics for that column. For non-empty columns, read the column statistics from the file and then perform predicate filtering.
[0157] 3) When reading data: First, check if the corresponding column is completely empty by using the Stripe footer. If the column is completely empty in the stripe, return the data through the empty column reader (this reader does not read the file, but directly constructs an empty value and returns the data), and the file will not be accessed again for this column; for columns that are not completely empty, read the data from the file and return it.
[0158] Based on the above optimizations, filtering and data return without accessing files can be achieved for completely empty columns, which can reduce the overhead of data protobuf decoding and decompression.
[0159] Data is sorted in memory and written to a file, such as... Figure 4 As shown.
[0160] For tables where data storage requires sorting, multiple rows of data are cached in memory. When memory reaches its limit, the data is sorted uniformly and written to a file all at once. The specific steps are as follows:
[0161] New data should be compressed according to the encoding method of this scheme;
[0162] Write the compressed data into the memory cache;
[0163] If the memory cache reaches its limit, sort the data as required;
[0164] Write the sorted data to a file. The file types that can be written include: ORC and parquet.
[0165] In this scenario, processing data based on compressed UnsafeRow has the following effect (the table has 176 fields, with an average of 100 fields having values):
[0166] The average memory size of a single row of data is reduced by 40% compared to the original UnsafeRow.
[0167] The number of lines of data written to a file per operation increased by 40%, and the file size increased by 40%. Note:
[0168] With a smaller single-line data size, more data lines can be cached in the same amount of memory, and more lines can be written at once. Larger files are more conducive to file system management and data processing.
[0169] Batch data import:
[0170] In scenarios involving massive amounts of data entering the database, the data is processed according to the following process:
[0171] Map tasks read data from input files on HDFS, calculate the hash value of the partitioner (default is HashPartitioner) based on the user-specified grouping key, and determine the reducer partition where the data should be written. When the data in the memory buffer reaches the threshold, the data is spilled to a temporary file on the local disk and partitioned before being written.
[0172] Map: This is the main stage of batch data processing. It is responsible for reading the input data and splitting, transforming, and processing the data according to specific rules.
[0173] Reduce: Responsible for handling the computation tasks of the Map phase partitions, including sorting, merging, and writing data to files.
[0174] Tungsten is Spark's "high-performance computing engine," responsible for efficiently processing data, making Spark run faster and use less memory.
[0175] The Reduce task pulls data from the corresponding partitions of all Map tasks and completes the Shuffle process. During the pulling process, the data is sorted and merged by key, and finally written to the ORC file in HDFS to complete the data import.
[0176] The data in the above process is encoded and processed using the compressed UnsafeRow format.
[0177] Based on the compression technology of this solution, the effect of batch data import of 60 million rows in a data import scenario is as follows:
[0178] Shuffle data volume reduced by 40%, disk and network I / O reduced by 40%;
[0179] Data processing time reduced by 20%;
[0180] The key points of this invention are: UnsafeRow memory layout optimization and compression method and corresponding data encoding, decoding and serialization method; and ORC empty column storage optimization and read / write method.
[0181] The key to UnsafeRow's memory compression technology lies in reducing the memory usage of empty fields and the total memory usage of sparse data through memory layout optimization.
[0182] Encoding methods for compressing UnsafeRow: reusing the writer for multiple rows of data to avoid repeated memory allocation and deallocation; calculating the overall memory allocation based on the writer's memory and data to avoid multiple memory allocations and deallocations during writing; and a compressed writing method for fixed-length data.
[0183] Decoding methods for compressed UnsafeRows: a method for locating the position of fields in a fixed-length region based on an empty bitmap; and a method for recording the number of non-empty fields and correcting the offset in a variable-length region.
[0184] The serialization and deserialization method for compressed UnsafeRows is as follows: During serialization, a compression flag and the number of empty fields are written; during deserialization, the compression flag ensures compatibility with both compressed and uncompressed formats. The number of empty fields ensures correct data decoding.
[0185] ORC's method for compressing null columns is as follows: Statistics for all null columns are no longer written to the Stripe, but all null column IDs are written to the StripeFooter; simultaneously, Stripe statistics for all null columns are no longer written to the file-level statistics, but the null column IDs are written to the file Footer.
[0186] ORC's reading method based on null column compression: When performing stripe filtering, statistical information is padded based on the file-level null column IDs without reading the file, and then stripe filtering is performed; within the stripe, statistical information is padded based on the null column IDs without reading the file, and then rowgroup filtering is performed; when reading null columns, data is directly constructed and returned using null value readers.
[0187] Reduce data memory footprint: UnsafeRow reduces data memory footprint by compressing empty fields. This is especially effective for sparse data scenarios, significantly reducing memory usage.
[0188] Reduce memory allocation and deallocation to lower GC pressure: Reuse UnsafeRow Writer and memory for multiple rows to avoid multiple memory allocation and deallocation for each row of data, thus reducing GC pressure.
[0189] Reduce network and storage I / O: By using UnsafeRow data compression, the amount of data transmitted over the network and written to and read from the disk is reduced, thereby reducing network and storage I / O consumption.
[0190] ORC optimization reduces storage space: During ORC writes, statistics on completely empty columns are filtered at both the Stripe and file levels, preventing them from being written to the file and thus reducing the amount of data written to the file, thereby reducing storage space.
[0191] Reduce ORC read / write computational overhead: Writing and reading fully empty columns no longer process the statistical information of fully empty columns, which reduces the computational overhead of protobuf encoding and decoding, as well as data compression and decompression. Reading data from fully empty columns no longer involves reading the file, decompression, and decoding, further reducing the computational overhead of data reading.
[0192] Seamless Integration: This invention is fully compatible with UnsafeRow's write, read, serialization, and deserialization methods, enabling seamless integration with Apache Spark. ORC optimizations are fully compatible with existing ORC interfaces and do not affect integration with other software.
[0193] It will be understood by those skilled in the art that the above descriptions are merely preferred examples of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the invention should be included within the scope of protection of the invention. All technical features in this embodiment can be freely combined according to actual needs.
[0194] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for compressing sparse data, characterized in that: Specifically, it includes the following steps; Step 1, UnsafeRow data memory layout optimization: Optimize the memory layout of UnsafeRow. The optimized memory layout includes field empty bitmap, fixed-length area and variable-length area. UnsafeRow is the row data encoding format in Apache Spark, which is a major data transmission and processing format in Spark. Step 2, UnsafeRow data encoding; Step 3, UnsafeRow data decoding; Step 4: Compress the serialization and deserialization of UnsafeRow; Step 5, ORC write optimization; ORC is an efficient, high-performance columnar storage format; Step 6, ORC read optimization.
2. The sparse data compression method according to claim 1, characterized in that: In step 1, UnsafeRow data memory layout optimization specifically includes: Step 1.1: Empty plots and variable-length regions are compatible with native formats; Step 1.2: Reduce the memory footprint of sparse data by optimizing the memory layout of fixed-length regions and the encoding / decoding methods. Fixed-length areas only store the values, offsets, or lengths of non-empty fields; The system utilizes encoding and decoding methods to provide interface-compatible read and write capabilities. Step 1.3: Record the number of empty fields to calculate the offset and length of the fixed-length area.
3. The sparse data compression method according to claim 1, characterized in that: In step 2, the UnsafeRow data encoding is as follows: Step 2.1: Allocate Writers from the memory pool according to the total number of fields in the data to avoid repeatedly allocating and releasing memory. When writing a row of data, a writer is requested from the pool. If the memory required for the row of data is less than the memory already in the writer, the memory in the writer is reused; if the memory required for the data is greater than the memory already in the writer, new memory is requested, and this memory can be reused for subsequent writes. If multiple data fields in the same table are identical, multiple rows of data can share the memory in the writer, avoiding multiple allocations and releases; Step 2.2, write the field value: Field value is empty: Set the corresponding field in the empty bitmap to empty and increment the number of empty fields by 1; Field value NOT null: Sets the corresponding field in the empty bitmap to NOT null. Fixed-length values: When a field is a fixed-length value such as bool, int, or long, the value of the corresponding field is written into the corresponding field in the fixed-length area; Non-fixed-length values: When the field value is a non-fixed-length value such as string or decimal, the data is first written to the variable-length area, and then the offset and length are written to the fixed-length area. Step 2.3, Generate Compressed UnsafeRow: After all fields have been written, compressed data is generated using the following algorithm: Calculate the compressed memory size: Memory size = data length in Writer - empty field data * 8; Allocate memory for compressed data, and allocate memory for the data in line a) as a data buffer. Encode the data as follows: copy the empty bitmap from the writer to the buffer; Writing to the fixed-length area: First, set the offset of the fixed-length area to 0, and then process it according to the field order as follows: If the value is not non-empty, write the 8-byte value of the field to the memory corresponding to the offset, and increment the offset by 8; if the value is empty, do not process it; copy the data of the writer's variable-length area to the variable-length buffer. Step 2.4, Release Writer: Return the writer to the pool for subsequent writing; Step 2.5: Return the compressed UnsafeRow.
4. The sparse data compression method according to claim 1, characterized in that: In step 3, the decoding of UnsafeRow data specifically includes: Step 3.1, Calculate the fixed-length area offset by index: Since the compressed UnsafeRow only stores the values of non-null fields in the fixed-length area, the data in the fixed-length area cannot be accessed directly using column subscripts. The following correction is required, and the method is as follows: The number of all non-empty fields preceding the given field is calculated using the empty bitmap and used as the index of the fixed-length region. For scenarios involving sequential access of multiple fields: read empty spaces from the previously read field, and calculate the number of non-empty fields before that field as the index of the fixed-length area; Step 3.2: If the data type of the column being read is a non-variable length value, read the value of the data using the subscript corrected in 1), and return it; Step 3.3: For variable-length data types, read the offset and length of the data using the index corrected in Step 3.1; Step 3.4, Calculate the variable-length offset: Since the offset is recorded in an uncompressed format, it needs to be extracted from the offset after compression. Subtract the length of the compressed memory; after modification, offset = offset - 8 * number of empty fields; Step 3.5, Read Data: Read data from the variable-length area using the corrected offset and return it.
5. The sparse data compression method according to claim 1, characterized in that: In step 4, the serialization and deserialization of compressed UnsafeRow are as follows: Among them, (a) serialization: Number of bytes written: Total number of bytes in the bitmap area, compressed fixed-length area, and variable-length area; Write data: Includes all data in the bitmap area, compressed fixed-length area, and variable-length area; Write compression identifier: This identifier is used to distinguish between compressed and uncompressed scenarios to ensure compatibility. Number of empty fields written; (b) Deserialization: Read the total number of bytes and allocate memory based on that number of bytes; Read data: including all data in the bitmap area, compressed fixed-length area, and variable-length area; Read compression flag: When this flag is true, decoding needs to be processed according to the compression format; Read the number of empty fields for use in decoding compressed data.
6. The sparse data compression method according to claim 1, characterized in that: In step 5, ORC write optimization specifically includes the following steps; Step 5.1, Writing Data: When writing data, the encoding and compression of the data in memory are completed according to the ORC process; when the memory size reaches the Stripe threshold, the memory data is written to the file as a whole according to the stripe. The optimized Stripe writing process is as follows: Determine if a column is completely empty based on the data in each column's statistics. If the column is completely empty, delete the rowgroup statistics for that column and record the column ID of the empty column in the StripeFooter. When writing to stripeIndex, only data in non-fully empty columns is written; data in fully empty columns and statistical information are not written to disk. After completing the writing of stripeIndex and RowData, update the non-empty column data and the length of the index in the footer, and write to StripeFooter; Step 5.2, when closing a file, it is necessary to write the file's metadata and file footer. The optimized steps are as follows: Based on file-level statistics, determine if the data in a column is completely empty. If it is, delete the Stripe statistics for that column and record the ID of the empty column in the Footer file. Add statistics for non-fully empty columns to the metadata and write them to a file; Write to FileFooter.
7. The sparse data compression method according to claim 1, characterized in that: In step 6, ORC read optimization specifically includes the following steps; Step 6.1, Filter Stripe by Predicate Conditions: First, read stripe statistics from the metadata, then insert null value column statistics into the corresponding column based on the null value column ID in the footer, with multiple columns sharing one null value column statistics; then perform stripe filtering based on predicate conditions. Step 6.2, filter rowgroups by predicate condition: First, determine whether the predicate contains a completely empty column by checking the null value column ID in the stripe footer. For completely empty columns, construct completely empty statistics in memory as the rowgroup statistics for that column. For non-empty columns, read the column statistics from the file and then perform predicate filtering. Step 6.3, when reading data: First, check if the corresponding column is completely empty by checking the Stripe footer. If the column is completely empty in the stripe, use the empty column reader. This reader does not read the file, directly constructs an empty value and returns the data. After returning the data, the column will no longer access the file. For columns that are not completely empty, read the data from the file and return it.