A method for batch processing of big data

Abstract

This invention specifically relates to a method for batch processing of large data, belonging to the field of big data processing technology. It includes: allocating a thread-local aggregation buffer aligned with the processor cache line size in an off-heap contiguous memory pool; and using the final selection vector as a mask, directly utilizing its data as a physical memory addressing index or calculation factor without expanding the compressed data payload area. In this invention, an adaptive coding decision engine matches the distribution characteristics of different data to generate compressed data blocks that support direct logical operations and numerical comparisons by the CPU. Core computational operations such as filtering, grouping, and aggregation are completed entirely in compressed state, with targeted delayed decompression performed only at the very last moment of the final result output. This design reduces the memory footprint and memory bus bandwidth consumption throughout the computational chain, avoiding the memory bloat problem caused by full data decompression.

Description

Technical Field

[0001] This invention relates to the field of big data processing technology, and in particular to a method for batch processing of big data. Background Technology

[0002] Currently, mainstream big data distributed processing frameworks and columnar storage technologies have formed mature technical systems, providing a solid foundation for distributed storage and parallel computing of massive amounts of structured and semi-structured data. The widespread application of various lightweight data compression and encoding schemes has also effectively reduced the storage costs of massive data and improved the overall efficiency of data transmission and processing. At the same time, enterprises are continuously increasing their requirements for low latency, high stability, and high resource utilization in batch processing of massive data. How to further optimize the execution efficiency of the entire data processing chain and reduce resource overhead during computation under limited hardware resources has become a core direction for continuous exploration in the field of big data processing technology.

[0003] At the level of collaboration between data compression and computation execution, existing mainstream compression coding technologies are primarily designed around optimizing data storage compression rates, resulting in significant technological accumulation and practical achievements in reducing storage footprint. However, in batch processing scenarios involving massive amounts of data, the common "decompress first, then compute" execution model requires restoring the entire compressed data to plaintext before performing filtering and comparison operations. This process significantly increases memory usage in the computation chain under ultra-large-scale data scenarios, placing high demands on memory bus bandwidth. Furthermore, the separation of data decompression and computation also lengthens the entire data processing cycle. In addition, existing common data storage layouts and null value handling methods still have room for further refinement in terms of adapting to CPU cache characteristics and simplifying computation execution instructions. Therefore, achieving deep collaboration between compression coding and computation execution becomes a key entry point for improving batch processing performance.

[0004] At the level of parallel computing and resource management, existing mainstream big data processing frameworks have built mature technical systems for multi-core concurrency and distributed task scheduling, capable of supporting the execution of parallel computing tasks on large-scale clusters. In high-concurrency batch processing scenarios with ultra-large-scale data, the hash table implementation commonly used for existing grouping and aggregation operations suffers from fixed execution overhead due to hash calculations and conflict handling. The discrete memory layout also impacts CPU cache utilization. Furthermore, issues such as global resource contention and false CPU cache sharing in multi-core concurrency scenarios also affect the linear scalability of multi-core hardware performance. Meanwhile, processing frameworks built on runtime environments with automatic garbage collection mechanisms are prone to processing latency fluctuations due to memory management issues during the processing of massive amounts of temporary data. Further optimizing computational execution efficiency, improving hardware resource utilization, and ensuring the stable operation of batch processing tasks remain key areas for continuous improvement within the industry. Summary of the Invention

[0005] The purpose of this invention is to provide a method for batch processing of large data in order to solve the above-mentioned problems.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: A method for batch processing of big data includes: The original data stream is acquired and parsed in a row-to-column manner in a contiguous off-heap memory pool. A lightweight encoding strategy is dynamically selected using a reservoir sampling algorithm to convert the original data stream into compressed data blocks, which include a metadata area, a null bitmap area, and a compressed data payload area. Receive query requests, extract metadata area information, rewrite the filtering predicate into underlying comparison logic for the compressed data payload area; call the instruction set to perform vectorized scanning to generate a matching bitmap without decompressing the compressed data payload area, and perform a bitwise AND operation between the matching bitmap and the null bitmap area to generate the final selection vector; In the off-heap contiguous memory pool, allocate a thread-local aggregation buffer aligned with the processor cache line size; using the final selection vector as a mask, without expanding the compressed data payload area, directly use its data as the physical memory address index or calculation factor, perform lock-free concurrent aggregation in the buffer, and obtain the intermediate aggregation result through global merging; In response to the global memory barrier instruction, the intermediate aggregation result is reverse mapped or compensated based on the metadata area information, and the plaintext result data is decompressed and restored with delay. The plaintext result data is assembled into a columnar memory layout and output with zero copy. Then, the global allocation pointer of the off-heap contiguous memory pool is reset to complete the memory reclamation.

[0007] Preferably, the row-to-column parsing in the off-heap contiguous memory pool includes: For each column of data, a continuous bit space is allocated as a null bitmap area; when non-null data is parsed, the corresponding bit position is set to the first state, and the real data is continuously written into the actual data storage area; when null value is parsed, the corresponding bit position is set to the second state, and the placeholder is not written into the actual data storage area.

[0008] Preferably, the step of dynamically selecting a lightweight coding strategy using a reservoir sampling algorithm includes: Sample data were extracted using a reservoir sampling algorithm, and the cardinality, mean run length, and range of the sample data were calculated. If the base number is lower than the preset base number ratio, then dictionary encoding is selected; If the average run length is greater than the preset length threshold, then run coding is selected; If the data is numerical and the range is less than the preset range threshold, then differential encoding combined with bit packing is selected.

[0009] Preferably, the process of converting the raw data stream into compressed data blocks includes: Construct an order-preserving dictionary table, sort the extracted unique plaintext values ​​in lexicographical order and assign them incrementing integer identifiers; The minimum bit width is dynamically calculated based on the total number of unique plaintext values. The original plaintext value is replaced with the corresponding integer identifier, and the integer identifier is concatenated and written into a contiguous memory space using bit shift and bitwise OR operations, with the minimum bit width.

[0010] Preferably, rewriting the filtering predicate into low-level comparison logic for the compressed data payload region includes: The target value is searched in the order-preserving dictionary table contained in the metadata area. If the target value is not found or is out of range, the early blocking mechanism is triggered, and a matching bitmap with all values ​​in the second state is directly generated and the processing of the current compressed data block is skipped. If the target value is found, the comparison logic for variable-length strings is rewritten as the comparison logic for fixed-width integer identifiers.

[0011] Preferably, the calling instruction set performs vectorized scanning to generate a matching bitmap without decompressing the compressed data payload area, including: The rewritten target value is broadcast to the processor's vector register; By using direct memory access pointers, a continuous stream of compressed data is loaded from the compressed data payload area into another vector register at once; The single instruction multiple data stream comparison instruction is invoked to perform concurrent comparisons. The result mask is directly generated by the hardware, and the result mask is concatenated in sequence to form a matching bitmap.

[0012] Preferably, when the compressed data payload area uses run-length encoding, the process of generating a matching bitmap includes: Read the current value and counter in the run-length pair; Compare the current value with the filter criteria: If there is no match, the pointer of the matching bitmap is jumped forward by the corresponding length in memory according to the value of the counter, and the bits are initialized to the second state in batches. If a match is found, the corresponding number of bits are sequentially set to the first state based on the counter value.

[0013] Preferably, when performing grouped aggregation and the data is dictionary-encoded, performing lock-free concurrent aggregation in the buffer includes: A fixed-length contiguous array is pre-allocated in memory as a thread-local aggregation buffer. The length of the contiguous array is equal to the number of unique values ​​in the dictionary. The read compressed integer identifier is directly used as the physical index of the continuous array. The corresponding slot is located through the base address addressing mode and the accumulation or summation operation is performed.

[0014] Preferably, when performing aggregation and the data uses run-length encoding, performing lock-free concurrent aggregation in the buffer includes: Read the run-length pair and perform an interval intersection operation between the physical row range covered by the run-length pair and the final selection vector; Use bitwise operations to count the number of bits in the first state in the final selection vector; The number of bits is multiplied by mathematical multiplication and the current value in the run-length pair. The product is then accumulated into the thread-local aggregation buffer all at once.

[0015] Preferably, the reverse mapping of the intermediate aggregation result based on metadata area information includes: The search is performed using a double-array contiguous memory layout, which includes an offset array that stores memory address offsets and a character payload array that tightly concatenates plaintext strings. The integer identifier to be restored is used as the physical index to access the offset array, and the exact length of the target string is calculated by subtracting the offsets of adjacent slots. The absolute physical memory address of the character payload array is located using the base pointer and the starting offset, and the plaintext result data is extracted using a memory copy instruction.

[0016] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: 1. This invention uses an adaptive coding decision engine to match the distribution characteristics of different data and generate compressed data blocks that support direct logical operations and numerical comparisons by the CPU. Core computational operations such as filtering, grouping, and aggregation are completed entirely in the compressed state, with targeted delayed decompression performed only at the last moment of the final result output. This design reduces the memory usage and memory bus bandwidth consumption of the entire computation chain and avoids the memory bloat problem caused by full data decompression.

[0017] 2. This invention achieves pre-allocation and fine-grained management of memory throughout its entire lifecycle through a perceptive off-heap memory pool. There is no dynamic memory allocation or temporary object creation throughout the process. Once a batch processing task ends, memory reuse is achieved simply by resetting the allocation pointer, avoiding processing latency fluctuations caused by garbage collection and ensuring high task stability. Simultaneously, the thread-local aggregation buffer and cache line alignment design eliminate lock contention and CPU cache pseudo-sharing issues in concurrent computing. Array-level grouping based on dictionary IDs replaces the traditional hash table scheme, significantly reducing fixed computational overhead, improving CPU cache utilization, and achieving linear scaling of multi-core CPU performance. Attached Figure Description

[0018] Further details, features, and advantages of this application are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which: Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0019] Several embodiments of this application will now be described in more detail with reference to the accompanying drawings to enable those skilled in the art to implement this application. This application may be embodied in many different forms and for various purposes and should not be limited to the embodiments set forth herein. These embodiments are provided to make this application thorough and complete, and to fully convey the scope of this application to those skilled in the art. The embodiments described do not limit this application.

[0020] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It will be further understood that terms such as those defined in commonly used dictionaries shall be interpreted as having a meaning consistent with their meaning in the relevant field and / or the context of this specification, and shall not be interpreted in an idealized or overly formal sense unless expressly defined herein.

[0021] Example 1

[0022] Its specific implementation method is combined with the appendix Figure 1 Please provide a detailed explanation.

[0023] Appendix Figure 1The flowchart of a big data batch processing method provided in this embodiment of the invention illustrates the complete steps from obtaining the original data stream to resetting the global allocation pointer of the off-heap contiguous memory pool to complete memory reclamation.

[0024] In this embodiment, it includes: Step 1: Obtain the raw data stream and perform row-to-column parsing in the off-heap contiguous memory pool; use the reservoir sampling algorithm to dynamically select a lightweight encoding strategy to convert the raw data stream into compressed data blocks, which include metadata area, null bitmap area and compressed data payload area; The core technical objective of this step is to transform externally accessed, unordered, row-based raw plaintext data into a highly structured, column-gathered "lightweight compressed data block" with self-describing metadata. This compressed data block must meet a stringent condition: in subsequent computations, the CPU can directly perform logical operations on the compressed binary bits without having to restore them to plaintext.

[0025] To achieve this goal, step one is strictly divided into the following seven sub-steps: Sub-step 1.1: Initialization of the NUMA-aware off-heap memory pool: To avoid garbage collection (GC) pauses, a common problem in big data batch processing, this solution first builds a customized off-heap memory pool at the operating system level.

[0026] During system startup, the memory management module initiates a system call (such as the mmap function in Linux) to the operating system kernel to request a large, contiguous block of physical memory. During this request, the system reads the current server's NUMA node topology. For each NUMA node, the system allocates a dedicated memory pool. This means that when a CPU core needs to process data, the system will force the allocation of that data to the memory module physically closest to that CPU core, thereby eliminating bus bandwidth bottlenecks and latency caused by cross-NUMA node access.

[0027] The allocated contiguous large blocks of memory are further divided into fixed-size "memory pages". To achieve optimal compatibility with the underlying hardware, the size of the memory page is strictly set to an integer multiple of the CPU L1 / L2 cache line size (typically a cache line is 64 bytes, but this solution allows setting the basic unit of the memory page to 4096 bytes or larger). Each memory page is assigned a globally unique memory address pointer. All subsequent data ingestion, encoding, and packaging will be performed directly through these pre-allocated memory pages using pointer offsets, without triggering any dynamic memory allocation (such as malloc or new operations), thus ensuring extreme ingestion performance.

[0028] Sub-step 1.2: Row-to-column transformation and null value removal of the original data: When external data (such as JSON-formatted logs, CSV files, or change data capture streams from relational databases) enters the system, it first enters the "row-to-column parser".

[0029] The parser reads the raw data line by line and, according to a preset data pattern, breaks each line down into multiple independent field values. At this point, the system does not immediately encode the data; instead, it continuously writes data belonging to the same field (column) into specific memory pages allocated in sub-step 1.1. This process is called "data shredding."

[0030] During the row-to-column transformation process, this scheme introduces a crucial feature: the physical removal of null values. In traditional databases, null values ​​are often mixed into the real data using special placeholders (such as specific maximum or minimum values), which severely disrupts the continuity of subsequent encoding and compression ratio.

[0031] This scheme uses a separate "null bitmap" to manage null values. Specifically, for each column of data, the system allocates a contiguous block of bits in memory. When a row of data in that column is parsed to be non-null, the corresponding bit is set to "1"; when the data is parsed to be null, the corresponding bit is set to "0", and no placeholders are ever written to the actual data storage area of ​​that column.

[0032] Through this physical stripping, the actual data storage area becomes an absolutely compact, continuous sequence of values ​​without any gaps. This not only greatly improves the efficiency of subsequent run-length encoding or differential encoding, but also allows the CPU to instantly filter out all rows containing null values ​​with a simple bitwise AND instruction in subsequent batch processing, without performing any complex conditional branch judgments, thus avoiding CPU pipeline flushing.

[0033] Sub-step 1.3: Adaptive coding decision engine based on reservoir sampling: When the data in a certain column accumulates in memory pages to a preset threshold (e.g., reaching one million rows, or occupying a certain number of megabytes of memory), the system will trigger the "adaptive encoding decision engine." The engine's function is to dynamically select the most suitable lightweight encoding algorithm for the current batch of column data.

[0034] To make decisions in a very short time, the system cannot perform a full scan of this million rows of data. Therefore, this solution uses a "reservoir sampling algorithm" for unbiased sampling. The system randomly selects a fixed number of samples (e.g., 8,192) from the current data block.

[0035] For this batch of sample data, the decision engine will calculate three core statistical indicators: First, cardinality, which is the number of distinct unique values ​​in a sample. The system quickly calculates cardinality by constructing a lightweight hash set.

[0036] Second, the average run length, which is the average number of consecutive occurrences of the same value in the sample. The system calculates this metric by traversing the sample and recording the length of consecutive sequences of adjacent elements with the same value.

[0037] Third, numerical range, which is only applicable to numerical data, calculates the difference between the maximum and minimum values ​​in the sample.

[0038] Based on these three metrics, the decision engine assigns coding strategies using a strict set of deterministic rule trees: Rule 1: If the calculated cardinality is extremely low (e.g., the number of unique values ​​accounts for less than 5% of the total sample), then the column is determined to be a low cardinality column (e.g., gender, province, status code), and the system will forcibly assign a "dictionary code".

[0039] Rule 2: If the calculated average run length is greater than a preset threshold (e.g., the average number of consecutive repetitions is greater than four), then the column is determined to have strong local continuity (e.g., the online status of devices sorted by time), and the system will forcibly assign a "run code".

[0040] Rule 3: If the column contains numerical data with a high cardinality but a small range (e.g., continuously collected temperature sensor data or auto-incrementing primary key ID), the system will forcibly assign "differential encoding combined bit packing".

[0041] Rule 4: If none of the above conditions are met (e.g., a random UUID string), the system will abandon lightweight encoding of the data block and instead directly store the plaintext or degenerate into regular block-level compression (such as LZ4), and mark it as "unencoded" in the metadata.

[0042] Sub-step 1.4: The micro-construction process of dictionary encoding: Once the decision engine selects a dictionary encoding for a certain column of data, the system will enter an extremely sophisticated dictionary construction and data replacement process.

[0043] The first step is the construction of the dictionary table. The system iterates through all plaintext data in the column and extracts all unique values. To ensure the speed of subsequent calculations, the dictionary table in this scheme is not a simple hash mapping, but rather adopts an "order-preserving dictionary" structure. The system strictly sorts all extracted unique plaintext values ​​(e.g., city names in string type) in memory in lexicographical order. After sorting, the system assigns an integer identifier to each plaintext value, starting from zero and incrementing. Because it is an order-preserving allocation, the size relationship between plaintext values ​​is perfectly mapped to the integer IDs (for example, if the ID of the plaintext "Apple" is 0 and the ID of "Banana" is 1, then in subsequent calculations, judging "Apple" < "Banana" is directly equivalent to judging 0 < 1).

[0044] Secondly, there's the extreme compression of bit width. The system dynamically calculates the minimum number of bits required to store these integer IDs based on the total number of unique values ​​in the dictionary. For example, if the column only has sixty different cities, then only six bits (2 to the power of 6 equals 64, enough to cover 60) are needed to represent all the IDs, instead of using a standard 32-bit integer. The system records this calculated "minimum bit width" in the metadata.

[0045] Finally, plaintext replacement and bit concatenation are performed. The system iterates through the original data in the column again, replacing each plaintext value with its corresponding integer ID by looking up the dictionary. Then, using bitwise shift and bitwise OR operations, the system tightly concatenates these non-standard width integer IDs (e.g., consecutive six-bit data) together and writes them into contiguous memory space, eliminating the space wastage caused by byte alignment. At this point, the massive string column is compressed into an extremely compact binary bit stream, accompanied by a very small, order-preserving dictionary.

[0046] Sub-step 1.5: The micro-construction process of run-length encoding: When the decision engine chooses run-length encoding, the system aims to compress continuously repeating redundant data into a very simple mathematical expression.

[0047] The system initializes two pointers: a "current value pointer" and a "counter". The system scans starting from the first data item in the column, recording the first data item as the "current value" and setting the counter to one. Subsequently, the pointer moves forward. If the next data item read is the same as the "current value" (represented as identical binary content in memory), the counter is incremented; if the next data item read is different from the "current value", the system writes the current "current value" and the "counter" as a "run pair" to the target memory area, updates the "current value" with the new data, and resets the counter to one.

[0048] To further optimize storage space, this solution employs a unique physical storage structure for the "run-length pairs." For the "current value" portion, the system directly stores its original binary content. However, for the "counter" portion, since the number of repetitions can range from extremely small (e.g., twice) to extremely large (e.g., one million times), uniformly using a 64-bit integer to store the counter would result in significant space waste. Therefore, this solution utilizes variable-length integer technology to store the counter.

[0049] The core logic of VarInt technology is to divide the binary representation of an integer into groups of seven bits. For each group, the system adds a "continuation bit" to its highest bit. If the continuation bit is "1", it indicates that the next byte still belongs to the current integer; if the continuation bit is "0", it indicates that the current byte is the last byte of the integer. Through this fine-grained bit-level control, for a small number of repetitions, the system only needs one byte to complete the storage, greatly improving the compression limit of run-length encoding under complex data distributions.

[0050] Sub-step 1.6: Microscopic construction process of differential coding and bit packing: For numerical data that changes gradually, the system uses differential coding combined with bit packing technology.

[0051] First, the system scans the current data block, extracts the minimum value in the block, defines it as the "baseline value", and records it in the metadata header of the data block.

[0052] Next, the system iterates through each value in the block, subtracts the "baseline value" from that value, and obtains a "difference". Because the data changes gradually, these differences Typically, these are very small positive integers. The system transforms the original, massive numerical sequence into a very small sequence of differences.

[0053] The system then scans this sequence of differences to find the largest difference. Based on this largest difference, the system calculates the minimum bit width required to represent that difference (for example, if the largest difference is fifteen, then the minimum bit width is four bits).

[0054] Finally, the system enters the bit packing stage. The system seamlessly packs all differences into a 64-bit machine word using low-level shift instructions (such as left shift << and right shift >>), strictly adhering to the calculated minimum bit width. For example, a 64-bit machine word can perfectly fit sixteen four-bit differences. This packing method not only achieves extremely high compression rates, but more importantly, it ensures that the data arrangement in memory is strictly aligned with the CPU's word length, laying a perfect physical foundation for subsequent steps using SIMD instructions for concurrent unpacking and computation.

[0055] Sub-step 1.7: Final encapsulation and immutability solidification of compressed data blocks After any of the above encoding processes, the system needs to assemble the encoded data, null bitmap, and related metadata into a standard, self-contained "compressed data block." This data block is the smallest physical unit for subsequent batch processing computations.

[0056] This scheme designs a binary file header structure. The beginning of a data block is a four-byte "magic number" used for quick verification of the data block's validity. This is followed by a two-byte version number, used for compatibility with future encoding algorithm upgrades.

[0057] Next is the metadata area. The metadata area strictly records the following information according to fixed byte offsets: the total number of rows in the current data block (four-byte integer), the encoding type identifier (a single-byte enumeration value, such as 0x01 for dictionary encoding, 0x02 for run-length encoding), the relative byte offset of the null bitmap within the current data block (four-byte integer), and the relative byte offset of the actual compressed data body within the current data block (four-byte integer). If dictionary encoding is used, the metadata area will also include the length of the dictionary table and the serialized byte stream of the dictionary table itself; if differential encoding is used, it will record the base value and bit width information.

[0058] Following the metadata area is the null bitmap area, and finally the compressed data payload area.

[0059] Once all data has been written to the memory page according to the strict physical layout described above, the system executes a crucial operating system-level instruction: setting the access permission of the memory page to "read-only". This step grants the compressed data block "immutability". Immutability is an extremely important characteristic in distributed big data processing. It means that in subsequent multi-threaded concurrent computations, any thread can safely read the data block without needing to add any mutex locks or read-write locks, thus fundamentally eliminating the lock contention overhead in concurrent computations.

[0060] At this point, the original plaintext data has been reshaped into "compressed data blocks" with self-describing metadata, an extremely compact physical structure, and excellent CPU cache friendliness. These data blocks will be pushed in batches to a distributed file system or reside directly in memory, awaiting direct invocation by subsequent computation operators.

[0061] Step 2: Receive the query request, extract the metadata area information, rewrite the filtering predicate to the underlying comparison logic for the compressed data payload area; call the instruction set to perform vectorized scanning to generate a matching bitmap without decompressing the compressed data payload area, and perform a bitwise AND operation between the matching bitmap and the null bitmap area to generate the final selection vector; In batch processing tasks, filtering operations (i.e., the WHERE clause in SQL) are usually the first hurdle in the data processing pipeline. The core technical objective of this step is to accurately locate the data rows that meet the business conditions and generate a high-density "matching bitmap" by manipulating only metadata and a very small number of bits, without touching or decompressing the underlying massive "compressed data payload area".

[0062] This step is divided into the following five sub-steps: Sub-step 2.1: Abstract Syntax Tree (AST) parsing and "lexicographical rewriting" of filtering conditions When the system receives a batch query request from the upper-layer business (e.g., the query condition is "city equals 'Beijing' and status code is greater than '200'"), the query optimizer first parses the request into an abstract syntax tree (AST). The system traverses the syntax tree and extracts all the leaf nodes, which are the specific filtering predicates.

[0063] At this point, the system will absolutely not read the massive data volume, but will first read the header metadata area of ​​the "compressed data block" generated in step one. Based on the offset in the metadata, the system loads a very small "dictionary table" into the CPU's L1 cache.

[0064] Next, the system performs a crucial "condition rewriting" operation. Taking "city equals 'Beijing'" as an example, the system performs a binary search on the string "Beijing" in the ordered dictionary.

[0065] There are two extreme optimization branches here: Branch 1 (Early Exit): If the string "Beijing" is not found in the dictionary, or if the target value exceeds the maximum / minimum range of the dictionary during a range query (e.g., "time greater than a certain day"), the system will directly determine that no data in the current compressed data block meets the condition. In this case, the system directly generates an empty bitmap filled with "0"s and immediately skips all subsequent processing of the current data block. This "Early Exit" mechanism can save hundreds or thousands of CPU clock cycles when processing massive amounts of data.

[0066] Branch 2 (Precise Rewrite): If "Beijing" is found in the dictionary, assuming its corresponding integer identifier is "05", the system will immediately modify the abstract syntax tree, rewriting the complex comparison logic (City=='Beijing') for variable-length strings into a very simple comparison logic (ID==05) for fixed-width integers in the underlying physical execution plan.

[0067] Sub-step 2.2: Vectorized bitmap scanning based on SIMD instruction set (for dictionary encoding) After conditional rewriting is completed, the system needs to find all rows in the data block where the ID is equal to "05". Since all IDs have been extremely compressed and tightly concatenated in step one (e.g., each ID occupies only 5 bits), the traditional method of reading and comparing them one by one will lead to serious CPU branch prediction failures and bit shifting overhead.

[0068] To address this, this solution introduces a single instruction multiple data (SIMD) vectorized scanning technique based on the underlying hardware.

[0069] The system first broadcasts the target value "05" to the CPU's vector register (such as a 512-bit wide register that supports the AVX-512 instruction set). This means that the target value "05" is copied multiple times in the 512-bit register, filling the entire register.

[0070] Subsequently, the system loads a continuous 512-bit compressed data from the "compressed data payload area" into another vector register at once via a direct memory access pointer.

[0071] Next, the system calls a low-level SIMD comparison instruction (such as a variant of _mm512_cmpeq_epi8, which performs mask adjustment for a custom bit width). Within a single CPU clock cycle, multiple arithmetic logic units (ALUs) inside the CPU concurrently compare dozens of IDs in this 512-bit data.

[0072] The comparison does not output specific row numbers; instead, the hardware directly generates a "result mask." In this mask, if the ID at a certain position equals "05," the corresponding bit is automatically set to "1" by the hardware; otherwise, it is set to "0." The system concatenates these hardware-generated masks in sequence to form a preliminary "matching bitmap."

[0073] Sub-step 2.3: Rapid filtering based on run-length jumps: If the metadata indicates that the current column uses "run-length encoding (RLE)," the logic of the filtering operation will be radically changed, and its time complexity will plummet from O(N) (where N is the number of data rows) to O(R) (where R is the number of runs).

[0074] The system reads the first "run-pair" of the run-length encoding, which contains the "current value" and the "counter (number of repetitions)"; the system then compares the "current value" with the filtering criteria.

[0075] If the comparison result is a mismatch: the system does not need to process line by line, but directly reads the value of the "counter" (let's say it's 1000), and then directly jumps the pointer of the "match bitmap" 1000 bits forward in memory, and initializes these 1000 bits to "0" in batch (usually implemented through the underlying memset instruction, which is extremely fast).

[0076] If the comparison result is a match: the system also reads the value of the "counter" (assuming it is 1000), and then in the "match bitmap", starting from the current pointer position, sets 1000 bits to "1" in batch.

[0077] Through this "run-skip" mechanism, even if a data block contains hundreds of millions of rows of data, as long as its physical arrangement is continuous, the system can complete the filtering of the entire data block in microseconds with just a few simple comparisons and batch assignments in memory.

[0078] Sub-step 2.4: Comparison of dense states based on baseline offset: For numerical data that uses differential encoding and bit packing, the system also does not need to unpack it to restore it to the original large value.

[0079] The system first reads the "baseline value" from the metadata. Suppose the filter condition is "find values ​​greater than 1000", and the baseline value of the current data block is "900".

[0080] Before performing the comparison, the system first performs a "reverse offset calculation" on the filtering conditions: subtracting the baseline value of 900 from the target value of 1000 to obtain the difference threshold "100".

[0081] At this point, the filter condition is rewritten as: "In the compressed load region, look for packing differences greater than 100".

[0082] Subsequently, the system again utilizes SIMD vectorization technology to load the continuously packed difference data into a vector register, directly performing concurrent vectorized greater than comparisons with the threshold "100", and outputting a matching bitmap. Throughout the entire process, the data remains in the packed "dense state," greatly saving memory bandwidth.

[0083] Sub-step 2.5: Bitwise AND merging and null removal of bitmaps In complex business scenarios, filtering conditions often involve multiple columns (e.g., condition A AND condition B).

[0084] The advantage of this approach is that the system generates an independent "matching bitmap" for each column's filtering. Once all columns have been filtered, the system does not require any complex row-level data association.

[0085] The system simply loads these individual bitmaps into the CPU registers and performs a basic bitwise AND operation. Only when the corresponding bit in all the bitmaps is "1" will the final result be "1".

[0086] More importantly, at this step, the system forcibly introduces the "null bitmap" generated in step one. The system then performs another bitwise AND operation with the null bitmap after merging the above bitmap. This step physically eliminates all dirty data rows containing null values ​​(NULL), ensuring the absolute accuracy of subsequent aggregation calculations.

[0087] The resulting bitmap is called the "final selection vector". It records the filtering results with extremely high information density (each bit represents the retention status of a row of data) and is passed to the next computation operator.

[0088] Step 3: In the off-heap contiguous memory pool, allocate a thread-local aggregation buffer aligned with the processor cache line size; using the final selection vector as a mask, without expanding the compressed data payload area, directly use its data as the physical memory address index or calculation factor, perform lock-free concurrent aggregation in the buffer, and obtain the intermediate aggregation result through global merging; Step two yields a "final selection vector" indicating which rows need to be processed. The core of step three is to use this selection vector as a mask to perform grouping and statistical operations directly on the compressed binary data without decompressing the original data, and to resolve lock contention issues during multi-threaded concurrent aggregation.

[0089] Sub-step 3.1: Masking based on selection vector In traditional batch processing engines, filtered data is copied to a new memory area before aggregation. This memory copying is a fatal performance bottleneck for massive amounts of data.

[0090] This solution employs a "mask execution" mechanism to achieve zero-copy aggregation.

[0091] When the aggregation operator is ready to read compressed data, it must simultaneously receive the "final selection vector". The aggregation operator reads the bits in the selection vector synchronously as it traverses the compressed data payload area.

[0092] The system utilizes the CPU's conditional transfer instructions or masked load instructions from the SIMD instruction set. Only when the corresponding bit in the selection vector is "1" will the CPU load the compressed data corresponding to that row into the Arithmetic Logic Unit (ALU) for aggregation calculations; if the bit is "0", the CPU hardware automatically masks the loading of that data and skips it. This mechanism avoids writing numerous if (match) conditional statements at the code level, thus ensuring the absolute smoothness of the CPU instruction pipeline.

[0093] Sub-step 3.2: Array-level grouping based on dictionary IDs: In regular SQL execution, the GROUPBY operation typically relies on building a large hash table. The system needs to calculate the hash value for each string, handle hash collisions, and dynamically allocate linked lists or open address space in memory. This not only consumes a lot of CPU computing power but also leads to serious CPU cache misses.

[0094] If the grouping column uses "dictionary encoding", then in step one, all grouping keys (such as city names) have been mapped to integer IDs that increment continuously from zero (e.g., 0 to 59).

[0095] When performing grouping, the system discards the hash table and instead pre-allocates a fixed-length contiguous array in memory as an aggregation buffer. The length of the array is strictly equal to the number of unique values ​​in the dictionary (e.g., a length of 60).

[0096] When the system reads a compressed ID (e.g., "05") that matches a row, it directly uses that ID as the physical index of the array. The system uses base addressing mode to directly locate the 5th slot of the array and performs an increment (Count++) or summation (Sum+=Value) operation on that slot.

[0097] This "array-level grouping" reduces the O(1) addressing process in traditional hash tables, which incurs huge constant overhead, to true absolute physical memory addressing requiring only a single CPU assembly instruction. It eliminates hash calculations, hash collisions, and dynamic memory allocation, pushing grouping performance to the limits of hardware physical performance.

[0098] Sub-step 3.3: O(1) time complexity aggregation based on run-length encoding: Suppose the business requirement is to calculate the total number of occurrences of a certain state (COUNT) or the sum of a series of values ​​(SUM).

[0099] In the traditional model, if a state occurs 100,000 times consecutively, the CPU needs to execute 100,000 addition instructions.

[0100] However, in this scheme, the system reads a run pair (containing "current value" and "counter").

[0101] The system first performs an interval intersection operation between the physical row range covered by the run pair and the "final selection vector". Through low-level bit operations (such as the popcount instruction, which is used to quickly count the number of "1"s in a binary word), the system can accurately calculate within a few clock cycles how many of these 100,000 rows are marked as "1" in the selection vector (assuming that 90,000 rows are marked as "1").

[0102] Next, the system directly uses mathematical multiplication instead of loop accumulation: If it is a COUNT operation, the system directly adds 90000 to the count value in the aggregate buffer.

[0103] If it is a SUM operation, the system directly multiplies the "current value" by 90000, and then adds the product to the aggregation buffer all at once.

[0104] Through this mechanism, regardless of how many tens of thousands of rows of data the run covers, the time complexity of the aggregation computation is always firmly fixed at the O(1) level. This makes the system perform several orders of magnitude better than traditional open-source frameworks when processing time-series big data with high local continuity.

[0105] Sub-step 3.4: Lock-free concurrent aggregation based on cache line alignment and thread-local storage In the final stage of big data batch processing, hundreds or thousands of worker threads typically write results to the global aggregation buffer simultaneously. If traditional mutexes or spinlocks are used to ensure data consistency, it will lead to extremely severe lock contention, causing the performance of multi-core CPUs to not only fail to scale linearly, but also to plummet due to frequent context switching.

[0106] This solution designs the following deterministic lock-free concurrent aggregation architecture: First, the system allocates an independent "thread-local aggregation buffer" in off-heap memory for each worker thread participating in the computation. This means that throughout the entire lengthy scanning and local aggregation phase, each thread only writes data to its own private memory area, and there is absolutely no shared state between threads, thus eliminating the need for locks at the physical root.

[0107] Secondly, to prevent the "false sharing" performance disaster common in multi-core CPU architectures, the system mandates that the size of each aggregation slot must be strictly aligned with the CPU cache line size (typically 64 bytes) when allocating these thread-local buffers. Even if the actual aggregation result only requires 8 bytes, the system will pad it with 56 bytes of invalid zero values. This ensures that memory addresses operated on by different threads will never be mapped to the same CPU physical cache line, thus avoiding cache miss storms caused by underlying hardware-level cache coherence protocols (such as the MESI protocol).

[0108] Finally, after all threads have completed their local aggregation, the system enters the "global merge" phase. At this point, the system uses a divide-and-conquer approach or lock-free instructions based on hardware primitives to perform a final deterministic merge of the results in each thread's local buffer. Since the data volume has been extremely condensed at this point (e.g., only grouping results for a few dozen cities remain), the overhead of the merge operation is negligible.

[0109] Step 4: Respond to the global memory barrier instruction, perform reverse mapping or baseline compensation on the intermediate aggregation result based on the metadata area information, and decompress and restore the plaintext result data with delay; assemble the plaintext result data into a columnar memory layout and output it with zero copy, and then reset the global allocation pointer of the off-heap contiguous memory pool to complete memory reclamation; In traditional von Neumann computing architectures, data decompression typically occurs at the very beginning of the computation pipeline (i.e., "early materialization"), leading to catastrophic memory bloat and memory bus bandwidth exhaustion. This invention's "delayed decompression" tightly presses the decompression operation to the absolute end of the entire computational directed acyclic graph (DAG), meaning that targeted decompression only occurs at the very last moment when the final human-readable result must be output to an external system (such as a client, downstream database, or network socket).

[0110] This step is strictly divided into the following five sub-steps: Sub-step 4.1: Synchronize computation graph terminal triggering and result set memory barrier Once all worker threads have completed the local aggregation in step three and the global merging has been completed using lock-free compare-and-swap (CAS) instructions, the system's master scheduling thread will receive a hardware-level interrupt signal indicating "computation complete".

[0111] At this point, the system does not immediately begin decompression. Instead, it first inserts a "global memory barrier" into memory. This memory barrier is a low-level CPU instruction (such as the mfence instruction in the x86 architecture). Its absolute function is to force all levels of cache (L1 / L2 / L3) of all multi-core CPUs to flush all dirty data that has not yet been written back to main memory to physical main memory, and to make all modifications made by threads to the aggregation result buffer absolutely visible globally.

[0112] Only after the memory barrier instruction has been executed can the system officially confirm that the "intermediate aggregation result" is in an absolutely consistent and immutable state, thus safely triggering the subsequent delayed decompression pipeline. This hardware instruction-based synchronization mechanism avoids the performance overhead caused by complex thread synchronization locks in high-level programming languages.

[0113] Sub-step 4.2: Reverse mapping of a dictionary based on a double array structure: Suppose our intermediate aggregation result is a grouped statistic based on dictionary encoding: [ID:05, Count:50000]. The task now is to quickly restore the meaningless integer ID "05" back to its original plaintext string (e.g., "Beijing").

[0114] In order to ensure the speed of reverse mapping when constructing the dictionary table in step one, this scheme designs an extremely ingenious "double array contiguous memory layout" in memory.

[0115] The first array is the "offset array," which is a contiguous block of memory storing 32-bit unsigned integers; the second array is the "character payload array," which is a contiguous stream of bytes tightly concatenated with all plaintext strings, without any delimiters.

[0116] When the system needs to reverse-search for ID "05", the execution logic is as follows: The system directly uses ID "05" as the physical index to access the 5th slot of the "offset array" and reads a memory address offset (let's say it's 120); then, the system reads the offset of the 6th slot (i.e., ID "05" plus 1) (let's say it's 126).

[0117] By simply subtracting these two offsets (126-120=6), the system instantly knows that the exact length of the target string is 6 bytes.

[0118] Subsequently, the system uses the base pointer plus the starting offset of 120 to directly locate the absolute physical memory address in the "character payload array" and calls the underlying memory copy instruction (such as memcpy or wide byte copy using SIMD registers) to directly extract these 6 bytes into the final output buffer.

[0119] The entire reverse mapping process involves no tree structure traversal and no hash collision handling; it consists of only two consecutive array lookups and one fixed-length memory copy. Its time complexity is absolutely O(1), and it is extremely friendly to the CPU's data prefetcher.

[0120] Sub-step 4.3: Differential reference value compensation and bit unpacking If the aggregation result involves numerical data using differential encoding and bit packing (e.g., finding the maximum value of a time series indicator), the intermediate result may be a small difference in the packed value (e.g., a packing difference of 15).

[0121] The system first reads the "baseline value" (assumed to be 1000) recorded in step one from the metadata header of the data block.

[0122] Next, the system performs a "bit unpacking" operation. The system reads the 64-bit machine word containing the difference, and based on the bit width recorded in the metadata (assumed to be 4 bits), uses low-level bitwise AND mask operations and right shift instructions to precisely extract these 4 bits from the machine word and store them in a standard 64-bit register.

[0123] Finally, the system performs "baseline compensation". The system calls a simple CPU addition instruction to add the extracted difference (15) to the base value (1000), instantly restoring the original real physical value (1015). The entire process is completed inside the CPU registers without generating any intermediate memory allocation.

[0124] Sub-step 4.4: Zero-copy output assembly for column-oriented memory formats (such as Apache Arrow): The decompressed and restored data needs to be sent to downstream systems. To avoid creating performance bottlenecks again during the output stage, this solution rejects traditional methods such as line-by-line concatenation, comma delimitation (like CSV), or complex JSON serialization.

[0125] This solution uses a "column-oriented standardized memory layout" (whose physical structure is highly compatible with the international standard Apache Arrow format) to reorganize the results.

[0126] The system pre-allocates multiple contiguous "column vector buffers" in off-heap memory. During the restoration operations in substeps 4.2 and 4.3, the system continuously writes the restored plaintext string to the "string column buffer," continuously writes the restored numerical value to the "numerical column buffer," and simultaneously updates an independent "validity bitmap" to mark any possible null values.

[0127] Once all the result data is assembled into this standardized columnar memory layout, if a downstream system (such as another distributed computing node or a database that supports Arrow format) requests to retrieve data, this solution directly calls the operating system-level "zero-copy" network transmission command (such as the sendfile system call under Linux, or kernel bypass technology based on DPDK).

[0128] The network interface card (NIC) hardware reads these columnar data directly from the off-heap memory buffer of this solution and sends them to the network link through the direct memory access (DMA) controller. The whole process bypasses the kernel-mode network protocol stack copy and user-mode memory copy of the operating system, pushing the network output throughput to the limit of the physical NIC bandwidth.

[0129] Sub-step 4.5: Destroy the "Anti-GC" memory pool based on pointer reset Once the batch processing task is complete and the results have been safely delivered downstream, the system must clean up the massive memory usage.

[0130] In traditional Java / Scala big data frameworks, hundreds of millions of temporary objects would trigger an extremely severe global garbage collection by the Java Virtual Machine, causing the entire node to fall into a "fake dead" state for several seconds or even tens of seconds.

[0131] In this solution, a customized off-heap large page memory pool is used in step one. All compressed data blocks, dictionary tables, bitmaps and aggregation buffers are allocated in this contiguous physical memory using simple pointer offsets.

[0132] Therefore, the memory destruction process in this scheme can be described as a "dimensionality reduction attack": the system will never traverse and release those tiny data structures one by one. The system only needs to execute an extremely simple instruction to directly reset the "global allocation pointer" of the memory pool to the starting physical address of the memory block (i.e., the zeroing operation).

[0133] The execution time of this instruction is less than one nanosecond. Logically, this huge block of memory is instantly cleared and can be immediately used to receive the next batch of big data processing tasks; physically, the operating system has no idea what happened, no memory fragmentation is generated, no system call overhead is generated, and the GC pause problem in the field of big data batch processing is eliminated.

[0134] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

[0135] It should be noted that, in this document, the use of relational terms such as "first" and "second" is merely for distinguishing one entity or operation from another, and does not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0136] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0137] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0138] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0139] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0140] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0141] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0142] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A method for batch processing of big data, characterized in that, include: The original data stream is acquired and parsed in a row-to-column manner in a contiguous off-heap memory pool. A lightweight encoding strategy is dynamically selected using a reservoir sampling algorithm to convert the original data stream into compressed data blocks, which include a metadata area, a null bitmap area, and a compressed data payload area. Receive query requests, extract metadata area information, and rewrite the filtering predicates into underlying comparison logic for the compressed data payload area; The instruction set is invoked to perform a vectorized scan to generate a matching bitmap without decompressing the compressed data payload area. The matching bitmap is then bitwise ANDed with the null bitmap area to generate the final selection vector. In the off-heap contiguous memory pool, allocate a thread-local aggregated buffer that is aligned with the processor cache line size; Using the final selection vector as a mask, without expanding the compressed data payload area, its data is directly used as the physical memory address index or calculation factor to perform lock-free concurrent aggregation in the buffer, and the intermediate aggregation result is obtained through global merging. In response to the global memory barrier instruction, the intermediate aggregation result is reverse mapped or compensated based on the metadata area information, and the plaintext result data is decompressed and restored with delay. The plaintext result data is assembled into a columnar memory layout and output with zero copy. Then, the global allocation pointer of the off-heap contiguous memory pool is reset to complete the memory reclamation.

2. The method for batch processing of big data according to claim 1, characterized in that, Row-to-column parsing is performed in the off-heap contiguous memory pool, including: For each column of data, a continuous bit space is allocated as a null bitmap area; when non-null data is parsed, the corresponding bit position is set to the first state, and the real data is continuously written into the actual data storage area; when null value is parsed, the corresponding bit position is set to the second state, and the placeholder is not written into the actual data storage area.

3. The method for batch processing of big data according to claim 2, characterized in that, Dynamically select a lightweight coding strategy using a reservoir sampling algorithm, including: Sample data were extracted using a reservoir sampling algorithm, and the cardinality, mean run length, and range of the sample data were calculated. If the base number is lower than the preset base number ratio, then dictionary encoding is selected; If the average run length is greater than the preset length threshold, then run coding is selected; If the data is numerical and the range is less than the preset range threshold, then differential encoding combined with bit packing is selected.

4. The method for batch processing of big data according to claim 3, characterized in that, The process of converting a raw data stream into compressed data blocks includes: Construct an order-preserving dictionary table, sort the extracted unique plaintext values ​​in lexicographical order and assign them incrementing integer identifiers; The minimum bit width is dynamically calculated based on the total number of unique plaintext values. The original plaintext value is replaced with the corresponding integer identifier, and the integer identifier is concatenated and written into a contiguous memory space using bit shift and bitwise OR operations, with the minimum bit width.

5. The method for batch processing of big data according to claim 1, characterized in that, The filtering predicate is rewritten as low-level comparison logic for the compressed data payload region, including: The target value is searched in the order-preserving dictionary table contained in the metadata area. If the target value is not found or is out of range, the early blocking mechanism is triggered, and a matching bitmap with all values ​​in the second state is directly generated and the processing of the current compressed data block is skipped. If the target value is found, the comparison logic for variable-length strings is rewritten as the comparison logic for fixed-width integer identifiers.

6. The method for batch processing of big data according to claim 5, characterized in that, The instruction set is invoked to perform vectorized scanning to generate a matching bitmap without decompressing the compressed data payload area, including: The rewritten target value is broadcast to the processor's vector register; By using direct memory access pointers, a continuous stream of compressed data is loaded from the compressed data payload area into another vector register at once; The single instruction multiple data stream comparison instruction is invoked to perform concurrent comparisons. The result mask is directly generated by the hardware, and the result mask is concatenated in sequence to form a matching bitmap.

7. The method for batch processing of big data according to claim 1, characterized in that, When run-length encoding is used in the compressed data payload area, the process of generating the matching bitmap includes: Read the current value and counter in the run-length pair; Compare the current value with the filter criteria: If there is no match, the pointer of the matching bitmap is jumped forward by the corresponding length in memory according to the value of the counter, and the bits are initialized to the second state in batches. If a match is found, the corresponding number of bits are sequentially set to the first state based on the counter value.

8. The method for batch processing of big data according to claim 1, characterized in that, When performing grouped aggregations and the data is dictionary-encoded, lock-free concurrent aggregations are performed in the buffer, including: A fixed-length contiguous array is pre-allocated in memory as a thread-local aggregation buffer. The length of the contiguous array is equal to the number of unique values ​​in the dictionary. The read compressed integer identifier is directly used as the physical index of the continuous array. The corresponding slot is located through the base address addressing mode and the accumulation or summation operation is performed.

9. A method for batch processing of big data according to claim 1, characterized in that, When performing aggregation and the data is run-length encoded, lock-free concurrent aggregation is performed in the buffer, including: Read the run-length pair and perform an interval intersection operation between the physical row range covered by the run-length pair and the final selection vector; Use bitwise operations to count the number of bits in the first state in the final selection vector; The number of bits is multiplied by mathematical multiplication and the current value in the run-length pair. The product is then accumulated into the thread-local aggregation buffer all at once.

10. A method for batch processing of big data according to claim 1, characterized in that, The intermediate aggregation results are reverse-mapped based on metadata area information, including: The search is performed using a double-array contiguous memory layout, which includes an offset array that stores memory address offsets and a character payload array that tightly concatenates plaintext strings. The integer identifier to be restored is used as the physical index to access the offset array, and the exact length of the target string is calculated by subtracting the offsets of adjacent slots. The absolute physical memory address of the character payload array is located using the base pointer and the starting offset, and the plaintext result data is extracted using a memory copy instruction.

Images