A bitmap fast operation method based on CPU hardware instruction
By using the AVX512 and BMI2 instruction sets to process bitmap indexes, the data positions in the bitmap indexes can be quickly parsed, solving the problem of high time overhead in obtaining the position of bitmap index '1' in existing technologies and improving database query efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANJING UNIV OF POSTS & TELECOMM
- Filing Date
- 2025-01-15
- Publication Date
- 2026-05-29
AI Technical Summary
While existing technologies improve performance and reduce storage overhead when using bitmap indexes for queries, they suffer from significant time overhead when retrieving the positions of '1's in the compressed bitmap to obtain relevant information.
Employing the AVX512 and BMI2 hardware instruction sets, this method defines and processes the word structure in the bitmap index, uses masking operations and bit manipulation instructions to quickly parse the bitmap index, calculate the location and quantity of valid data, avoid data overflow, and improve data processing efficiency.
It significantly shortens the time to obtain relevant data, improves the system throughput of database queries, and reduces the time overhead of data acquisition.
Smart Images
Figure CN119988428B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing technology, specifically to a method for fast bitmap manipulation based on CPU hardware instructions. Background Technology
[0002] Bitmap indexes are a special database indexing technique that uses bit arrays for storage and computation. They use bitmaps to represent the presence or absence of data in the database. Bitmap indexes are primarily created for columns with a large number of identical values, such as category, operator, department ID, and warehouse ID. For each possible value, a bitmap is created. If a table has n rows of data, then the bitmap for a specific value will have n bits, with each bit corresponding to a row in the table. If the value in that column of that row is that specific value, the corresponding bit is set to 1; otherwise, it is 0.
[0003] Bitmap indexes are widely used in scientific and commercial databases, particularly effective for specific types of queries, such as equality and selective range queries. This is because bitmap indexes can answer queries by performing bit-by-bit logical operations on the bitmap. Compression algorithms like WAH can significantly reduce storage overhead while maintaining high performance. However, retrieving the positions of '1's in the original compressed bitmap for relevant information can lead to significant time overhead. Therefore, this paper proposes a fast bitmap manipulation method based on CPU hardware instructions. By utilizing some instructions from AVX512 and BMI2, the location of all '1's in two types of words compressed using the WAH algorithm can be quickly achieved, significantly reducing the time required to retrieve relevant data. Summary of the Invention
[0004] To address the shortcomings of existing technologies, this invention provides a fast bitmap manipulation method based on CPU hardware instructions. This method solves the problem that while compression algorithms such as WAH can significantly reduce storage overhead while achieving high performance in bitmap indexing, obtaining the positions of '1's in the original compressed bitmap to retrieve relevant information can lead to substantial time consumption.
[0005] To achieve the above objectives, the present invention provides the following technical solution: a method for fast bitmap manipulation based on CPU hardware instructions, specifically including the following steps:
[0006] Step 1: Define the compressed bitmap index Define the 8 bits of data to be read as a word, define , Bitmap Index Given the combination of the i-th 64 words, define the total number of bits stored in the bitmap index as n bits. The number of all bits that are '1' stored in the bitmap index is nsets. The word that stores a large number of consecutive '0' or '1' sequences in the bitmap index is defined as the fill word, and the word that stores a mixed sequence is defined as the literal word.
[0007] Define `mark` as the numerical value that identifies the fill word and literal word, located at the beginning of the word. 1 represents the fill word, and 0 represents the literal word. The binary format of the literal word is 0xxxxxxx. The last 7 bits are a continuous sequence of 0s and 1s. Define `base` as the continuous data stored in the fill word. The value of `base` is 0 or 1, located at the second position in the fill word. Define the fill word as 1-fill word (binary format 11xxxxxx) when the data is compressed with all 1s, and 0-fill word (binary format 10xxxxxx) when the data is compressed with all 0s. Define `block` as 7 consecutive 1s or 0s. Define the last 6 bits of data in the fill word as `nblock`, where `nblock` is the number of blocks. Define the actual number of bits stored in the fill word as `nbit`.
[0008] Step 2: Read the compressed bitmap index Call the read instruction in the AVX512 instruction set to retrieve from Read sequentially ;
[0009] Step 3: Use AVX512 instructions to specifically distinguish between signed and unsigned integers, and identify them by calling the mask comparison instruction. 64-bit mask for 1-fill word 64-bit mask for literal words ;
[0010] Step 4, Definition The number of '1's in the literal word stored in is By calling the mask counting instruction in combination Get the marked literal word ;
[0011] Step 5, Definition The 512 bits of the nblock that identifies the fill word are = Calculate the number of blocks contained in each fill word. ;
[0012] definition The 512 bits that identify all words in the nblock are = ,calculate The number of blocks corresponding to all words in the text ;
[0013] Step Six: To avoid Directly adding 64 words in a single call will cause data overflow in the int8_t storage space. and Split and obtain , , , , and
[0014] Step 7, Definition The sum of the values in each word is , The sum of the values in each word is The _mm512_reduce_add_epi64 function is called to process the data. Get The _mm512_reduce_add_epi64 function is called to process the data. Get ,calculate The total number of nblocks in the fillword;
[0015] Step 8: Definition The sum of the values in each word is , The sum of the values in each word is , definition will The data compressed using AVX commands is ,Will The compressed data is defined as , definition will The bit-extended data is and , After steps six and seven, obtain and Obtain by adding the two together The total number of blocks, calculation Total number of digits ;
[0016] Step 9: Process sequentially using the BMI2 instruction set. In the word, extract the last '1' from the current word, extract the position of that '1', and then remove the last '1' from the word. Repeat step nine until the word is 0, and then continue to process the next word.
[0017] The present invention is further configured such that the calculation method for nbit in step one includes:
[0018] .
[0019] The present invention is further configured such that: the read instruction in the AVX512 instruction set in step two is the _mm512_loadu_epi8 instruction, and each instruction operation is processed... 8-bit Word document.
[0020] The present invention is further configured such that: the mask comparison instruction in step three is the _mm512_cmp_epu8_mask instruction, the usage of which includes:
[0021] The `_mm512_cmp_epu8_mask` instruction is called to compute the word as a `uint8_t`, comparing each word with 192 (11000000) and marking it. 64-bit mask for 1-fill word ;
[0022] The word is calculated as an int8_t by calling the _mm512_cmp_epi8_mask instruction, and each word is compared with 0 (00000000) to indicate... 64-bit mask for Chinese literal words .
[0023] The present invention is further configured such that the mask counting instruction in step four is the _mm512_maskz_sub_epi8 instruction.
[0024] The present invention is further configured such that: in step five, the number of blocks contained in each word is calculated. The methods include:
[0025] Calling the _mm512_maskz_sub_epi8 instruction to... Marked Modify the Word document in the document, and... Obtain by deleting the headers of each word. This allows you to obtain the number of blocks contained in each word. :
[0026] _mm512_maskz_sub_epi8( ).
[0027] The present invention is further configured such that: in step five, the calculation The number of blocks corresponding to all words in the text The methods include:
[0028] Calling the _mm512_maskz_set1_epi8 instruction to... Marked The literal word in the code is directly set to '1', obtained by calling the _mm512_and_epi32 instruction. The number of blocks corresponding to all words in the text :
[0029] _mm512_maskz_set1_epi8( ).
[0030] The present invention is further configured such that: in step six, the acquisition and The methods include:
[0031] definition The first 32 words are The last 32 words are ,definition The first 32 words are The last 32 words are ;
[0032] Calling _mm512_mask_reduce_add_epi64 will... and The result is obtained by summing four 64-bit words. and :
[0033] _mm512_mask_reduce_add_epi64 ( )
[0034] _mm512_mask_reduce_add_epi64 ( )
[0035] Calling the functions _mm_cvtsi64_si128 and _mm_cvtepu8_epi32 will... Each 8-bit word is converted to 64 bits to obtain and .
[0036] The present invention is further configured such that: in step eight, the calculation The total number of nblocks for fill words The methods include:
[0037] .
[0038] The present invention is further configured such that: in step eight, the calculation Total number of digits The methods include;
[0039] .
[0040] This invention provides a fast bitmap manipulation method based on CPU hardware instructions. It has the following advantages:
[0041] (1) This invention uses the hardware instruction set of AVX512 and BMI2 to parse the data stored in the bitmap index compressed by WAH, quickly obtain the position and total number of valid data in the current bitmap index, quickly locate the '1', significantly shorten the time to call related data, quickly query the relevant information of the data when querying the database, and significantly improve the throughput of the system.
[0042] (2) By making full use of CPU resources, this invention can process a larger scale of data at once compared with the traditional traversal method, greatly reducing the time overhead of data acquisition and improving data processing efficiency. Attached Figure Description
[0043] Figure 1 This is a word structure diagram of the compression algorithm in an embodiment of the present invention;
[0044] Figure 2 This is an example diagram of WAH in an embodiment of the present invention;
[0045] Figure 3 Example diagram of liteMask and nsetInLite acquisition in embodiments of the present invention;
[0046] Figure 4 Example diagrams are obtained for fillMask, nblockInFill512 and nblockInAll512 in the embodiments of the present invention;
[0047] Figure 5 This is an example diagram of obtaining the totalblock in an embodiment of the present invention;
[0048] Figure 6 This is an example diagram of BMI2 processing in an embodiment of the present invention. Detailed Implementation
[0049] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.
[0050] Please see Figure 1-6 The present invention provides the following technical solution: a method for fast bitmap manipulation based on CPU hardware instructions, specifically including the following steps:
[0051] Step 1: Define the compressed bitmap index Define the 8 bits of data to be read as a word, define , Bitmap Index Given the combination of the i-th 64 words, define the total number of bits stored in the bitmap index as n bits. The number of all bits that are '1' stored in the bitmap index is nsets. The word that stores a large number of consecutive '0' or '1' sequences in the bitmap index is defined as the fill word, and the word that stores a mixed sequence is defined as the literal word.
[0052] Define `mark` as the numerical value indicating the fill word and literal word, located at the beginning of the word. 1 represents the fill word, and 0 represents the literal word. The binary format of the literal word is `0xxxxxxx`. The last 7 bits are a continuous sequence of 0s and 1s. Define `base` as the continuously stored data in the fill word, with a value of 0 or 1, located at the second position in the fill word. Define the fill word as 1-fill word (binary format: `11xxxxxx`) if the data is compressed with all 1s, and 0-fill word (binary format: `10xxxxxx`) if the data is compressed with all 0s. Define `block` as 7 consecutive 1s or 0s. Define the last 6 bits of the fill word as `nblock`, where `nblock` is the number of blocks. Define the actual number of bits stored in the fill word as `nbit`, calculated as follows:
[0053] .
[0054] Step 2: Read the compressed bitmap index The _mm512_loadu_epi8 instruction in the AVX512 instruction set is invoked from... Read sequentially Each instruction operation is processed 8-bit Word document.
[0055] Step 3: Use AVX512 instructions to specifically distinguish between signed and unsigned integers, and identify them by calling the _mm512_cmp_epu8_mask instruction. 64-bit mask for 1-fill word 64-bit mask for literal words Specific methods include:
[0056] The `_mm512_cmp_epu8_mask` instruction is called to compute the word as a `uint8_t`, comparing each word with 192 (11000000) and marking it. 64-bit mask for 1-fill word ;
[0057] The word is calculated as an int8_t by calling the _mm512_cmp_epi8_mask instruction, and each word is compared with 0 (00000000) to indicate... 64-bit mask for Chinese literal words .
[0058] Step 4, Definition The number of '1's in the literal word stored in is By calling the _mm512_maskz_sub_epi8 instruction in combination with Get the marked literal word .
[0059] Step 5, Definition The 512 bits of the nblock that identifies the fill word are = Calculate the number of blocks contained in each fill word. Specific methods include:
[0060] Calling the _mm512_maskz_sub_epi8 instruction to... Marked Modify the Word document in the document, and... Obtain by deleting the headers of each word. This allows you to obtain the number of blocks contained in each word. :
[0061] _mm512_maskz_sub_epi8( ).
[0062] definition The 512 bits that identify all words in the nblock are = ,calculate The number of blocks corresponding to all words in the text Specific methods include:
[0063] Calling the _mm512_maskz_set1_epi8 instruction to... Marked The literal word in the code is directly set to '1', obtained by calling the _mm512_and_epi32 instruction. The number of blocks corresponding to all words in the text :
[0064] _mm512_maskz_set1_epi8( ).
[0065] Step Six: To avoid Directly adding 64 words in a single call will cause data overflow in the int8_t storage space. and Split and define The first 32 words are The last 32 words are ,definition The first 32 words are The last 32 words are ;
[0066] Calling _mm512_mask_reduce_add_epi64 will... and The result is obtained by summing four 64-bit words. and :
[0067] _mm512_mask_reduce_add_epi64 ( )
[0068] _mm512_mask_reduce_add_epi64 ( )
[0069] Calling the functions _mm_cvtsi64_si128 and _mm_cvtepu8_epi32 will... Each 8-bit word is converted to 64 bits to obtain and .
[0070] Step 7, Definition The sum of the values in each word is , The sum of the values in each word is The _mm512_reduce_add_epi64 function is called to process the data. Get The _mm512_reduce_add_epi64 function is called to process the data. Get ,calculate The total number of nblocks in the fillword The calculation formula includes:
[0071] .
[0072] Step 8: Definition The sum of the values in each word is , The sum of the values in each word is , definition will The data compressed using AVX commands is ,Will The compressed data is defined as , definition will The bit-extended data is and ,definition The sum of the values in each word is , The sum of the values in each word is After executing steps six and seven, obtain and Obtain by adding the two together The total number of blocks, calculation Total number of digits The calculation methods include:
[0073] .
[0074] Step 9: Process sequentially using the BMI2 instruction set. In the word list, extract the last '1' character, extract its position, and then remove the last '1' character from the word list. Repeat step nine until the word list returns 0, then continue processing the next word list. Specifically, this includes:
[0075] The `_blsi_u32` function in the BMI2 instruction set is used to extract `tmp2` from `tmp1`. `tmp2` contains only the last '1' from `tmp1`. The `anchor` function is used to record the starting position of the current word in the bitmap index. The `_blsr_u32` instruction is used to remove the last '1' from `tmp1`.
[0076]
[0077]
[0078] .
[0079] In summary, using AVX512 to process compressed bitmap data allows for processing 512 bits of data at a time. It generates corresponding fillmasks and litemasks based on the characteristics of each word, calculates the number of bits and 1s in the compressed bitmap using masking instructions, and avoids the risk of bit overflow during calculation using bit scaling instructions, significantly improving the time efficiency of data statistics. Using BMI2 to process compressed bitmap data allows for the rapid location of '1's in literal words within the compression algorithm, and the data can be quickly obtained through calculation formulas, further improving time efficiency.
[0080] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A method for fast bitmap manipulation based on CPU hardware instructions, characterized in that: Specifically, the following steps are included: Step 1: Define the compressed bitmap index Define the 8 bits of data to be read as a word, define , Bitmap Index Given the combination of the i-th 64 words, define the total number of bits stored in the bitmap index as n bits. The number of all bits that are '1' stored in the bitmap index is nsets. The word that stores a large number of consecutive '0' or '1' sequences in the bitmap index is defined as the fill word, and the word that stores a mixed sequence is defined as the literal word. Define `mark` as the numerical value that identifies the fill word and literal word, located at the beginning of the word. 1 represents the fill word, and 0 represents the literal word. The binary format of the literal word is 0xxxxxxx. The last 7 bits are a continuous sequence of 0s and 1s. Define `base` as the continuously stored data in the fill word. The value of `base` is 0 or 1, located at the second position in the fill word. Define the fill word as 1-fill word (binary format 11xxxxxx) when the data is compressed with all 1s, and 0-fill word (binary format 10xxxxxx) when the data is compressed with all 0s. Define `block` as 7 consecutive 1s or 0s. Define the last 6 bits of data in the fill word as `nblock`, where `nblock` is the number of blocks. Define the actual number of bits stored in the fill word as `nbit`. Step 2: Read the compressed bitmap index Call the read instruction in the AVX512 instruction set to retrieve from Read sequentially ; Step 3: Use AVX512 instructions to specifically distinguish between signed and unsigned integers, and identify them by calling the mask comparison instruction. 64-bit mask for 1-fill word 64-bit mask for literal words ; Step 4, Definition The number of '1's in the literal word stored in is By calling the mask counting instruction in combination Get the marked literal word ; Step 5, Definition The 512 bits of the nblock that identifies the fill word are = Calculate the number of blocks contained in each fill word. ; definition The 512 bits that identify all words in the nblock are = ,calculate The number of blocks corresponding to all words in the text ; Step Six, for and Split and obtain , , , , and ; Step 7, Definition The sum of the values in each word is , The sum of the values in each word is The _mm512_reduce_add_epi64 function is called to process the data. Get The _mm512_reduce_add_epi64 function is called to process the data. Get ,calculate The total number of nblocks in the fillword; Step 8: Definition The sum of the values in each word is , The sum of the values in each word is , definition will The data compressed using AVX commands is ,Will The compressed data is defined as , definition will The bit-extended data is and , After steps six and seven, obtain and Obtain by adding the two together The total number of blocks, calculated Total number of digits ; Step 9: Process sequentially using the BMI2 instruction set. In the word, extract the last '1' of the current word, extract the position of that '1', and then remove the last '1' from the word. Repeat step nine until the word is 0, and then continue to process the next word. The calculation in step five The number of blocks corresponding to all words in the text The methods include: Calling the _mm512_maskz_set1_epi8 instruction to... Marked The literal word in the code is directly set to '1', obtained by calling the _mm512_and_epi32 instruction. The number of blocks corresponding to all words in the text : _mm512_maskz_set1_epi8( )。 2. The bitmap fast operation method based on CPU hardware instructions according to claim 1, characterized in that: The calculation method for nbit in step one includes: 。 3. The bitmap fast operation method based on CPU hardware instructions according to claim 1, characterized in that: In step two, the read instruction in the AVX512 instruction set is the _mm512_loadu_epi8 instruction, and each instruction operation is processed... 8-bit Word document.
4. The bitmap fast operation method based on CPU hardware instructions according to claim 1, characterized in that: The mask comparison instruction in step three is the _mm512_cmp_epu8_mask instruction, and its usage includes: The `_mm512_cmp_epu8_mask` instruction is called to compute the word as a `uint8_t`, comparing each word with 192 (11000000) and marking it. 64-bit mask for 1-fill word ; The word is calculated as an int8_t by calling the _mm512_cmp_epi8_mask instruction, and each word is compared with 0 (00000000) to indicate... 64-bit mask for Chinese literal words .
5. The bitmap fast operation method based on CPU hardware instructions according to claim 1, characterized in that: The mask counting instruction in step four is the _mm512_maskz_sub_epi8 instruction.
6. The bitmap fast operation method based on CPU hardware instructions according to claim 1, characterized in that: In step five, the number of blocks contained in each word is calculated. The methods include: Calling the _mm512_maskz_sub_epi8 instruction to... Marked Modify the Word document in the document, and... Obtain by deleting the headers of each word. This allows you to obtain the number of blocks contained in each word. : _mm512_maskz_sub_epi8( )。 7. The bitmap fast operation method based on CPU hardware instructions according to claim 1, characterized in that: In step six, the acquisition and The methods include: definition The first 32 words are The last 32 words are ,definition The first 32 words are The last 32 words are ; Calling _mm512_mask_reduce_add_epi64 will... and The result is obtained by summing four 64-bit words. and : ； ； Calling the functions _mm_cvtsi64_si128 and _mm_cvtepu8_epi32 will... Each 8-bit word is converted to 64 bits to obtain and .
8. The bitmap fast operation method based on CPU hardware instructions according to claim 1, characterized in that: The calculation in step seven The total number of nblocks for fill words The methods include: 。 9. A method for fast bitmap manipulation based on CPU hardware instructions according to claim 1, characterized in that: The calculation in step eight Total number of digits The methods include; 。