A database boolean flag bitmap storage method, device and medium

Abstract

This invention discloses a method, device, and medium for storing Boolean flags in a database using a bitmap format, relating to the field of electronic digital data processing technology. The method includes: performing a compression storage operation on Boolean flags in the database according to a preset basic bitmap granularity, converting the Boolean flags from their original array form into a Boolean flag bitmap; wherein the compression storage operation is used to compress the number of storage units for the Boolean flags; reorganizing the Boolean flag bitmap according to the real-time access characteristics corresponding to the Boolean flags, obtaining a reorganized global Boolean flag bitmap; and responding to an operation request sent by a user, calling a preset bitwise operation interface to process a specified Boolean flag in the global Boolean flag bitmap or a Boolean flag in a specified bitmap block.

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing technology, specifically to a database Boolean flag bitmap storage method, device, and medium. Background Technology

[0002] In relational database systems, tables consist of multiple fields. During storage and retrieval, Boolean flags need to be recorded for each column of data. These flags indicate whether the column is NULL, whether the field has been calculated, whether a result is valid, or whether it needs to be skipped. Currently, Boolean arrays are commonly used to store these flags, with each flag occupying one byte. However, while a Boolean flag only requires one bit to represent, the actual storage unit is typically one byte, and may even occupy more memory due to compiler alignment rules. This results in extremely low storage space utilization, especially in wide table scenarios, leading to significant memory waste and performance overhead. Furthermore, the existing Boolean flag storage structure is rigid and cannot achieve a balance between storage compression and access performance under complex load environments. Additionally, the sparse distribution of Boolean arrays in memory requires a large number of cache lines, causing the CPU to load a large amount of irrelevant content simultaneously when reading valid data, reducing cache hit rate and memory access efficiency. Summary of the Invention

[0003] To address the aforementioned problems, this invention proposes a database Boolean flag bitmap storage method, comprising:

[0004] According to the preset basic bitmap granularity, the Boolean flags in the database are compressed and stored to convert the Boolean flags from the original array form into a Boolean flag bitmap; wherein, the compression and storage operation is used to compress the number of storage units of the Boolean flags;

[0005] Based on the real-time access characteristics corresponding to the Boolean flags, the Boolean flag bitmap is reorganized to obtain the reorganized global Boolean flag bitmap;

[0006] In response to an operation request sent by the user, a preset bitwise operation interface is invoked to process a specified Boolean flag in the global Boolean flag bitmap or a Boolean flag in a specified bitmap block.

[0007] In one implementation of the present invention, the Boolean flags in the database are compressed and stored according to a preset basic bitmap granularity, so as to convert the Boolean flags from the original array form into a Boolean flag bitmap, specifically including:

[0008] Determine the number of Boolean flags contained in each memory cell;

[0009] The Boolean flags in the database are compressed to reduce the original array of Boolean flags, which are stored as single bytes, to a storage unit containing the number of bits of the Boolean flags; wherein each Boolean flag corresponds to one bit.

[0010] Based on the total number of Boolean flags in the database and the number of Boolean flags, the corresponding number of storage units is determined, and based on a preset basic bitmap granularity, the Boolean flags are converted into a Boolean flag bitmap containing several bitmap blocks; wherein, the Boolean flag bitmap includes multiple bitmap blocks, and each bitmap block includes at least one storage unit.

[0011] In one implementation of the present invention, the Boolean flag bitmap is reorganized according to the real-time access characteristics corresponding to the Boolean flag, specifically including:

[0012] According to the preset monitoring interval, the real-time access characteristics of the Boolean flag are obtained, and the conflict rate and access frequency corresponding to the bitmap block where the Boolean flag is located are determined based on the real-time access characteristics.

[0013] Based on the conflict rate, the access frequency, and the storage density of the bitmap block, determine the granularity adjustment strategy corresponding to the bitmap block;

[0014] Based on the granularity adjustment strategy, the granularity of the basic bitmap is adjusted, and the bitmap blocks are merged or split using the adjusted granularity of the basic bitmap to achieve the reorganization of the Boolean flag bitmap.

[0015] In one implementation of the present invention, a granularity adjustment strategy corresponding to the bitmap block is determined based on the conflict rate, the access frequency, and the storage density of the bitmap block, specifically including:

[0016] If the conflict rate is greater than a preset conflict rate threshold, the granularity adjustment strategy of the bitmap block is determined to be a granularity splitting strategy.

[0017] If the conflict rate is less than a preset conflict security threshold, and the access frequency is lower than a preset frequency threshold, and the storage density of the bitmap block is lower than a preset density threshold, then the granularity adjustment strategy of the bitmap block is determined to be a granular merging strategy.

[0018] In one implementation of the present invention, before adjusting the granularity of the basic bitmap based on the granularity adjustment strategy, the method further includes:

[0019] The historical transaction logs of the Boolean flag bitmap are analyzed to identify whether there are access associations between the bits in the Boolean flag bitmap; wherein, the access association is used to indicate that the bitmap bits can be accessed together in the same transaction;

[0020] Based on the strength of the access associations, construct a bitmap access association matrix;

[0021] By accessing the association matrix through the bitmap, groups of Boolean flag bitmap blocks that have strong associations in the Boolean flag bitmap are filtered out.

[0022] In one implementation of the present invention, the granularity of the basic bitmap is adjusted based on the granularity adjustment strategy, and the bitmap blocks are merged or split using the adjusted basic bitmap granularity, specifically including:

[0023] Based on the granularity adjustment strategy, the basic bitmap granularity is adjusted according to the conflict rate, the access frequency, and the storage density to obtain the adjusted basic bitmap granularity.

[0024] When the granularity adjustment strategy is the granularity splitting strategy, hot spot conflict areas in the bitmap block are identified, and the bitmap block is split into several sub-bitmap blocks according to the adjusted basic bitmap granularity. Boolean flags in the hot spot conflict areas and non-hot spot conflict areas are cross-recombined into the sub-bitmap blocks.

[0025] When the granularity adjustment strategy is the granularity merging strategy, the bitmap blocks are merged according to the adjusted basic bitmap granularity, and during the merging process, the Boolean flag bitmap blocks are combined into the same or adjacent bitmap blocks.

[0026] In one implementation of the present invention, a preset bitwise operation interface is invoked to process a specified Boolean flag in the global Boolean flag bitmap or a Boolean flag in a specified bitmap block, specifically including:

[0027] When the operation request is for a single Boolean flag, a preset bitwise operation interface is invoked to perform lock-free processing on the specified Boolean flag in the global Boolean flag bitmap through atomic operations;

[0028] When the operation request is a batch operation for multiple Boolean flags, the specified bitmap block containing the multiple Boolean flags is loaded by SIMD instructions, and the segment lock corresponding to the specified bitmap block is obtained. Based on the segment lock, a preset bit operation interface is called to process the Boolean flags in the specified bitmap block.

[0029] In one implementation of the present invention, lock-free processing of a specified Boolean flag in the global Boolean flag bitmap is performed through atomic operations, specifically including:

[0030] Determine the index value of the specified Boolean flag in the global Boolean flag bitmap, and perform a right shift operation on the index value to determine the target storage unit where the Boolean flag is located;

[0031] The index value is ANDed with a preset value to determine the bit offset of the Boolean flag in the target storage unit;

[0032] Based on the operation request, determine the bit operation interface that the operation request needs to call; wherein, the bit operation interface includes a set judgment interface, a set interface, and a clear interface;

[0033] The specified Boolean flag is read from the target storage unit by calling the set judgment interface, and after the specified Boolean flag is right-shifted by the bit offset, a bitwise AND operation is performed with the value 1 to determine whether the specified Boolean flag is set.

[0034] The setting interface is called to shift the value 1 to the left by the bit offset, and then perform an OR operation with the current value stored in the target storage unit to set the specified Boolean flag to 1;

[0035] The clear interface is invoked, the value 1 is shifted left by the bit offset and then inverted. The result of the inversion operation is ANDed with the current value to clear the specified Boolean flag.

[0036] This invention provides a database Boolean flag bitmap storage device, the device comprising:

[0037] At least one processor;

[0038] And, a memory communicatively connected to the at least one processor;

[0039] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to perform a database Boolean flag bitmap storage method as described in any of the preceding claims.

[0040] This invention provides a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured as follows:

[0041] A database Boolean flag bitmap storage method as described in any of the preceding items.

[0042] The database Boolean flag bitmap storage method proposed in this invention can bring the following beneficial effects:

[0043] By using bitmap-based compression to store Boolean flags, the flags are transformed from byte arrays into bitmaps containing multiple bits, effectively improving storage space utilization and reducing memory consumption. Simultaneously, bitmap-based storage allows Boolean flags to be distributed contiguously in memory, reducing the loading of invalid data into cache lines. Combined with batch processing via bitwise operations, this effectively improves CPU cache utilization and data read / write efficiency, avoiding the cache resource waste caused by traditional sparse array storage. Dynamically reorganizing Boolean flags based on real-time access characteristics, compared to a fixed storage structure, achieves a balance between data compression ratio and access performance under complex environments, adapting to the performance of bitmap blocks and load conditions. Attached Figure Description

[0044] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0045] Figure 1 This is a flowchart illustrating a method for storing Boolean flags in a database according to an embodiment of the present invention.

[0046] Figure 2 This is a schematic diagram of the structure of a database Boolean flag bitmap storage device provided in an embodiment of the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0048] The technical solutions provided by the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0049] like Figure 1 As shown in the figure, an embodiment of the present invention provides a database Boolean flag bitmap storage method, comprising:

[0050] S101: Perform a compression storage operation on the Boolean flags in the database according to the preset basic bitmap granularity, so as to convert the Boolean flags from the original array form into a Boolean flag bitmap; wherein, the compression storage operation is used to compress the number of storage units of the Boolean flags.

[0051] Traditional Boolean flags are stored as arrays, with each flag occupying one byte. This not only wastes storage resources but also causes the CPU to read a large amount of useless data, reducing cache utilization. To solve these problems, this invention provides a database Boolean flag bitmap storage method, aiming to compress multiple Boolean flags into a bitmap. By compressing the number of storage units for Boolean flags, more efficient storage and access of Boolean data is achieved. A bitmap refers to a continuous string of bits, with each bit corresponding to a Boolean flag. Converting Boolean flags from array form to a bitmap compresses storage space from the byte level to the bit level, effectively saving storage space. The Boolean flag storage method used in this invention is based on a preset basic bitmap granularity. The basic bitmap granularity refers to the basic storage block size used when organizing the bitmap, corresponding to the number of storage units included in the bitmap block. Multiple bitmap blocks constitute a complete bitmap, and each bitmap block includes at least one storage unit. The storage unit described here can be a byte or a 64-bit integer, etc. Different data types will affect the storage and operation efficiency of the bitmap. The embodiments of the present invention can select the appropriate storage unit type according to actual needs and scenarios. For ease of description, in the embodiments of the present invention, only the bitmap storage method that uses bytes as Boolean flag storage units is described. The basic storage unit using bytes as Boolean flags is that each Boolean flag occupies 1 bit, and correspondingly, each byte contains eight bits, storing eight Boolean flags.

[0052] In one embodiment, based on a preset basic bitmap granularity, a compression storage operation is performed on the Boolean flags in the database. This compresses and reassembles the Boolean flags originally stored as independent bytes, converting them into a global Boolean flag bitmap composed of multiple bitmap blocks. During the initial conversion, firstly, the number of Boolean flags contained in each storage unit is determined. If a single byte is used as the storage unit, then the corresponding number of Boolean flags is 8. After determining the amount of data to be stored in each storage unit, a compression storage operation is performed on the Boolean flags in the database. This compresses the Boolean flags stored as single bytes in the original array into storage units containing a certain number of bits of Boolean flags, thereby optimizing the utilization of storage resources. Then, based on the total number of Boolean flags in the database and the number of Boolean flags corresponding to each storage unit, the required number of storage units is determined. Assuming there are 100 Boolean flags in the database, if bytes are used as storage units, with each byte storing 8 Boolean flags, then 13 bytes are needed to store these Boolean flags. It should be noted that the calculation result needs to be rounded up when calculating the number of storage units. Once the number of storage units is determined, the Boolean flags need to be allocated to various bitmap blocks according to certain rules based on a preset basic bitmap granularity, forming a Boolean flag bitmap. For example, if the preset basic bitmap granularity contains 2 storage units per bitmap block, i.e., 16 Boolean flags, then 100 Boolean flags will be allocated to 7 bitmap blocks. In this way, the Boolean flags, which originally existed as single bytes, are compressed into a Boolean flag map containing several bitmap blocks. This not only greatly reduces the storage space required but also improves data access efficiency. In subsequent operations, whether it is an operation on a single Boolean flag or a batch operation on multiple Boolean flags, storage and access can be performed on an aligned bitmap block basis, which is more conducive to batch parallel operations in the database and effectively improves database performance.

[0053] It should be noted that Boolean flags representing business logic and NULL values ​​representing missing data need to be physically stored separately. Therefore, when storing data in bitmap format, Boolean flags and NULL flags need to be stored separately as two independent bitmaps to avoid interference between different types of flags and improve the accuracy and efficiency of data processing. The processing operation for NULL bitmaps is the same as that for Boolean flag bitmaps, and will not be described separately thereafter.

[0054] S102: Based on the real-time access characteristics corresponding to the Boolean flags, the Boolean flag bitmap is reorganized to obtain the reorganized global Boolean flag bitmap.

[0055] The basic bitmap granularity is used to divide the data into several bitmap blocks. Each bitmap block contains an evenly distributed number of Boolean flags. However, in real-world applications, as data changes and operations continue, the usage and performance of different bitmap blocks can vary. For example, some bitmap blocks may become performance bottlenecks due to frequent access or high conflict rates, while others may waste storage resources due to extremely low access frequency. Therefore, it is necessary to dynamically adjust the basic bitmap granularity of the Boolean flag bitmap based on the real-time access characteristics corresponding to the Boolean flags. A global Boolean flag bitmap is obtained by reorganizing the Boolean flag bitmap to ensure a good balance between storage compression ratio and access performance under different load environments.

[0056] In one embodiment, when adjusting the granularity, real-time access characteristics represented by Boolean need to be obtained according to a preset monitoring interval. These real-time access characteristics include operation type, access count, and number of concurrent conflicts. Concurrent conflicts refer to bitmap block access failures caused by atomic operation failures or lock wait timeouts. Based on the real-time access characteristics, the conflict rate and access frequency for each bitmap block are calculated accordingly. The conflict rate reflects the probability of concurrent access contention events occurring on the bitmap block, while the access frequency reflects the frequency with which the bitmap block is used. The access frequency can be calculated by dividing the total number of operations initiated against the bitmap block by the monitoring interval, reflecting the average access frequency of the bitmap block per unit time. The conflict rate is the ratio of the number of conflicts occurring on the bitmap block within the monitoring interval to the total number of accesses.

[0057] Based on the conflict rate, access frequency, and storage density of each bitmap block, it can be determined whether some bitmap blocks in the current basic bitmap granularity division will have excessively high conflict rates or abnormal access frequencies, thus affecting overall performance. If so, the Boolean flag bitmap needs to be reorganized, and the granularity adjustment strategy for the corresponding bitmap block needs to be further determined. The granularity adjustment strategy includes granularity splitting strategy and granularity reorganization strategy. The granularity splitting strategy splits the bitmap block into smaller granularity sub-bitmap blocks to improve the utilization of storage resources, while the granularity merging strategy merges multiple bitmap blocks into a larger granularity bitmap block.

[0058] When determining the granularity adjustment strategy, it is necessary to comprehensively consider the collision rate, access frequency, and storage density of the bitmap block. Storage density refers to the proportion of the total number of bits of the Boolean flag that are actually effective in the bitmap block, which is used to reflect the utilization efficiency of the storage space in the bitmap block. When the Boolean flag is 1, it is in an effective state.

[0059] Specifically, several key thresholds are pre-set: a preset conflict rate threshold to trigger bitmap block splitting, and preset frequency and density thresholds to trigger bitmap block merging. After calculating the conflict rate of a bitmap block, it is compared with the preset conflict rate threshold. If the conflict rate of the bitmap block is greater than the preset conflict rate threshold, it indicates that the bitmap block is a hotspot area under high contention pressure. To alleviate lock contention pressure and improve concurrent processing capabilities, a granular splitting strategy needs to be adopted for this bitmap block. The splitting strategy aims to reduce the contention intensity of a single lock and improve multi-threaded parallel efficiency by dividing the bitmap block into multiple smaller storage units and distributing the access load.

[0060] The decision to merge bitmap blocks is triggered only if the conflict rate of a bitmap block is less than the conflict safety threshold. In this embodiment, the conflict safety threshold is significantly lower than the preset conflict rate threshold. The conflict safety threshold aims to constrain the merging operation of bitmap blocks to only be performed when the concurrency risk is low, thereby avoiding further exacerbation of conflicts after merging blocks that already have competition risks. If the access frequency is lower than the preset frequency threshold and its storage density is lower than the preset density threshold, it indicates that the block belongs to the cold data area, is rarely accessed by business, and most bits in the bitmap block are in an unused invalid or default state, resulting in serious space waste. To avoid unnecessary storage overhead, a granular merging strategy will be adopted for this bitmap block. The merging strategy aims to integrate multiple such low-activity or low-utilization bitmap blocks into a larger bitmap block, reducing the number of blocks in the overall bitmap structure and the total number of bitmap management metadata (such as lock objects, statistical structures, and index entries), thereby reducing management overhead, improving cache utilization efficiency, and, due to the continuity of memory access, effectively improving the cache hit rate when the merged bitmap block is used for sequential scanning or batch operations.

[0061] In one embodiment, after determining which granularity adjustment strategy to adopt, it is also necessary to determine the adjustment range corresponding to the basic bitmap granularity. Based on the collision rate, access frequency, and storage density of the bitmap blocks, the basic bitmap granularity is adjusted to obtain the adjusted basic bitmap granularity. For example, when using a granularity splitting strategy, the basic bitmap granularity is reduced according to the adjustment factor, causing the bitmap block to be split into smaller sub-bitmap blocks. When using a granularity merging strategy, the basic bitmap granularity is increased according to the adjustment factor, merging multiple bitmap blocks into a larger granularity bitmap block.

[0062] Specifically, if a granular splitting strategy is adopted, the target conflict rate obtained after basic testing needs to be determined first. The target conflict rate reflects the ideal conflict level that a bitmap block should achieve. Simultaneously, the maximum number of splits corresponding to the bitmap block needs to be determined. The maximum number of splits refers to the maximum number of times the bitmap block can be split, and its value is related to the initial size of the bitmap block and the storage unit type. When the conflict rate corresponding to a bitmap block exceeds a preset conflict rate threshold, the basic bitmap granularity is adjusted using the following formula:

[0063]

[0064] in, Indicates the maximum number of splits. Indicates the current conflict rate. The target conflict rate is represented by K, which represents the adjusted basic bitmap granularity, i.e., splitting a bitmap block into K sub-bitmap blocks.

[0065] When splitting bitmap blocks using a granular splitting strategy, it's also necessary to consider the conflict rate of different Boolean flags within the bitmap block. This is because different Boolean flags may exhibit different conflict scenarios in actual use. Some Boolean flags may have a relatively high conflict rate due to complex associated business logic involving multiple concurrent operations, while others have a relatively low conflict rate due to simpler usage scenarios. If the differences between Boolean flags within a bitmap block are not considered during splitting, the resulting sub-bitmap blocks may still suffer from uneven conflict rates. Therefore, when splitting bitmap blocks, the number of atomic operation failures or lock wait times for each Boolean flag within the bitmap block are analyzed, and consecutive bit ranges with significantly higher concurrent conflict counts than the average conflict count for that bitmap block are marked as hotspot conflict areas.

[0066] After identifying hotspot conflict areas, the original bitmap block is split into several sub-bitmap blocks according to the calculated adjusted basic bitmap granularity. During splitting, it's not simply a matter of cutting according to address order; instead, a cross-recombination process is performed. Boolean flags in hotspot conflict areas are dispersed and redistributed with those in non-hotspot conflict areas, ensuring that each sub-bitmap block contains both hotspot and non-hotspot data. If hotspot conflict areas are concentrated in a few Boolean flag bits within a bitmap block, one possible implementation is to combine odd-numbered Boolean flags from hotspot conflict areas with even-numbered Boolean flags from non-hotspot conflict areas into sub-bitmap block A, and vice versa. This cross-recombination strategy effectively prevents some sub-bitmap blocks from becoming new conflict hotspots after splitting, thus balancing the concurrent access pressure of each sub-bitmap block and maximizing the load distribution effect of the splitting.

[0067] If a granular merging strategy is adopted, the adjusted basic bitmap granularity is determined based on the access frequency and storage density, which can be expressed by the following formula:

[0068]

[0069]

[0070] Where M represents the adjusted basic bitmap granularity, that is, the bitmap block needs to be merged into M sub-bitmap blocks; This represents the maximum number of merges, usually set to 8. This represents the base number of merges, which is generally not less than 2. This represents the demand factor, used to indicate the degree of insufficient access frequency and wasted storage space; Indicates the preset frequency threshold. Indicates the access frequency of the bitmap block. This indicates a preset density threshold. Indicates storage density, and This represents a weighting factor used to adjust the relative importance between storage density and access frequency. The specific values ​​can be set according to actual needs.

[0071] Based on the adjusted basic bitmap granularity, bitmap blocks are merged. During the merging process, not only the physical adjacency of bitmap blocks needs to be considered, but also Boolean flag bitmap block groups with access relationships need to be given priority. By merging bitmap blocks belonging to the same Boolean flag bitmap block group into the same or adjacent new bitmap blocks, Boolean flags that are frequently accessed in the same transaction are gathered in the same or adjacent storage units, thereby significantly reducing the number of memory accesses and cache line loadings during transaction execution, and improving cache hit rate and transaction processing speed.

[0072] Specifically, a Boolean flag bitmap block group refers to a set of bitmap blocks with access associations. The Boolean flags in these blocks are often accessed jointly during business operations. For example, different attribute flags belonging to the same entity, or multiple status flags that must be processed simultaneously in the same business process. To further quantify the associations between bitmap blocks, historical transaction logs of the Boolean flag bitmap are analyzed to identify whether access associations exist between different bitmap blocks. That is, by statistically analyzing the frequency and patterns of co-accessing any two or more bitmap blocks in the same transaction, it is determined whether the bitmap blocks can be accessed simultaneously within the same transaction. If they can be accessed simultaneously, it indicates that there is an access association between the bitmap blocks. Based on the strength of the access associations, a bitmap access association matrix is ​​constructed. The rows and columns of the matrix correspond to each bitmap block in the Boolean flag bitmap, and each element represents the association strength between the corresponding two bitmap blocks. The association strength is usually measured by calculating the probability of their co-occurrence in historical transactions, for example, using the Jaccard similarity coefficient, which is the union of the number of times two bitmap blocks co-occur in the same transaction divided by their individual occurrence counts. The stronger the association, the greater the likelihood that bitmap blocks are accessed together. Based on the bitmap access association matrix, groups of Boolean flag bitmap blocks with strong associations can be filtered out. When filtering Boolean flag bitmap block groups, a set strength threshold is usually used to identify whether there is a strong association between bitmap blocks. All bitmap blocks with an association strength exceeding the threshold will be automatically clustered to form a strongly associated Boolean flag bitmap block group.

[0073] When merging bitmap blocks, multiple small bitmap blocks belonging to the same Boolean flag bitmap group should be merged into the same bitmap block first. If they cannot be completely merged due to size limitations, they should be merged into bitmap blocks with adjacent physical addresses. When processing transactions, the multiple related flag bits that need to be accessed have a very high probability of being located in the same or a few adjacent memory units. This allows the CPU to complete all data retrieval with the fewest cache line loads, significantly reducing memory access latency and cache miss rate. At the same time, it also reduces the number of locks that need to be acquired and managed during transactions, reducing lock overhead and potential deadlock risks, thereby significantly improving the overall throughput and response speed of the system in high-concurrency environments.

[0074] S103: In response to an operation request sent by the user, call the preset bit operation interface to process the specified Boolean flag in the global Boolean flag bitmap or the Boolean flag in the specified bitmap block.

[0075] Based on the real-time reconstructed global Boolean flag bitmap, when a user needs to query or modify the state of a Boolean flag, they send a corresponding operation request to the database system. This request includes the Boolean flag's identifier and the type of operation to be performed, which includes read, set, and clear. Upon receiving the request, the system calls a pre-defined bitwise operation interface to calculate the specific byte and bit position of the requested Boolean flag in the Boolean flag bitmap, and finally performs the bitwise operation corresponding to the operation type. It should be noted that the operation request may be for a single Boolean flag or a batch operation for multiple Boolean flags; accordingly, it is also necessary to process the specified Boolean flags in the global Boolean flag bitmap or the Boolean flags in a specified bitmap block.

[0076] In one embodiment, when a user's operation request involves only a single Boolean flag, such as querying the NULL status of a specific field in a row, or setting a validity flag individually, a preset bitwise operation interface needs to be invoked to perform lock-free atomic operations on the specified Boolean flag in the global Boolean flag bitmap. The prerequisite for performing bitwise operations on a specified Boolean flag is to map the specified Boolean flag to the Kth bit of the Mth storage unit in physical storage, thus achieving precise location of the specified Boolean flag.

[0077] Specifically, the index value of the specified Boolean flag in the global Boolean flag bitmap is determined, i.e., the sequential number of the Boolean flag in the global Boolean flag bitmap (starting from 0), such as the i-th flag. Then, a right shift operation is performed on the index value to determine the target storage unit where the Boolean flag is located. This operation can be represented as i >> 3. Since a storage unit stores eight Boolean flags, shifting the index of the specified Boolean flag 3 bits to the right is equivalent to i / 8, thus locating the target storage unit where the specified Boolean flag is located. The bit offset of the Boolean flag in the target storage unit is determined by performing a bitwise AND operation on the index value and a preset value. Here, the preset value is 7, used to mark the specific position of the Boolean flag in the storage unit. The bit offset calculated by i & 7 can accurately locate the bit position of the specified Boolean flag in the target storage unit. After completing the above location operation, the bit operation interface to be called by the operation request is determined according to the operation request. The bit operation interface includes a set judgment interface, a set interface, and a clear interface, used to implement read, set to 1, and clear to 0 operations, respectively.

[0078] The process of determining whether a bit is set essentially involves reading the value of a specified Boolean flag to determine if that bit is set. The specific function is represented as FLAG_IS_SET(bitmap, i) = ((bitmap[i>>3]>>(i&7))&1). Through the set-determination interface, the value corresponding to the specified Boolean flag in the target memory unit is atomically read into the register. The specified Boolean flag is then shifted right by a bit offset, moving to the least significant bit (0). The shifted result is then ANDed with the value 1. The result of the AND operation determines whether the specified Boolean flag is set. If the result is 1, the specified Boolean flag is set; if it is 0, the specified Boolean flag is not set.

[0079] For the "set to 1" operation, this process requires calling the setting interface to set the flag bit of the specified Boolean flag to 1. The specific operation function is represented as FLAG_SET(bitmap, i) = (bitmap[i>>3] |= 1<<(i&7)). Through the setting interface, the value 1 is shifted left by a bit offset, generating a bitmask. The bitmask is then ORed with the current value stored in the target memory location. Since any bit ORed with 1 results in 1, this operation only sets the flag bit containing the specified Boolean flag to 1, leaving other bits unchanged.

[0080] For the clear operation, the value of the specified Boolean flag needs to be cleared to 0. The specific operation function is represented as FLAG_CLEAR(bitmap, i) = (bitmap[i>>3]&= ~(1<<(i&7))). Calling the clear interface first shifts the value 1 left by the bit offset to generate a bitmask. Then, the bitmask is inverted bit by bit. The result of the inversion operation is ANDed with the current value to clear the specified Boolean flag to 0, while other flag bits remain unchanged.

[0081] When a user's operation request involves batch operations on multiple Boolean flags, batch processing of the Boolean flags is required using SIMD and segmented locks. First, the specified bitmap block containing the Boolean flags to be operated on is identified. Then, SIMD instructions are used to load the Boolean flags to be operated on from memory into the CPU's wide vector register in one go. Before performing batch operations, a segmented lock associated with the specified bitmap block is acquired. Each bitmap block in the global Boolean flag bitmap maintains a lightweight lock. Segmented locks ensure that other threads cannot modify the specified bitmap block simultaneously during batch operations, thus maintaining consistency among multiple flags within the specified bitmap block. Under the protection of the segmented lock, the operation targets are the Boolean flags in the specified bitmap block already loaded into the SIMD register. A pre-defined bitwise operation interface is invoked, and bitwise operations are performed using the SIMD instruction set to process multiple Boolean flags simultaneously with a single instruction. After vectorization calculation is completed, the processing results are first stored in the thread's local memory. When the amount of data accumulates to a certain level, or when the thread is idle, the processing results are written to shared memory in batches, thereby reducing global atomic operations and lock contention and improving multi-threaded performance.

[0082] In a multithreaded environment, concurrent read / write operations on different Boolean flags by multiple threads can easily lead to concurrency conflicts. While using a global lock can ensure thread safety, it significantly reduces performance. Relying entirely on lock-free atomic operations can improve performance, but the complexity of these operations makes it difficult to guarantee consistency over a wide range of operations. Therefore, this invention combines atomic operations with segmented locks. It achieves lock-free access to a single Boolean flag through atomic operations, and manages the batch processing of a large range of Boolean flags through segmented locks. This approach ensures thread safety while achieving high-performance concurrent processing in the database system.

[0083] The above are embodiments of the method proposed in this invention. Based on the same idea, some embodiments of this invention also provide devices and non-volatile computer storage media corresponding to the above methods.

[0084] Figure 2 This is a schematic diagram of a database Boolean flag bitmap storage device provided in an embodiment of the present invention. Figure 2 As shown, it includes:

[0085] At least one processor; and,

[0086] At least one processor-communication-connected memory; wherein,

[0087] The memory stores instructions that can be executed by at least one processor, and the instructions, when executed by at least one processor, enable at least one processor to:

[0088] Perform a database Boolean flag bitmap storage method as described in any of the preceding items.

[0089] This invention provides a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured as follows:

[0090] A database Boolean flag bitmap storage method as described in any of the preceding items.

[0091] The various embodiments in this invention are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and medium embodiments are relatively simple in description because they are fundamentally similar to the method embodiments; relevant parts can be referred to the descriptions in the method embodiments.

[0092] The devices, media, and methods provided in the embodiments of the present invention are one-to-one correspondences. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0093] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0094] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0095] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0096] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0097] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0098] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0099] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0100] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0101] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A method for storing Boolean flags in a database using a bitmap format, characterized in that, The method includes: According to the preset basic bitmap granularity, the Boolean flags in the database are compressed and stored to convert the Boolean flags from the original array form into a Boolean flag bitmap; wherein, the compression and storage operation is used to compress the number of storage units of the Boolean flags; Based on the real-time access characteristics corresponding to the Boolean flags, the Boolean flag bitmap is reorganized to obtain the reorganized global Boolean flag bitmap; In response to an operation request sent by the user, a preset bit operation interface is invoked to process a specified Boolean flag in the global Boolean flag bitmap or a Boolean flag in a specified bitmap block. Based on the real-time access characteristics corresponding to the Boolean flags, the Boolean flag bitmap is reorganized, specifically including: According to the preset monitoring interval, the real-time access characteristics of the Boolean flag are obtained, and the conflict rate and access frequency corresponding to the bitmap block where the Boolean flag is located are determined based on the real-time access characteristics. Based on the conflict rate, the access frequency, and the storage density of the bitmap block, determine the granularity adjustment strategy corresponding to the bitmap block; Based on the granularity adjustment strategy, the granularity of the basic bitmap is adjusted, and the bitmap blocks are merged or split using the adjusted granularity of the basic bitmap to achieve the reorganization of the Boolean flag bitmap; Based on the conflict rate, the access frequency, and the storage density of the bitmap block, a granularity adjustment strategy corresponding to the bitmap block is determined, specifically including: If the conflict rate is greater than a preset conflict rate threshold, the granularity adjustment strategy of the bitmap block is determined to be a granularity splitting strategy. If the conflict rate is less than a preset conflict security threshold, and the access frequency is lower than a preset frequency threshold, and the storage density of the bitmap block is lower than a preset density threshold, then the granularity adjustment strategy of the bitmap block is determined to be a granular merging strategy. Based on the granularity adjustment strategy, before adjusting the granularity of the basic bitmap, the method further includes: The historical transaction logs of the Boolean flag bitmap are analyzed to identify whether there are access relationships between the bits in the Boolean flag bitmap; wherein, the access relationship is used to indicate that the bitmap bits can be accessed together in the same transaction; Based on the strength of the access associations, construct a bitmap access association matrix; By accessing the association matrix through the bitmap, groups of Boolean flag bitmap blocks with strong associations are filtered out from the Boolean flag bitmap. Based on the granularity adjustment strategy, the granularity of the basic bitmap is adjusted, and the bitmap blocks are merged or split using the adjusted granularity, specifically including: Based on the granularity adjustment strategy, the basic bitmap granularity is adjusted according to the conflict rate, the access frequency, and the storage density to obtain the adjusted basic bitmap granularity. When the granularity adjustment strategy is a granular splitting strategy, the target conflict rate obtained after passing the basic test is determined. The target conflict rate can reflect the conflict level that the bitmap block should reach under ideal conditions. The maximum number of splits corresponding to the bitmap block is determined. The maximum number of splits refers to the maximum number of times the bitmap block can be split. Its value is related to the initial size of the bitmap block and the storage unit type. When the collision rate of a bitmap block exceeds a preset collision rate threshold, the basic bitmap granularity is adjusted using the following formula: in, Indicates the maximum number of splits. Indicates the current conflict rate. The target conflict rate is represented by K, which represents the adjusted basic bitmap granularity, i.e., the bitmap block is split into K sub-bitmap blocks. When the granularity adjustment strategy is a granularity merging strategy, the adjusted basic bitmap granularity is determined based on the access frequency and storage density, which can be expressed by the following formula: Where M represents the adjusted basic bitmap granularity, that is, the bitmap block needs to be merged into M sub-bitmap blocks; This represents the maximum number of merges, usually set to 8. This represents the base number of merges, which is generally not less than 2. This represents the demand factor, used to indicate the degree of insufficient access frequency and wasted storage space; Indicates the preset frequency threshold. Indicates the access frequency of the bitmap block. This indicates a preset density threshold. Indicates storage density, and This represents a weighting factor used to adjust the relative importance between storage density and access frequency. The specific values ​​can be set according to actual needs; When the granularity adjustment strategy is the granularity splitting strategy, hot spot conflict areas in the bitmap block are identified, and the bitmap block is split into several sub-bitmap blocks according to the adjusted basic bitmap granularity. Boolean flags in the hot spot conflict areas and non-hot spot conflict areas are cross-recombined into the sub-bitmap blocks. When the granularity adjustment strategy is the granularity merging strategy, the bitmap blocks are merged according to the adjusted basic bitmap granularity, and during the merging process, the Boolean flag bitmap blocks are combined into the same or adjacent bitmap blocks; The preset bitwise operation interface is invoked to process a specified Boolean flag in the global Boolean flag bitmap or a specified Boolean flag in a specified bitmap block, specifically including: When the operation request is for a single Boolean flag, a preset bitwise operation interface is invoked to perform lock-free processing on the specified Boolean flag in the global Boolean flag bitmap through atomic operations; When the operation request is a batch operation for multiple Boolean flags, the specified bitmap block containing the multiple Boolean flags is loaded by SIMD instructions, and the segment lock corresponding to the specified bitmap block is obtained. Based on the segment lock, a preset bit operation interface is called to process the Boolean flags in the specified bitmap block. Lock-free processing of specified Boolean flags in the global Boolean flag bitmap is performed through atomic operations, specifically including: Determine the index value of the specified Boolean flag in the global Boolean flag bitmap, and perform a right shift operation on the index value to determine the target storage unit where the Boolean flag is located; The index value is ANDed with a preset value to determine the bit offset of the Boolean flag in the target storage unit; Based on the operation request, determine the bit operation interface that the operation request needs to call; wherein, the bit operation interface includes a set judgment interface, a set interface, and a clear interface; The specified Boolean flag is read from the target storage unit by calling the set judgment interface, and after the specified Boolean flag is right-shifted by the bit offset, a bitwise AND operation is performed with the value 1 to determine whether the specified Boolean flag is set. The setting interface is called to shift the value 1 to the left by the bit offset, and then perform an OR operation with the current value stored in the target storage unit to set the specified Boolean flag to 1; The clear interface is called to shift the value 1 to the left by the bit offset and then perform an inversion operation. The result of the inversion operation is then ANDed with the current value to clear the specified Boolean flag.

2. The database Boolean flag bitmap storage method according to claim 1, characterized in that, According to a preset basic bitmap granularity, the Boolean flags in the database are compressed and stored to convert the Boolean flags from their original array form into a Boolean flag bitmap. Specifically, this includes: Determine the number of Boolean flags contained in each memory cell; The Boolean flags in the database are compressed to reduce the original array of Boolean flags, which are stored as single bytes, to a storage unit containing the number of bits of the Boolean flags; wherein each Boolean flag corresponds to one bit. Based on the total number of Boolean flags in the database and the number of Boolean flags, the corresponding number of storage units is determined, and based on a preset basic bitmap granularity, the Boolean flags are converted into a Boolean flag bitmap containing several bitmap blocks; wherein, the Boolean flag bitmap includes multiple bitmap blocks, and each bitmap block includes at least one storage unit.

3. A database Boolean flag bitmap storage device, characterized in that, The device includes: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform a database Boolean flag bitmap storage method as described in any one of claims 1-2.

4. A non-volatile computer storage medium storing computer-executable instructions, characterized in that, The computer-executable instructions are set as follows: A database Boolean flag bitmap storage method as described in any one of claims 1-2.

Images