A method and apparatus for multi-version bitmap processing based on hierarchical grouping difference
By adopting a hierarchical grouping differential bitmap storage method, the memory consumption problem caused by bitmap version expansion in the HTAP database system is solved, achieving efficient multi-version storage and recycling of bitmaps, and improving system performance and stability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- DIGITAL YI TECH (BEIJING) CO LTD
- Filing Date
- 2026-03-30
- Publication Date
- 2026-07-17
Smart Images
Figure CN121935254B_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of database system technology, and more specifically, relates to a bitmap multi-version processing method and apparatus based on hierarchical grouping difference. Background Technology
[0002] Hybrid Transactional and Analytical Processing (HTAP) database systems are widely used because they can simultaneously support online transaction processing and online analytical processing. In the columnar storage engine of an HTAP database system, a multi-version concurrency control mechanism is typically employed to ensure transaction consistency without blocking read and write operations. Specifically, the system maintains a set of bitmap versions for each columnar storage data unit, using the bitmap to record the visibility state of data rows. This allows for the retention of multiple historical states of the data to meet the snapshot query requirements of different transactions.
[0003] However, in scenarios involving high-concurrency writes and long-running analytical transactions, systems need to maintain a large number of historical bitmap versions. Existing bitmap management technologies typically store each version independently as a complete bitmap. As business runtime and transaction commits increase, the number of bitmap versions accumulated in the system continues to grow, leading to severe memory resource consumption and consequently affecting the overall performance and stability of the database. Therefore, how to effectively reduce the memory footprint of bitmap versions is a pressing technical problem that needs to be solved in this field. Summary of the Invention
[0004] In view of the shortcomings of the prior art, the purpose of this application is to effectively reduce the memory usage of bitmap versions.
[0005] To achieve the above objectives, in a first aspect, this application provides a bitmap multi-version processing method based on hierarchical grouping difference, the method comprising:
[0006] Obtain bitmap modification records (records that describe the changes made to data by a transaction in a bitmap format) from the bitmap version log module. The bitmap version log module is used to store bitmap modification records corresponding to transactions.
[0007] The bitmap modification records are processed through the hierarchical grouping differential bitmap storage module;
[0008] The hierarchical grouped differential bitmap storage module is used to store one or more version groups. Each version group is configured with a reference bitmap and a differential bitmap. The reference bitmap is stored in the form of a complete bitmap. The differential bitmap is based on the reference bitmap and records the bit positions that have changed relative to the reference bitmap in the corresponding bitmap version. The bitmaps within the version group are compressed and stored using differential encoding.
[0009] The hierarchical grouped differential bitmap storage module is used to determine whether to create a new version group based on the group capacity. If so, a new version group is created based on the bitmap modification record; otherwise, a new differential bitmap is created based on the bitmap modification record, and the new differential bitmap is inserted into the most recently created version group. The group capacity is used to indicate the maximum number of versions in the version group.
[0010] It is understood that the bitmap multi-version processing method based on hierarchical grouping differential bitmap provided in this application constructs a version group structure through a hierarchical grouping differential bitmap storage module. Utilizing the high similarity between adjacent bitmap versions, it stores only a complete bitmap as a reference within each version group, while storing other bitmap versions within the group as differential bitmaps. The differential bitmap only records the bit positions that have changed relative to the reference bitmap. This differential encoding mechanism avoids data redundancy caused by storing all versions as complete bitmaps. Simultaneously, it dynamically controls the creation of version groups based on group capacity to maintain the rationality of the compression structure, thereby achieving high compression ratio storage while ensuring the integrity of multi-version data and effectively reducing memory consumption.
[0011] This section elaborates on achieving high compression ratio storage while ensuring the integrity of multi-version data: This method limits the size of versions within a single version group by setting a group capacity threshold. When the number of versions in a recently created version group has not reached the group capacity, new versions are continuously stored in the form of a differential bitmap to fully utilize the sparsity characteristics of differential coding and reduce storage overhead. When the number of versions reaches the group capacity, a new version group is forcibly created, and a new reference bitmap is generated as the new differential baseline. This dynamic control mechanism effectively prevents the differential bitmap data from becoming too dense and the linked list traversal path from becoming too long due to a single reference bitmap covering too long a transaction span. Thus, while ensuring the efficiency of differential coding, the computational cost of version reconstruction is controlled, achieving an optimized balance between storage space utilization and data access performance.
[0012] Therefore, this application constructs a version group structure that includes a reference bitmap stored in its complete form and a differential bitmap stored in a differential encoding form, and dynamically determines whether to create a new version group or a new differential bitmap based on the bitmap modification record according to the group capacity, thereby realizing hierarchical grouped compressed storage of bitmap multi-version data.
[0013] In one possible implementation, the hierarchical grouped differential bitmap storage module is constructed based on a two-level linked list structure, which specifically includes a reference bitmap chain as the first level and a differential bitmap chain as the second level.
[0014] A reference bitmap chain is formed by connecting reference bitmaps from multiple version groups sequentially according to the transaction commit order.
[0015] Each version group has an internal configuration of a differential bitmap chain. In any version group, the differential bitmap chain is composed of several differential bitmaps connected in sequence.
[0016] The reference bitmap is connected to the starting node of the differential bitmap chain within its version group via a pointer structure, and the reference bitmap is associated with version range information, which is used to characterize the transaction commit range covered by the version group.
[0017] Specifically, for reference bitmaps within a version group, the reference bitmap stores the data visibility state in a full bitmap format. The reference bitmap is associated with version range information representing the range of transaction commit sequence numbers covered by that version group; it is also associated with a pointer to the reference bitmap of the preceding version group (i.e., the version group with the earlier transaction commit order); and it is further associated with a pointer to the head of the differential bitmap chain—that is, the reference bitmap is connected to the starting node of the differential bitmap chain within its version group via a pointer structure.
[0018] Here, we will illustrate the example of how the above reference bitmaps are connected sequentially according to the transaction commit order. If there are three versions in the system, which are generated in order of their generation time, namely a first version group with a transaction commit sequence number range of [100, 110), a second version group with a range of [110, 125), and a third version group with a range of [125, 140), then the reference bitmap of the third version group serves as the head of the reference bitmap chain. The reference bitmap of the second version group points to the reference bitmap of the second version group through a pointer. The reference bitmap of the second version group points to the reference bitmap of the first version group through a pointer, thus forming a first-level linked list structure arranged in descending order of transaction commit sequence number.
[0019] Here, we will provide an example of the starting node of the differential bitmap chain mentioned above. The starting node is the differential bitmap with the largest transaction commit sequence number in the differential bitmap chain (i.e., the latest differential version in this version group). This starting node is directly indexed by the reference bitmap through a pointer structure and serves as the traversal entry point of the differential bitmap chain. The starting node points to the differential bitmap with the second largest transaction commit sequence number in this version group through the successor pointer.
[0020] Here, we will exemplify the above differential bitmap chain as being composed of several differential bitmaps connected sequentially. In a version group based on the reference bitmap with transaction commit sequence number 200, there exists a zero differential bitmap with transaction commit sequence number 200, a first differential bitmap with transaction commit sequence number 201, a second differential bitmap with transaction commit sequence number 202, and a third differential bitmap with transaction commit sequence number 203. The third differential bitmap is located at the head of the differential bitmap chain, and its pointer points to the second differential bitmap. The second differential bitmap then points to the first differential bitmap, and the first differential bitmap then points to the zero differential bitmap, thus forming a second-level linked list structure within the version group, connected in descending order of transaction commit sequence number.
[0021] In one possible implementation, the above-mentioned determination of whether to create a new version group based on group size includes:
[0022] Get the number of versions in the most recently created version group, and record it as the number of versions in the most recently created version group;
[0023] Compare the number of recent version groups with the group size;
[0024] If the number of versions in the most recent version group reaches the group capacity, a new version group will be created; if the number of versions in the most recent version group does not reach the group capacity, a new version group will not be created.
[0025] It is understandable that the reference bitmap of the most recently created version group is located at the head of the reference bitmap chain, so the most recently created version group can be obtained based on the reference bitmap at the head of the reference bitmap chain.
[0026] In one possible implementation, the creation of a new version group based on the bitmap modification record includes:
[0027] Based on the bitmap modification record, the reference bitmap in the most recently created version group, and the differential bitmap at the head of the differential bitmap chain in the most recently created version group (i.e., the latest differential bitmap version in the group), determine the target complete bitmap after the transaction commit corresponding to the bitmap modification record.
[0028] Create a new version group, store the target complete bitmap as the reference bitmap of the new version group, and insert the new version group at the head of the reference bitmap chain; create an initial node in the differential bitmap chain within the new version group, and fill the initial node with the all-zero bitmap as the differential bitmap corresponding to the target complete bitmap.
[0029] Understandably, the working principle of determining the target complete bitmap after the transaction commit corresponding to the bitmap modification record is a process of reconstructing the historical state through logical operations and superimposing the current changes to generate the latest full state. Specifically, an XOR operation is performed using the reference bitmap in the most recently created version group and the differential bitmap at the head of the differential bitmap chain within that version group (i.e., the latest differential version within that group) to restore the complete bitmap before the current transaction commit, which is defined as the intermediate complete bitmap. Subsequently, a bitwise OR operation is performed between the intermediate complete bitmap and the bitmap modification record to merge the current transaction's modification state of the data rows into the intermediate complete bitmap, thereby generating the target complete bitmap representing the latest visibility state of the data after the current transaction commit.
[0030] In one possible implementation, the above-mentioned creation of a new differential bitmap based on the bitmap modification record and insertion of the new differential bitmap into the most recently created version group includes:
[0031] Insert a placeholder node at the head of the differential bitmap chain within the most recently created version group;
[0032] Based on the bitmap modification history, the reference bitmap of the most recently created version group, and the differential bitmap adjacent to the placeholder node, perform logical operations to generate a new differential bitmap;
[0033] The new differential bitmap is filled into the placeholder node—thus completing the operation of inserting the new bitmap version into the most recently created version group.
[0034] Here is an explanation of the differential bitmaps adjacent to the placeholder nodes mentioned above: the differential bitmap at the head of the differential bitmap chain within the most recently created version group before the placeholder node was inserted.
[0035] Understandably, the working principle of performing the above logical operations to generate a new differential bitmap is as follows: logical operations are used to superimpose the modification increment generated by the current transaction onto the existing differential cumulative state. Specifically, logical operations are performed based on the bitmap modification record and the reference bitmap of the most recently created version group to extract the newly added change bit positions of the current transaction relative to the reference bitmap; then, a bitwise OR operation is performed between the newly added change bit positions and the adjacent differential bitmap (which records the cumulative changes of all previous transactions within the version group relative to the reference bitmap), thereby generating a new differential bitmap that aggregates historical change information and current change information, and filling the placeholder nodes to complete the version update.
[0036] One possible implementation also includes:
[0037] Obtain the snapshot information corresponding to the query transaction, and determine the target bitmap version that the query transaction is allowed to access based on the snapshot information;
[0038] Locate the target version group in the reference bitmap chain based on the transaction commit sequence number (CSN) of the target bitmap version;
[0039] Within the differential bitmap chain of the target version group, obtain the target differential bitmap corresponding to the target bitmap version;
[0040] Logical operations are performed on the reference bitmap and the target differential bitmap of the target version group to reconstruct the target bitmap version.
[0041] The data in the columnar storage is filtered based on the target bitmap version, and the filtered query results are returned to the query transaction.
[0042] This section elaborates on the snapshot information corresponding to the query transaction. Snapshot information refers to the time-point status identifier containing the global transaction commit sequence number obtained at the start of the query transaction. It is used to define data visibility boundaries and determine the target bitmap version that the current transaction is allowed to access during multi-version concurrency control.
[0043] One possible implementation also includes:
[0044] When the preset garbage collection triggering conditions are met, the version groups in the hierarchical grouped differential bitmap storage module are traversed to identify bitmap versions that are not referenced by active transactions.
[0045] Release the storage space occupied by bitmap versions that are not referenced by active transactions;
[0046] After completing the reclamation of storage space, the number of differential bitmaps remaining in each version group is checked; if the number of differential bitmaps is lower than the preset merging threshold, the corresponding version group is merged with the adjacent version group.
[0047] Here, we will explain the active events. An active transaction refers to a transaction that is currently in the execution state in the database system and has not yet completed the commit or rollback operation. This includes transactional transactions that perform data modification operations and analytical transactions that perform data query operations.
[0048] Understandably, by traversing the hierarchical grouped differential bitmap storage module built on a doubly linked list, the system accurately identifies and physically releases unreferenced invisible bitmap versions using active transaction reference information, thereby directly removing invalid data to reduce memory usage. Simultaneously, after garbage collection, a merging operation is performed on sparse version groups with a remaining number of differential bitmaps below the merging threshold, integrating sparse and scattered data into adjacent version groups. This eliminates the additional reference bitmap storage overhead caused by retaining too many low-density version groups, and shortens the traversal path of the doubly linked list by reducing the number of version group nodes, thereby improving the execution efficiency of subsequent garbage collection scanning and data access while deeply compressing storage space.
[0049] One possible implementation also includes:
[0050] If the current transaction is performing a data update or delete operation, generate the corresponding bitmap modification record based on the execution information of the current transaction;
[0051] Write bitmap modification records to the bitmap version log module to complete the immediate commit of transactions.
[0052] Secondly, this application provides a bitmap multi-version processing device based on hierarchical grouping differential, including: a bitmap version log recording module, a hierarchical grouping differential bitmap storage module, and a scheduling module;
[0053] The bitmap version log module is used to store bitmap modification records corresponding to transactions;
[0054] The hierarchical grouped differential bitmap storage module is used to store one or more version groups; each version group is configured with a reference bitmap and a differential bitmap; the reference bitmap is stored in the form of a complete bitmap; the differential bitmap is based on the reference bitmap and records the bit positions that have changed relative to the reference bitmap in the corresponding bitmap version.
[0055] The hierarchical grouped differential bitmap storage module is used to determine whether to create a new version group based on the group capacity. If so, a new version group is created based on the bitmap modification record; otherwise, a new differential bitmap is created based on the bitmap modification record, and the new differential bitmap is inserted into the most recently created version group. The group capacity is used to indicate the maximum number of versions in the version group.
[0056] The scheduling module is used to obtain bitmap modification records from the bitmap version log recording module; and to process the bitmap modification records through the hierarchical grouped differential bitmap storage module.
[0057] Thirdly, this application provides an electronic device, including: a memory and one or more processors; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code including computer instructions; the one or more processors invoke the computer instructions to cause the electronic device to perform the method described in the first aspect or any possible implementation of the first aspect.
[0058] It is understood that the beneficial effects of the second and third aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.
[0059] Overall, the technical solutions conceived in this application have the following beneficial effects compared with the prior art:
[0060] (1) This application effectively reduces redundant storage between adjacent bitmap versions by using a hierarchical grouping differential bitmap multi-version storage structure, which significantly reduces the memory usage of bitmap multi-version in columnar storage in the HTAP scenario.
[0061] (2) By introducing a visibility-based garbage collection and version merging mechanism, this application can promptly reclaim invisible bitmap versions (versions not referenced by active transactions) in the presence of long-running analytical transactions, thereby preventing the continuous expansion of the version chain and improving the stability of the system in the long run.
[0062] (3) By logging and asynchronously generating bitmap versions, this application reduces the impact of bitmap version insertion on the transaction execution path and maintains good TP transaction throughput.
[0063] (4) This application compares the HTAP database system integrating HierDiff with the original HTAP database system through experiments, and verifies the advantages of the HTAP database system integrating HierDiff in terms of memory usage, throughput and latency. It also explores the impact of group capacity parameters and grouping threshold parameters on storage efficiency and system performance, indicating that the proposed method has good parameter adjustability and engineering applicability. Attached Figure Description
[0064] Figure 1 This is a schematic diagram of bitmap multi-version processing based on hierarchical grouping differential provided in an embodiment of this application;
[0065] Figure 2 This is a schematic diagram of the hierarchical grouping differential bitmap compression logic provided in an embodiment of this application;
[0066] Figure 3 This is a schematic diagram of the hierarchical grouped differential bitmap storage module provided in an embodiment of this application;
[0067] Figure 4 This is a flowchart of inserting a new version provided in an embodiment of this application;
[0068] Figure 5 This is a schematic diagram illustrating the construction of a new bitmap and its insertion into the most recently created version group, as provided in an embodiment of this application.
[0069] Figure 6 This is a flowchart of the process for querying a specified bitmap version provided in an embodiment of this application;
[0070] Figure 7 This is a flowchart of bitmap-based garbage collection and merging provided in an embodiment of this application;
[0071] Figure 8 This is a schematic diagram of the memory usage experiment under TPC-C load provided in the embodiments of this application;
[0072] Figure 9 This is a schematic diagram of the memory usage experiment under TPC-H load provided in the embodiments of this application;
[0073] Figure 10 This is a schematic diagram of the throughput and latency experiments under TPC-C load provided in the embodiments of this application;
[0074] Figure 11 This is a schematic diagram of the throughput and latency experiments under TPC-H load provided in the embodiments of this application;
[0075] Figure 12 This is a schematic diagram of the combined threshold parameter experiment under TPC-H load provided in the embodiments of this application;
[0076] Figure 13This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation
[0077] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0078] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.
[0079] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.
[0080] First, the technical terms used in the embodiments of this application will be introduced.
[0081] Transactional transactions (TP transactions): These are transactions that primarily involve insert, delete, and update operations, have short execution times, and high concurrency.
[0082] Analytical transactions (AP transactions): These are transaction types that primarily involve large-scale scanning, aggregation, and statistical analysis operations and have long execution times.
[0083] Compression Unit (CU): The basic storage unit used to organize data in columnar storage. Each CU corresponds to a set of bitmap versions. A set of bitmap versions is maintained for each CU. These bitmaps record the visibility state of data rows (if the visibility state is visible, the corresponding data row can be accessed; if the visibility state is invisible, the corresponding data row cannot be accessed). This allows for the avoidance of directly modifying the original stored data when data is deleted or modified. Instead, by recording multiple historical states of the data through a set of bitmap versions, columnar storage provides the foundation for implementing Multi-Version Concurrency Control (MVCC), enabling the system to efficiently determine data visibility and thus ensure transaction consistency and high concurrency performance without blocking read and write operations.
[0084] Reference bitmap: A complete bitmap selected within a version group, used as a reference for differential encoding of other bitmap versions within the group.
[0085] Differential bitmap: A bitmap version that records the bit information that has changed relative to a reference bitmap, stored in a sparse format.
[0086] Version group: A logical grouping of multiple consecutive bitmap versions according to the transaction commit order. Each version group contains a reference bitmap and several differential bitmaps.
[0087] Group size: The maximum number of versions in a version group (when merging or combining versions, there is no limit to the group size).
[0088] Merge: After garbage collection, merge sparse version groups into adjacent version groups.
[0089] Merge threshold: The number of versions within a group that triggers a merge operation.
[0090] CSN (Commit Sequence Number): Transaction commit sequence number used to identify the bitmap version.
[0091] CU_id: CU (Compression Unit) identifier.
[0092] Row_id: Row identifier.
[0093] This application proposes a bitmap multi-version storage and reclamation scheme (called HierDiff) based on hierarchical grouping differential, and integrates it into the HTAP database system, specifically including:
[0094] Based on the HTAP database system, the multi-version concurrency control mechanism in columnar storage with CU as the storage unit is optimized, and a bitmap multi-version management overall structure based on hierarchical grouping differential is constructed.
[0095] To address the high similarity between adjacent bitmap versions, multiple bitmap versions are divided into multiple version groups. Within each version group, the earliest version is selected as the reference bitmap, and the remaining versions are stored in the form of differential bitmaps. This hierarchical structure enables a compressed representation of bitmap versions.
[0096] To address the issue of delayed garbage collection caused by long-running analytical transactions, invisible bitmap versions (versions not referenced by active transactions) are identified based on the fact that active transactions reference bitmap versions (when an active transaction references a bitmap version, it means that the bitmap version is visible to the active transaction). A combination and merge operation is performed on version groups with fewer versions than the combination and merge threshold to control the version size and maintain differential efficiency.
[0097] By introducing a bitmap version logging mechanism, the generation and insertion process of bitmap versions is decoupled from the transaction commit path, thereby reducing transaction execution latency;
[0098] The group capacity and the grouping threshold are theoretically proven, and their accuracy is verified through experiments.
[0099] Based on the above structure, a bitmap version insertion control mechanism suitable for multi-transaction concurrent environments is designed, and basic database operation processes such as insertion, deletion, update and query are provided.
[0100] Understandably, this application aims to address the problems in HTAP database systems where bitmap-based multi-version concurrency control mechanisms suffer from continuously increasing bitmap versions, significantly increased memory consumption, and decreased garbage collection efficiency when long-running analytical transactions (APs) are present. Furthermore, to address the storage redundancy caused by existing bitmap multi-version storage methods that typically use complete bitmaps and fail to fully utilize the high similarity between adjacent versions, this application proposes an efficient bitmap multi-version storage and garbage collection scheme.
[0101] The technical solution of this application enables compressed storage and efficient recycling of bitmap versions while ensuring the semantic correctness of multiple versions, thereby improving the memory utilization efficiency and system stability of the following formula storage engine in HTAP scenarios.
[0102] The embodiments of this application are described below with reference to the accompanying drawings.
[0103] Figure 1 This is a schematic diagram of bitmap multi-version processing based on hierarchical grouping differential provided in an embodiment of this application, such as... Figure 1 As shown, this application optimizes the multi-version concurrency control mechanism in columnar storage based on the HTAP database system. The overall structure includes a hierarchical grouped differential bitmap storage module, a bitmap version garbage collection and merging module, and a bitmap version logging module. These modules cooperate with each other to achieve efficient management and recycling of bitmap multiple versions.
[0104] In the transaction write path, when the database performs an insert operation, the system initializes the bitmap doubly linked list structure corresponding to the compression unit while creating a new CU. When performing a delete or update operation, an immediate commit mechanism is used to store the corresponding bitmap modification record (a record reflecting the modification in bitmap form) in the bitmap version log module (the smaller the CSN, the earlier the corresponding transaction occurred), without immediately generating a new bitmap version. The background thread periodically scans the bitmap version log module, generates a new bitmap version from top to bottom (in ascending order of CSN), and inserts the generated bitmap version into the hierarchical grouped differential bitmap storage module, thereby achieving asynchronous maintenance of bitmap versions.
[0105] In the query path, the analytical query transaction directly accesses the hierarchical grouped differential bitmap storage module, determines the target bitmap version that the query transaction can see ("visible" also "allowed to access") based on the transaction snapshot information, traverses the corresponding bitmap version chain, and reconstructs the target bitmap version to complete data filtering and query execution.
[0106] In addition, the background thread will periodically trigger the bitmap version garbage collection and merging module to collect invisible old versions (versions not referenced by active transactions) and merge sparse version groups in order to control the number of bitmap versions and maintain the stability of the system in the long term.
[0107] Figure 2 This is a schematic diagram of the hierarchical grouping differential bitmap compression logic provided in the embodiments of this application, such as... Figure 2 As shown, this application uses differential encoding to compress and store bitmap versions based on the characteristics of sparse updates and high similarity between multiple bitmap versions.
[0108] Specifically, the bitmaps stored in the prior art are as follows: Figure 2 The complete bitmap is shown below. The "complete bitmap" records which rows in the original data were modified (deleted or updated) from the time the original data was stored (column-oriented) until the corresponding transaction was committed. When subsequent events execute queries against the data version described by this "complete bitmap," the modified rows in the original data are not visible to subsequent events, while the unmodified rows in the original data are visible to subsequent events.
[0109] For example, such as Figure 2 As shown, for a complete bitmap with CSN 2, the 2nd and 4th bits of the complete bitmap store "1", and the 0th, 1st, 3rd, and 5th bits store "0". This complete bitmap indicates that during the period from when the original data was stored to when the transaction with CSN 2 was committed, the 2nd and 4th data rows in the original data were modified (deleted or updated), while the 0th, 1st, 3rd, and 5th data rows were not modified.
[0110] When an event with CSN 3 performs a query against the data version described by the complete bitmap with CSN 2, rows 2 and 4 of the original data are not visible to the event with CSN 3, while rows 0, 1, 3, and 5 of the original data are visible to the event with CSN 3.
[0111] This application uses a complete bitmap (e.g.) Figure 2The earliest complete bitmap in the version group (i.e., the complete bitmap with CSN of 2) is used as the reference bitmap. By performing an XOR operation between each complete bitmap in the version group and the reference bitmap, a differential bitmap is formed. The differential bitmap is used to record the bit positions that have changed in subsequent versions relative to the reference bitmap. By recording the information of the changed positions, the differential bitmap achieves a compact representation of the bitmap version.
[0112] As transactions are continuously committed, the number of bitmap versions gradually increases. If the same reference bitmap is always used as the differential base, the amount of change information recorded in the differential bitmap will gradually increase, thus affecting the compression effect. To address this, this application proposes a hierarchical grouping differential compression logic. By dividing the bitmap versions into multiple groups and selecting the earliest version within each group as the reference bitmap for all versions, the differential range is limited, ensuring that each differential bitmap contains only a small amount of change information.
[0113] Through the compression logic described above, this application can fully utilize the high similarity between adjacent versions while ensuring the semantic correctness of bitmap versions, thereby reducing the overall storage overhead of multiple bitmap versions and providing a foundation for subsequent rapid version reconstruction.
[0114] Figure 3 This is a schematic diagram of the hierarchical grouped differential bitmap storage module provided in an embodiment of this application, as shown below. Figure 3 As shown, the hierarchical grouped differential bitmap storage module adopts a two-level linked list structure, including multiple version groups and differential bitmap chains located within each version group.
[0115] Version groups are organized chronologically or in transaction commit order. Each version group includes a reference bitmap and several differential bitmaps. The reference bitmap is stored as a complete bitmap and serves as the basis for constructing the differential bitmaps within that version group. The differential bitmaps are based on the reference bitmap and only record the bit information that has changed in the corresponding version. By restricting differential bitmaps to differential encoding only within their respective version groups, this application can maintain the sparsity of the differential bitmaps, thereby achieving stable compression results.
[0116] Furthermore, for each reference bitmap, maintain its associated version range information (specifically, the version range information is...). Figure 3 The CSN range shown represents the transaction commit range covered by the version group and is connected to the corresponding differential bitmap chain via a pointer structure. During the query process, the system only needs to reconstruct the target version based on the reference bitmap and a single differential bitmap, thus achieving a good balance between compression efficiency and version reconstruction overhead. This is suitable for high-concurrency transaction processing and long-running analytical query scenarios.
[0117] Figure 4 This is a flowchart of the process for inserting a new version provided in an embodiment of this application, such as... Figure 4 As shown, the process of inserting a new version includes the following steps S101 to S104.
[0118] Step S101: Generate bitmap modification records and write them to the log. When the database system performs a delete or update operation, it generates corresponding bitmap modification records based on the execution information of the current transaction and writes the bitmap modification records to the bitmap version log module to complete the immediate commit of the transaction.
[0119] Step S102: Background scan of the bitmap version log module. After the transaction is committed, at least one background thread scans the bitmap version log module to read unprocessed bitmap modification records in order to obtain the modification information required for the new bitmap version to be generated.
[0120] Step S103: Determine the capacity of the first group. That is, determine whether the first group at the head of the current doubly linked list has reached its capacity.
[0121] Step S104: When the first group has not reached its capacity, create an empty placeholder node in the differential bitmap within the first group, and pre-insert the placeholder node into the corresponding position (at the head of the differential bitmap chain) to preserve the sequential position of the new version of the differential bitmap in the differential bitmap; then as follows Figure 5 As shown, the modification record is based on the bitmap (e.g.) Figure 5 The bitmap modification record corresponding to the transaction with CSN 16 (where bit 0 is "1", indicating that the transaction with CSN 16 modified the 0th row of the original data) is ORed with the reference bitmap in the first group (to ensure the bitmap after the OR operation contains the modification record represented by the reference bitmap), resulting in the ORed bitmap; then, an XOR operation is performed on the bitmap after the OR operation and the reference bitmap, resulting in the XORed bitmap; finally, the XORed bitmap is used to compare the bitmap with the differential bitmap adjacent to the placeholder node in the first group (e.g., ...). Figure 5 The differential bitmap corresponding to the transaction with CSN 15 is subjected to an OR operation (so that the bitmap after the OR operation has the modification records represented by the adjacent differential bitmaps) to obtain a new version of the differential bitmap, and then the placeholder nodes are filled with the new version of the differential bitmap.
[0122] When the first group has reached its capacity, based on the bitmap modification record, the reference bitmap of the first group, and the differential bitmap with the largest CSN in the first group (the differential bitmap located at the head of the chain in the differential bitmap layer within the first group), the complete bitmap after the transaction commit corresponding to the bitmap modification record is determined, a new version group is created, and the corresponding reference bitmap is generated in the new version group (the determined complete bitmap is used as the reference bitmap). At the same time, placeholder nodes are created and inserted in the differential bitmap layer within the new version group, and the placeholder nodes are filled with 0 bits (the result of the XOR operation between the determined complete bitmap and the reference bitmap is a bitmap with all 0 bits).
[0123] Figure 6 This is a flowchart of the process for querying a specified bitmap version provided in an embodiment of this application, such as... Figure 6 As shown, the process of querying a specified bitmap version includes the following steps S201 to S206.
[0124] Step S201: Receive query request and obtain transaction snapshot. When the database system receives a query request, it obtains the snapshot information corresponding to the current query transaction to determine the target bitmap version that the query transaction can see (i.e., "visible" or "accessible").
[0125] Step S202: Locate the target version group. Based on the transaction snapshot information, the system locates the version group (target version group) corresponding to the target bitmap version in the hierarchical grouped differential bitmap storage module.
[0126] Step S203: Obtain the reference bitmap and the differential bitmap. After locating the version group, the system obtains the reference bitmap in that version group and the differential bitmap corresponding to the target bitmap version (as the target differential bitmap).
[0127] Step S204: Reconstruct the target bitmap version. The system generates the target bitmap version by combining and reconstructing the reference bitmap and the target differential bitmap.
[0128] Step S205: Execute the query based on the target bitmap version. The system filters the data in the columnar storage based on the target bitmap version to complete the query operation.
[0129] Step S206: Return query results. After completing the data filtering, the query results are returned to the query transaction, completing the bitmap version query process.
[0130] Figure 7 This is a flowchart of bitmap-based garbage collection and merging provided in an embodiment of this application, such as... Figure 7 As shown, the bitmap version garbage collection and merging process includes the following steps S301 to S306.
[0131] Step S301: Determine if garbage collection is triggered. A background thread determines whether the preset garbage collection triggering conditions are met (e.g., reaching a memory threshold, system idle time, background timed triggering, etc.). When the triggering conditions are met, the bitmap version garbage collection and merging process is started.
[0132] Step S302: Scan the active transaction table. After the garbage collection process starts, a background thread scans the active transaction table in the system to obtain the bitmap version information referenced by all currently active transactions.
[0133] Step S303: Traverse version groups. The system traverses each version group in the hierarchical grouped differential bitmap storage module and performs bitmap version check and maintenance operations on each version group in sequence.
[0134] Step S304: Determine whether the bitmap version is referenced by an active transaction. For each bitmap version in the current version group, the system determines whether the bitmap version is referenced by any active transaction (if an active transaction references a bitmap version, it means that the bitmap version is visible to that active transaction); bitmap versions not referenced by any active transaction are identified as versions not visible to active transactions (or versions not referenced by active transactions).
[0135] Step S305: Reclaim bitmap versions that are not visible to active transactions. For bitmap versions identified as not visible to active transactions in step S304, a background thread performs a reclamation operation to release the storage space they occupy.
[0136] Step S306: Perform version group merging. After completing bitmap version recycling, the system performs merging on the current version group. When the number of remaining differential bitmaps in the version group is lower than a preset threshold, the version group is merged with adjacent version groups (during merging, it is not necessary to consider whether the number of versions in the merged group exceeds the group capacity), and the reference bitmap and differential bitmap structures are reconstructed. After completing the recycling and merging operations of all version groups, the current round of bitmap version garbage collection and merging process ends, and the background thread enters a waiting state.
[0137] The experiment compares the HTAP database system integrated with HierDiff with the original HTAP database system.
[0138] The experimental setup is as follows:
[0139] Baseline load: Two typical database benchmark loads, TPC-C and TPC-H, are used to represent transactional and analytical loads, respectively, with the data size set to 10GB for both.
[0140] HTAP scenario construction: Based on the TPC-H benchmark, update and delete operations are introduced to simulate the HTAP scenario;
[0141] TPC-C Configuration: Use the BenchmarkSQL framework to execute TPC-C workloads, setting up 100 concurrent terminals to run transactions simultaneously;
[0142] TPC-H configuration: Set 20 terminals to perform analytical queries and 40 terminals to perform transactional operations, with 20 terminals used for update operations and 20 terminals used for delete operations;
[0143] Comparison scheme: Compare the original HTAP database system (hereinafter referred to as the first database system) with the HTAP database system integrated with HierDiff in this application (hereinafter referred to as the second database system).
[0144] Parameter settings: Under the default configuration, the version group capacity parameter is set to 25, and the merging threshold is set to 13; in the parameter sensitivity experiment, the above parameters are adjusted to analyze their impact.
[0145] Evaluation metrics: storage overhead of bitmap multi-version, system throughput and latency performance during transaction processing.
[0146] The memory usage experiment is as follows:
[0147] The multi-version storage and reclamation method designed in this application significantly reduces the storage overhead caused by multiple bitmap versions in terms of memory usage. By introducing a hierarchical grouped differential bitmap storage structure and combining it with a visibility-based version reclamation and merging mechanism, this application can effectively reduce redundant storage between adjacent bitmap versions, avoiding the continuous expansion of the number of bitmap versions under long-running analytical transactions. This application was verified under TPC-C and TPC-H loads, testing the total bitmap size and the number of bitmap versions of a single doubly linked list, and obtaining the average version size by dividing the two. Figure 8 and Figure 9 As shown, experimental results demonstrate that, compared to schemes without a hierarchical differential mechanism, this application can significantly reduce the overall memory footprint of the column-store bitmap while maintaining the semantic correctness of multiple versions.
[0148] Throughput and latency experiments are as follows:
[0149] The multi-version storage and reclamation method designed in this application demonstrates excellent performance in terms of system throughput and transaction latency. By decoupling the bitmap version generation and insertion process from the transaction commit path, this application effectively mitigates the impact of bitmap version insertion on transaction execution in high-concurrency update scenarios. Through a doubly linked list structure, this application reduces the number of pointer jumps during the query process. Figure 10 and Figure 11As shown, experimental results demonstrate that this application can maintain stable transaction throughput and high query efficiency during the analysis query execution process without introducing significant additional latency.
[0150] The version group capacity parameter experiment is as follows:
[0151] The multi-version storage and reclamation method designed in this application verifies the impact of version group capacity parameters on system performance and storage efficiency through experiments. Experimental results show that the hierarchical grouping differential structure proposed in this application has good adjustability and engineering applicability in terms of parameter configuration.
[0152] The experiment involving combining threshold parameters is as follows:
[0153] The multi-version storage and recycling method designed in this application, such as Figure 12 As shown, experiments were conducted to verify the impact of the merge threshold parameter on the long-term stability of the system. The experiments recorded the increase in the average number of versions after garbage collection under different merge thresholds; a smaller increase indicates a better merge effect. The version merge mechanism helps maintain the stability of the bitmap multi-version structure under different load conditions, thereby improving the overall reliability of the system.
[0154] The bitmap multi-version processing apparatus based on hierarchical grouping difference provided in this application is described below. The bitmap multi-version processing apparatus based on hierarchical grouping difference described below can be referred to in correspondence with the bitmap multi-version processing method based on hierarchical grouping difference described above.
[0155] This application also provides a bitmap multi-version processing device based on hierarchical grouping differential, including: a bitmap version logging module, a hierarchical grouping differential bitmap storage module, and a scheduling module. Wherein:
[0156] The bitmap version log module is used to store bitmap modification records corresponding to transactions;
[0157] The hierarchical grouped differential bitmap storage module is used to store one or more version groups; each version group is configured with a reference bitmap and a differential bitmap; the reference bitmap is stored in the form of a complete bitmap; the differential bitmap is based on the reference bitmap and records the bit positions that have changed relative to the reference bitmap in the corresponding bitmap version.
[0158] The hierarchical grouped differential bitmap storage module is used to determine whether to create a new version group based on the group capacity. If so, a new version group is created based on the bitmap modification record; otherwise, a new differential bitmap is created based on the bitmap modification record, and the new differential bitmap is inserted into the most recently created version group. The group capacity is used to indicate the maximum number of versions in the version group.
[0159] The scheduling module is used to obtain bitmap modification records from the bitmap version log recording module; and to process the bitmap modification records through the hierarchical grouped differential bitmap storage module.
[0160] It is understood that the detailed functional implementation of each of the above units / modules can be found in the description in the aforementioned method embodiments, and will not be repeated here.
[0161] It should be understood that the above-described device is used to execute the methods in the above embodiments. The implementation principle and technical effect of the corresponding program modules in the device are similar to those described in the above methods. The working process of the device can be referred to the corresponding process in the above methods, and will not be repeated here.
[0162] Based on the methods in the above embodiments, this application provides an electronic device, such as... Figure 13 As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute the methods in the above embodiments.
[0163] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.
[0164] Based on the methods in the above embodiments, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to execute the methods in the above embodiments.
[0165] Based on the methods in the above embodiments, this application provides a computer program product that, when run on a processor, causes the processor to execute the methods in the above embodiments.
[0166] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.
[0167] The method steps in this application embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.
[0168] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).
[0169] It is understood that the various numerical designations used in the embodiments of this application are merely for the convenience of description and are not intended to limit the scope of the embodiments of this application.
[0170] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.
Claims
1. A bitmap multi-version processing method based on hierarchical grouping difference, characterized in that, include: Get bitmap modification records from the bitmap version log module, which is used to store bitmap modification records corresponding to transactions; The bitmap modification records are processed through the hierarchical grouping differential bitmap storage module; The hierarchical grouped differential bitmap storage module is used to store one or more version groups; each version group is configured with a reference bitmap and a differential bitmap; the reference bitmap is stored in the form of a complete bitmap; the differential bitmap is based on the reference bitmap and records the bit positions that have changed relative to the reference bitmap in the corresponding bitmap version. The hierarchical grouped differential bitmap storage module is also used to determine whether to create a new version group based on the group capacity; if so, a new version group is created based on the bitmap modification record; otherwise, a new differential bitmap is created based on the bitmap modification record, and the new differential bitmap is inserted into the most recently created version group; the group capacity is used to indicate the maximum number of versions in the version group. The hierarchical grouped differential bitmap storage module is built on a two-level linked list structure, which specifically includes a reference bitmap chain as the first level and a differential bitmap chain as the second level. A reference bitmap chain is formed by connecting reference bitmaps from multiple version groups sequentially according to the transaction commit order. Each version group has an internal configuration of a differential bitmap chain. In any version group, the differential bitmap chain is composed of several differential bitmaps connected in sequence. The reference bitmap is connected to the starting node of the differential bitmap chain within its version group via a pointer structure, and the reference bitmap is associated with version range information, which is used to characterize the transaction commit range covered by the version group.
2. The bitmap multi-version processing method based on hierarchical grouping difference according to claim 1, characterized in that, The determination of whether to create a new version group based on group size includes: Get the number of versions in the most recently created version group, and record it as the number of versions in the most recently created version group; Compare the number of recent version groups with the group size; If the number of versions in the most recent version group reaches the group capacity, a new version group will be created; if the number of versions in the most recent version group does not reach the group capacity, a new version group will not be created.
3. The bitmap multi-version processing method based on hierarchical grouping difference according to claim 1, characterized in that, The creation of a new version group based on bitmap modification records includes: Based on the bitmap modification record, the reference bitmap in the most recently created version group, and the differential bitmap at the head of the differential bitmap chain in the most recently created version group, determine the target complete bitmap after the transaction corresponding to the bitmap modification record is committed. Create a new version group, store the target complete bitmap as the reference bitmap of the new version group, and insert the new version group at the head of the reference bitmap chain; create an initial node in the differential bitmap chain within the new version group, and fill the initial node with the all-zero bitmap as the differential bitmap corresponding to the target complete bitmap.
4. The bitmap multi-version processing method based on hierarchical grouping difference according to claim 1, characterized in that, The step of creating a new differential bitmap based on the bitmap modification record and inserting the new differential bitmap into the most recently created version group includes: Insert a placeholder node at the head of the differential bitmap chain within the most recently created version group; Based on the bitmap modification history, the reference bitmap of the most recently created version group, and the differential bitmap adjacent to the placeholder node, perform logical operations to generate a new differential bitmap; Fill the placeholder nodes with the new differential bitmap.
5. The bitmap multi-version processing method based on hierarchical grouping difference according to claim 1, characterized in that, Also includes: Obtain the snapshot information corresponding to the query transaction, and determine the target bitmap version that the query transaction is allowed to access based on the snapshot information; Locate the target version group in the reference bitmap chain based on the transaction commit sequence number of the target bitmap version; Within the differential bitmap chain of the target version group, obtain the target differential bitmap corresponding to the target bitmap version; Logical operations are performed on the reference bitmap and the target differential bitmap of the target version group to reconstruct the target bitmap version. The data in the columnar storage is filtered based on the target bitmap version, and the filtered query results are returned to the query transaction.
6. The bitmap multi-version processing method based on hierarchical grouping difference according to claim 1, characterized in that, Also includes: When the preset garbage collection triggering conditions are met, the version groups in the hierarchical grouped differential bitmap storage module are traversed to identify bitmap versions that are not referenced by active transactions. Release the storage space occupied by bitmap versions that are not referenced by active transactions; After completing the reclamation of storage space, the number of differential bitmaps remaining in each version group is checked; if the number of differential bitmaps is lower than the preset merging threshold, the corresponding version group is merged with the adjacent version group.
7. The bitmap multi-version processing method based on hierarchical grouping difference according to claim 1, characterized in that, Also includes: If the current transaction is performing a data update or delete operation, generate the corresponding bitmap modification record based on the execution information of the current transaction; Write bitmap modification records to the bitmap version log module to complete the immediate commit of transactions.
8. A bitmap multi-version processing device based on hierarchical grouping difference, characterized in that, include: The module consists of a bitmap version log recording module, a hierarchical grouped differential bitmap storage module, and a scheduling module. The bitmap version log module is used to store bitmap modification records corresponding to transactions; The hierarchical grouped differential bitmap storage module is used to store one or more version groups; each version group is configured with a reference bitmap and a differential bitmap. The reference bitmap is stored in the form of a complete bitmap; the differential bitmap is based on the reference bitmap and records the bit positions that have changed relative to the reference bitmap in the corresponding bitmap version. The hierarchical grouped differential bitmap storage module is also used to determine whether to create a new version group based on the group capacity; if so, a new version group is created based on the bitmap modification record; otherwise, a new differential bitmap is created based on the bitmap modification record, and the new differential bitmap is inserted into the most recently created version group; the group capacity is used to indicate the maximum number of versions in the version group. The scheduling module is used to obtain bitmap modification records from the bitmap version log recording module; and to process the bitmap modification records through the hierarchical grouped differential bitmap storage module. The hierarchical grouped differential bitmap storage module is built on a two-level linked list structure, which specifically includes a reference bitmap chain as the first level and a differential bitmap chain as the second level. A reference bitmap chain is formed by connecting reference bitmaps from multiple version groups sequentially according to the transaction commit order. Each version group has an internal configuration of a differential bitmap chain. In any version group, the differential bitmap chain is composed of several differential bitmaps connected in sequence. The reference bitmap is connected to the starting node of the differential bitmap chain within its version group via a pointer structure, and the reference bitmap is associated with version range information, which is used to characterize the transaction commit range covered by the version group.
9. An electronic device, characterized in that, include: Memory and one or more processors; The memory is coupled to the one or more processors, and the memory is used to store computer program code, the computer program code including computer instructions; The one or more processors invoke the computer instructions to cause the electronic device to perform the method as described in any one of claims 1-7.