A method, medium, and apparatus for real-time updating bitmap indexing

Abstract

This invention discloses a real-time updatable bitmap indexing method, comprising: inputting the content to be searched; finding the value of the content to be searched through the corresponding bitmap index string; acquiring a lock according to the operation, modifying the database, and setting a timestamp; incrementing the global timestamp by 1, releasing the lock, so that when the system performs a UDI operation, a new query operation can be performed without waiting for the UDI operation to complete. This solves the problem that this technology cannot be parallelized, enabling update, insert, and delete operations to not block newly arriving query operations, providing a high-performance query operation. Experimental demonstration shows that the technology proposed in this patent can significantly reduce the latency of bitmap indexing operations in parallel and improve the throughput of the entire system.

Description

Technical Field

[0001] This invention relates to the field of bitmap indexing technology, specifically to a bitmap indexing method, medium, and apparatus that can be updated in real time. Background Technology

[0002] In database management systems (DBMS), data retrieval is a frequently performed operation. To reduce query latency and improve query efficiency, indexes have emerged. An index is to a DBMS what a table of contents is to a book; by searching the table of contents, one can quickly jump to the corresponding content instead of flipping through the entire book. In a database system, indexing technology allows specific records to be found directly from a table without sequentially scanning the entire table, thus significantly improving query performance. Based on the above, it is clear that indexes play a crucial role in database management systems.

[0003] Bitmap indexes are a common type of index, supported by mainstream commercial and open-source databases such as Oracle, DB2, and PostgreSQL.

[0004] Compared to other common indexes, bitmap indexes have unique advantages. Because a bitmap index creates a corresponding bitvector for different values, this mechanism can directly select the value to be queried during multi-value queries, filtering out other useless information and thus improving query efficiency. Furthermore, with the support of modern hardware, computers are highly efficient at processing bitwise logical operations, making bitmap indexes well-suited for handling query-intensive operations. Especially when dealing with large-scale datasets, the query performance advantage of bitmap indexes is significant, making them commonly used in large data warehouses and scientific data analysis.

[0005] However, with the widespread application of bitmap indexing technology, its problems have gradually become apparent—specifically, the update issue. Even mature, large-scale commercial databases like Oracle rely on inefficient delete and rebuild methods. To address this, many new technologies have been proposed. For example, Canahuate's UCB technology solves the update problem by maintaining an ExistenceBitmap. Currently, the most cutting-edge technology in this field is UpBit technology proposed by Athanassoulis. Athanassoulis's UB and fencepointer technologies have also solved the aforementioned problems.

[0006] However, with the development of multi-core technology, the industry's demand for parallel operations suitable for multi-core processors is growing. Currently, bitmap indexing technology is only suitable for sequential access and does not support parallel execution between UDI operations and query operations. That is, when a UDI operation is being executed, the query operation cannot be executed, which greatly reduces system availability and can cause serious data inconsistency problems.

[0007] While parallelization can be achieved using reader-writer locks, this introduces the problem of lock usage. Locking causes UDI operations to block query operations; that is, when a UDI operation needs to modify a bitvector, it locks that bitvector. Other query operations accessing this bitvector must wait for the UDI operation to finish and release the lock. This mechanism introduces significant latency, which is unacceptable for systems with frequent queries. Furthermore, as the number of cores increases, the scalability issues associated with lock usage become increasingly apparent, leading to even higher latency and lower throughput. Summary of the Invention

[0008] The purpose of this section is to outline some aspects of embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0009] In view of the above-mentioned problems, the present invention is proposed.

[0010] Therefore, the technical problem solved by this invention is that existing bitmap indexing methods have large latency and low throughput, and that the use of locks can cause UDI operations to block queries.

[0011] To solve the above-mentioned technical problems, the present invention provides the following technical solution: a bitmap indexing method that can be updated in real time, comprising:

[0012] Enter the content you want to search for;

[0013] The value of the content to be searched is found by using the corresponding bitmap index string;

[0014] Acquire the lock based on the operation, then modify the database and set a timestamp;

[0015] The global timestamp is incremented by 1, the lock is released, and the system can perform queries without waiting for the UDI operation to complete when a new query operation arrives while performing a UDI operation.

[0016] As a real-time updatable bitmap indexing method according to the present invention, the method is characterized in that: the value retrieval includes: establishing a corresponding bitmap index string, and finding the value through the corresponding bitmap index string according to the content to be searched.

[0017] As a real-time updatable bitmap indexing method according to the present invention, the use of the lock includes:

[0018] When a UDI operation needs to modify a bitmap index string, lock the bitmap index string.

[0019] When other query operations need to access this bitmap index string, they wait for the UDI operation to finish and release the lock.

[0020] As a real-time updatable bitmap indexing method according to the present invention, the UDI operation includes: update operation, deletion operation and insertion operation;

[0021] When a query transaction is committed to the system, the system checks its timestamp;

[0022] The system will delete all obsolete versions in the chain that are less than or equal to this timestamp.

[0023] As a real-time updatable bitmap indexing method according to the present invention, the update operation further includes:

[0024] Find the value before the update based on the line number, and denote it as old_value;

[0025] Find the bitmap index string corresponding to the version based on old_value, the updated value new_value, and the operation timestamp;

[0026] Request and acquire the lock, copy the bitmap index strings of the corresponding versions of old_value and new_value, modify the value of the corresponding row number, and set its timestamp;

[0027] Insert the bitmap index strings of old_value and new_value into the corresponding chains respectively;

[0028] Increment the global timestamp by 1 and release the lock.

[0029] As a bitmap indexing method that can be updated in real time according to the present invention, the deletion operation further includes:

[0030] Find the value to be deleted based on the line number, and denote it as delete_value;

[0031] Find the bitmap index string of the corresponding version based on delete_value and operation timestamp;

[0032] Request the lock, copy the bitmap index string, modify the value of the corresponding row number, and set its timestamp;

[0033] Insert the new bitmap index string of delete_value into the delete_value chain;

[0034] Increment the global timestamp by 1 and release the lock.

[0035] As a real-time updatable bitmap indexing method according to the present invention, the insertion operation further includes:

[0036] Request and acquire the lock, copy the bitmap index string of the corresponding version of the value to be inserted (insert_value), modify the value of the new row number, and set its timestamp;

[0037] Copy the bitmap index string of the corresponding version of the remaining values, modify the value of the new row number, and set its timestamp;

[0038] Insert all new bitmap index strings into their corresponding chains;

[0039] Increment the global row count variable N_ROWS by 1;

[0040] Increment the global timestamp by 1 and release the lock.

[0041] As a bitmap indexing method that can be updated in real time according to the present invention, it is characterized by:

[0042] The query operation also includes:

[0043] Based on the query_value and the timestamp of the query operation, traverse the bitmap index string chain to find the corresponding version of the bitmap index string;

[0044] Iterate through this bitmap index string and return the results.

[0045] A computer device includes: a memory and a processor; the memory stores a computer program, characterized in that: when the processor executes the computer program, it implements the steps of the method described in any one of the present invention.

[0046] A computer-readable storage medium having a computer program stored thereon, characterized in that: when the computer program is executed by a processor, it implements the steps of the method described in any one of the present invention.

[0047] The beneficial effects of this invention are as follows: This patent provides an innovative method that combines bitmap indexing technology with multi-version concurrency control technology using timestamps. It improves bitmap indexing technology, solving the problem of its inability to operate in parallel, and ensures that update, insert, and delete operations do not block newly arriving query operations, providing a high-performance query operation. Experimental results demonstrate that the technology proposed in this patent can significantly reduce the latency of parallel bitmap indexing operations and improve the overall system throughput. Attached Figure Description

[0048] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0049] Figure 1 The overall flowchart of a real-time updatable bitmap indexing method provided in the first embodiment of the present invention;

[0050] Figure 2 A diagram illustrating the concurrency issues faced by bitmap indexing in a real-time updatable bitmap indexing method provided in the first embodiment of the present invention;

[0051] Figure 3 This is a basic structural diagram of a real-time updatable bitmap indexing method provided in the first embodiment of the present invention;

[0052] Figure 4 The first embodiment of the present invention provides a real-time updatable bitmap indexing method, with an update operation flowchart.

[0053] Figure 5 The first embodiment of the present invention provides a real-time updatable bitmap indexing method in the flowchart of the deletion operation;

[0054] Figure 6 The first embodiment of the present invention provides a real-time updatable bitmap indexing method in the insertion operation flowchart;

[0055] Figure 7 The first embodiment of the present invention provides a real-time updatable bitmap indexing method with a query operation flowchart;

[0056] Figure 8 A bitmap indexing method that can be updated in real time is provided as a second embodiment of the present invention. Detailed Implementation

[0057] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0058] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0059] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0060] This invention is described in detail with reference to the schematic diagrams. When detailing the embodiments of this invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not adhering to the usual scale. Furthermore, the schematic diagrams are merely examples and should not be construed as limiting the scope of protection of this invention. In actual fabrication, the three-dimensional spatial dimensions of length, width, and depth should be included.

[0061] Furthermore, in the description of this invention, it should be noted that the terms "upper," "lower," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used solely for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. In addition, the terms "first," "second," or "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0062] Unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" in this invention should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; similarly, they can refer to mechanical connections, electrical connections, or direct connections, or indirect connections through an intermediate medium, or internal connections between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0063] Example 1

[0064] Reference Figure 1-7As an embodiment of the present invention, a bitmap indexing method that can be updated in real time is provided, comprising:

[0065] S1: Enter the content you want to search for;

[0066] Create a corresponding bitmap index string, and find the value by using the corresponding bitmap index string to find the content you need.

[0067] Furthermore, the value of the content to be searched can be found through the corresponding bitmap index string;

[0068] It should be noted that the bitmap index is built based on a specific attribute of the table, which has several different values; for each value, a corresponding bitmap index string bitvector is built.

[0069] S2: Acquire the lock according to the operation, then modify the database and set a timestamp;

[0070] Furthermore, the use of read-write locks can cause UDI operations to block query operations.

[0071] It should be noted that when a UDI operation modifies a bitmap index string, it will lock the bitmap index string; when other query operations need to access this bitmap index string, they can only wait for the UDI operation to finish and release the lock.

[0072] It should also be noted that UDI operations include: update operations, delete operations, and insert operations.

[0073] After the system has been running for a period of time and has performed multiple insertion and deletion operations, the system's bitvector chain will contain multiple versions of data, but some of this data will no longer be useful. Therefore, this patent designs a method for automatically clearing outdated versions.

[0074] When a query transaction is submitted to the system, the system checks its timestamp; the system will delete all obsolete versions in the chain that are less than or equal to this timestamp to increase system availability, reduce system memory usage, and enable the system to run more stably and for a longer period of time.

[0075] S3: Increment the global timestamp by 1 and release the lock, so that when the system is performing a UDI operation, a new query operation can be performed without waiting for the UDI operation to complete.

[0076] Furthermore, the update operations also include:

[0077] Find the value before the update based on the line number, and denote it as old_value;

[0078] Find the bitmap index string corresponding to the version based on old_value, the updated value new_value, and the operation timestamp;

[0079] Request and acquire the lock, copy the bitmap index strings of the corresponding versions of old_value and new_value, modify the value of the corresponding row number, and set its timestamp;

[0080] Insert the bitmap index strings of old_value and new_value into the corresponding chains respectively;

[0081] Increment the global timestamp by 1 and release the lock.

[0082] The deletion operation also includes:

[0083] Find the value to be deleted based on the line number, and denote it as delete_value;

[0084] Find the bitmap index string of the corresponding version based on delete_value and operation timestamp;

[0085] Request the lock, copy the bitmap index string, modify the value of the corresponding row number, and set its timestamp;

[0086] Insert the new bitmap index string of delete_value into the delete_value chain;

[0087] Increment the global timestamp by 1 and release the lock.

[0088] Insertion operations also include:

[0089] Request and acquire the lock, copy the bitmap index string of the corresponding version of the value to be inserted (insert_value), modify the value of the new row number, and set its timestamp;

[0090] Copy the bitmap index string of the corresponding version of the remaining values, modify the value of the new row number, and set its timestamp;

[0091] Insert all new bitmap index strings into their corresponding chains;

[0092] Increment the global row count variable N_ROWS by 1;

[0093] Increment the global timestamp by 1 and release the lock.

[0094] It should also be noted that the query operation further includes:

[0095] Based on the query_value and the timestamp of the query operation, traverse the bitmap index string chain to find the corresponding version of the bitmap index string;

[0096] Iterate through this bitmap index string and return the results.

[0097] It should also be noted that the algorithm's key data structures include: BtvDesc, which includes the timestamp when the Bitvector was inserted and a pointer to the Bitvector; global variables, btvs, which includes a radix of cardinality of bitvector chains; a global timestamp TIMESTAMP; N_ROWS, used to record the global row count; and a lock bitmap_lk used to protect all global variables.

[0098] Data Structures:

[0099] structBtvDesc

[0100] intcommit_ts:64;

[0101] Bitvector*btv;

[0102] BtvDesc*next;

[0103] Global variables:

[0104] BtvDesc*btvs[cardinality];

[0105] intTIMESTAMP: 64; / * Global timestamp * /

[0106] intN_ROWS:32; / *Global variable number_of_rows* /

[0107] mutexbitmap_lk;

[0108] Using the above techniques, query operations and UDI operations can be performed in parallel. That is, when a new query operation arrives while the system is executing a UDI operation, the query operation does not need to wait for the UDI operation to complete. The system will find the corresponding version of the bitvector based on the timestamp of the query operation. This technique significantly reduces the latency of query operations and provides high-performance query operations.

[0109] This embodiment also provides a computing device, including a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the method proposed in the above embodiments.

[0110] This embodiment also provides a storage medium on which a computer program is stored, which, when executed by a processor, implements the method proposed in the above embodiments.

[0111] The storage medium proposed in this embodiment belongs to the same inventive concept as the method proposed in the above embodiments. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0112] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory, magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory, magnetic variable memory, ferroelectric memory, phase change memory, graphene memory, etc. Volatile memory can include random access memory or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory or dynamic random access memory, etc. The databases involved in the embodiments provided in this application can include at least one of relational databases and non-relational databases. Non-relational databases can include blockchain-based distributed databases, etc., and are not limited thereto.

[0113] The processors involved in the various embodiments provided in this application may be general-purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited thereto.

[0114] Example 2

[0115] Reference Figure 8 As an embodiment of the present invention, a bitmap indexing method that can be updated in real time is provided. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0116] The throughput and latency of this algorithm are compared with those of traditional methods, and it is also verified that the algorithm has good scalability as the number of cores increases.

[0117] We generated a test dataset with randomly generated data, containing 100 million records, and a bitmap index with a cardinality of 100. For the operations to be validated, 97% were query operations, and 3% were update, delete, and insert operations.

[0118] The obtained throughput is as follows Figure 8As shown in the figure, the horizontal axis represents the change in the number of cores, from 1 to 32, and the vertical axis represents the system throughput, which is an important indicator for measuring the scalability of the algorithm. The lines circled in blue represent the algorithm provided in this patent, while the others represent the latest UpBit algorithm, UCB algorithm, etc. As can be seen from the figure, the values ​​shown by the algorithm proposed in this patent are significantly better than other traditional algorithms. This indicates that the algorithm proposed in this patent can handle more user requests per unit time, greatly improving the availability of the system compared to other traditional algorithms, and also demonstrating the scalability of the algorithm proposed in this patent.

[0119] Table 1 shows the update latency of each algorithm. The first row represents the change in the number of cores, and the first column compares the algorithm proposed in this patent with other algorithms. As can be seen from the table, the update operation latency of the algorithm proposed in this patent is significantly better than that of other algorithms. That is, the algorithm proposed in this patent uses less latency for each update, and there is no obvious performance bottleneck as the number of cores increases. In contrast, other traditional algorithms have poor performance, and the update operation performance drops sharply as the number of cores increases.

[0120] Table 2 shows the insertion latency of each algorithm. The first row represents the change in the number of cores, and the first column compares the algorithm proposed in this patent with other algorithms. As can be seen from the table, the insertion latency of the algorithm proposed in this patent is much lower than that of other traditional algorithms, and it does not decrease sharply with the increase of the number of cores. In contrast, traditional algorithms not only have high latency, but their performance also decreases sharply with the increase of the number of cores.

[0121] Table 3 shows the deletion latency of each algorithm. The first row represents the change in the number of cores, and the first column compares the algorithm of this patent with other algorithms. As can be seen from the table, the deletion operation latency of the algorithm proposed in this patent is much smaller than that of other algorithms. That is, the time required for each deletion is less than that of other algorithms. As the number of cores increases, the time of this patent algorithm increases slowly, indicating that this algorithm has good scalability.

[0122] Table 1 Update Delay Table

[0123]

[0124]

[0125] Table 2 Insertion Delay Table

[0126]

[0127] Table 3 Deletion Delay Table

[0128]

[0129] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A bitmap indexing method that can be updated in real time, characterized in that, include: Enter the content you want to search for; The value of the content to be searched is found by using the corresponding bitmap index string; Acquire the lock based on the operation, then modify the database and set a timestamp; The global timestamp is incremented by 1, the lock is released, and the system can perform a query without waiting for the UDI operation to complete when a new query operation arrives while performing a UDI operation. The value selection includes: establishing the corresponding bitmap index string, and finding the value through the corresponding bitmap index string according to the content to be searched; The use of the lock includes: When a UDI operation needs to modify a bitmap index string, lock the bitmap index string. When other query operations need to access this bitmap index string, they wait for the UDI operation to finish and release the lock; The UDI operations include: update operations, delete operations, and insert operations; When a query transaction is committed to the system, the system checks its timestamp; The system will delete all obsolete versions in the chain that are less than or equal to this timestamp; The update operation also includes: Find the value before the update based on the line number, and denote it as old_value; Find the bitmap index string corresponding to the version based on old_value, the updated value new_value, and the operation timestamp; Request and acquire the lock, copy the bitmap index strings of the corresponding versions of old_value and new_value, modify the value of the corresponding row number, and set its timestamp; Insert the bitmap index strings of old_value and new_value into the corresponding chains respectively; Increment the global timestamp by 1 and release the lock; The deletion operation also includes: Find the value to be deleted based on the line number, and denote it as delete_value; Find the bitmap index string of the corresponding version based on delete_value and operation timestamp; Request the lock, copy the bitmap index string, modify the value of the corresponding row number, and set its timestamp; Insert the new bitmap index string of delete_value into the delete_value chain; Increment the global timestamp by 1 and release the lock.

2. The real-time updatable bitmap indexing method as described in claim 1, characterized in that: The insertion operation also includes: Request and acquire the lock, copy the bitmap index string of the corresponding version of the value to be inserted (insert_value), modify the value of the new row number, and set its timestamp; Copy the bitmap index string of the corresponding version of the remaining values, modify the value of the new row number, and set its timestamp; Insert all new bitmap index strings into their corresponding chains; Increment the global row count variable N_ROWS by 1; Increment the global timestamp by 1 and release the lock.

3. The real-time updatable bitmap indexing method as described in claim 2, characterized in that: The query operation also includes: Based on the query_value and the timestamp of the query operation, traverse the bitmap index string chain to find the corresponding version of the bitmap index string; Iterate through this bitmap index string and return the results.

4. A computer device, comprising: Memory and processor; The memory stores a computer program, characterized in that: when the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 3.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 3.

Images