A method and system for realizing hash index persistence based on value log
By combining high-speed sequential writing of value log technology with key-value integrated disk writing, the problems of write amplification and high hardware cost in hash index persistence are solved, achieving efficient hash index persistence and inexpensive storage.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- JIANGSU DAMENG DATABASE CO LTD
- Filing Date
- 2023-10-18
- Publication Date
- 2026-07-24
AI Technical Summary
Hash index persistence suffers from write amplification and high hardware costs, especially the data redundancy caused by multiple writes and the high hardware costs.
The method employs value log technology combined with high-speed sequential write and key-value integrated disk writing. By pointing the value log through LSN address, redundant data storage is reduced, write amplification is mitigated, and hash index persistence is achieved using inexpensive storage hardware.
It effectively solves the write amplification problem of hash index persistence, reduces hardware costs, and maintains insertion and query efficiency, making it suitable for inexpensive storage hardware.
Smart Images
Figure CN117271523B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data storage and retrieval technology, and in particular to a method and system for persistent hash indexing based on value logs. Background Technology
[0002] There are currently several solutions for hash index persistence in the industry, such as implementing the HAMT (Hash Array Mapped Trie) algorithm and using persistent memory. However, some problems may be encountered in the implementation process: 1) Write amplification (multiple writes or redundant data storage): Hash index persistence requires storing the key value and the disk offset of the actual data, which can lead to redundant data storage. In addition, when inserting data into the system table, the data updated to the hash index needs to be written to disk after the actual data is written to disk, which can lead to multiple writes. 2) High hardware costs: Persistent memory is a new type of storage device, and the purchase price of hardware is relatively high, resulting in increased usage costs. Summary of the Invention
[0003] The technical problem to be solved by this invention is to provide a method and system for hash index persistence based on value logs, which solves the problem of hash index persistence write amplification by combining the advantages of high-speed sequential writing of value log technology and key-value integrated disk writing.
[0004] To address the aforementioned technical problems, this invention provides a method for implementing hash index persistence based on value logs, comprising the following steps:
[0005] Step 1: Create a hash index; When a user initiates a request to create a hash index, the database system, based on value log technology, internally traverses the data table records, calculates the hash value according to the data type of the key, compares the key values, and rebuilds the hash index.
[0006] Step 2: Insert, update, or delete on the hash index; When users perform insert, update, and delete operations on the data table, the database system based on value log technology converts all operations into sequentially written value logs and generates corresponding LSNs. It calculates the hash value according to the data type of the key and compares the key values to insert, update, or delete on the hash index.
[0007] Step 3: Query the hash index; When a user initiates a query request, the database system based on value log technology receives the query request, calculates the hash value according to the data type of the query key, compares the key value, and performs the query.
[0008] Preferably, step 1, creating the hash index, specifically includes the following steps:
[0009] Step 11: The user initiates a request to create a hash index. The database system based on value log technology internally constructs a fixed-size array in memory according to the actual size of the data table, and at the same time starts a background index rebuilding thread, and returns a message to the user that the index was successfully created.
[0010] Step 12: The background index rebuilding thread reads the value log offset address (LSN) of the first record from the data table, and reads the value of the first record from the value log using the LSN.
[0011] Step 13: Determine if this record belongs to the current table. If not, proceed to step 17; otherwise, proceed to step 14.
[0012] Step 14: Retrieve the Key from the read record to be inserted. Calculate the hash value using the corresponding hash function based on the Key's data type. Locate the array slot in memory using the hash value. Sequentially retrieve the LSN value log from the array slot collision list and read the corresponding record value. Compare the Key in the read record with the Key to be inserted.
[0013] Step 15: If a matching key is found and the current record operation is insert or update, then update the current LSN to the LSN of the new record. If the current record operation is delete, remove the LSN from the conflict chain.
[0014] Step 16: If no matching key is found and the current record operation is insert or update, insert the LSN of the new record into the conflict chain. If the current record operation is delete, do nothing.
[0015] Step 17: Read the next log entry. If it exists, proceed to step 13; otherwise, proceed to step 18.
[0016] Step 18: Hash index reconstruction complete.
[0017] Preferably, step 2, inserting, updating, or deleting the hash index, specifically includes the following steps:
[0018] Step 21: When a user performs insert, update, and delete operations on a data table, the value logging system will convert all operations into sequentially written logs and generate corresponding LSNs.
[0019] Step 22: Obtain the Key from the operation record, calculate the hash value using the corresponding hash function based on the Key's data type, locate the array slot in memory using the hash value, retrieve the corresponding record value from the LSN value log of the array slot collision list, and compare the Key in the record with the Key to be inserted.
[0020] Step 23: If a matching key is found and the current record operation is insert or update, then update the current LSN to the LSN of the new record. If the current record operation is delete, remove the LSN from the conflict chain.
[0021] Step 24: If no matching key is found and the current record operation is insert or update, insert the LSN of the new record into the conflict chain. If the current record operation is delete, do nothing.
[0022] Preferably, step 3, querying the hash index, specifically includes the following steps:
[0023] Step 31: The user initiates a query hash index request;
[0024] Step 32: When the database system based on value log technology receives a query request, it calculates the hash value using the corresponding hash function according to the data type of the query key; it locates the array slot in memory using the hash value, and retrieves the corresponding record value from the LSN value log from the array slot collision list in turn, and compares the key in the read record with the key currently being queried.
[0025] Step 33: If a matching key is found, return the current record to the user;
[0026] Step 34: If no matching key is found, no action is taken.
[0027] Preferably, the LSN pointing to the offset of the value log file is implemented as a physical address. During the process of searching for the value log, the LSN is directly used to locate the physical offset address of the value log file, thereby reading the log.
[0028] Preferably, the LSN pointing to the offset of the value log file is implemented as a logical address. By dividing the value log file into pages, the LSN value records the page number and the offset address within the page in the high and low bits, respectively. During the search of the value log, the page number is used to locate the physical block in the value log, and the offset address within the page is used to locate the offset address within the physical block, thereby reading the log.
[0029] Correspondingly, a system for implementing hash index persistence based on value logs includes: a hash index creation module, a hash index operation module, and a hash index query module; in the hash index creation module, the user initiates a hash index creation request, the database system based on value log technology internally traverses the data table records, calculates the hash value according to the data type of the key, compares the key values, and reconstructs the hash index;
[0030] In the hash index operation module, when users perform insert, update, and delete operations on the data table, the database system based on value log technology converts all operations into sequentially written logs and generates corresponding LSNs. It calculates the hash value based on the data type of the key and compares the key values to perform insert, update, or delete operations on the hash index.
[0031] In the hash index query module, when a user initiates a query request for records, the database system based on value log technology receives the query request, calculates the hash value according to the data type of the query key, compares the key value, and performs the query.
[0032] The beneficial effects of this invention are as follows: By combining the advantages of high-speed sequential writing of value log technology and key-value integrated disk writing, this invention transforms the redundant data that originally needed to be stored for hash index persistence into LSN addresses pointing to the value log, thus eliminating the write amplification problem of hash index persistence, while not significantly affecting insertion and query efficiency. Attached Figure Description
[0033] Figure 1 A schematic diagram illustrating the process of creating a hash index for this invention.
[0034] Figure 2 This is a schematic diagram illustrating the process of inserting, updating, or deleting hash indexes according to the present invention.
[0035] Figure 3 This is a schematic diagram illustrating the process of querying a hash index according to the present invention. Detailed Implementation
[0036] A method for implementing hash index persistence based on value logs includes the following steps:
[0037] Step 1: Create a hash index; When a user initiates a request to create a hash index, the database system, based on value log technology, internally traverses the data table records, calculates the hash value according to the data type of the key, compares the key values, and rebuilds the hash index.
[0038] Step 2: Insert, update, or delete on the hash index; When users perform insert, update, and delete operations on the data table, the database system based on value log technology converts all operations into sequentially written value logs and generates corresponding LSNs. It calculates the hash value according to the data type of the key and compares the key values to insert, update, or delete on the hash index.
[0039] Step 3: Query the hash index; When a user initiates a query request, the database system based on value log technology receives the query request, calculates the hash value according to the data type of the query key, compares the key value, and performs the query.
[0040] like Figure 1 As shown, step 1, creating a hash index specifically includes the following steps:
[0041] Step 11: The user initiates a request to create a hash index. The database system based on value log technology internally constructs a fixed-size array in memory according to the actual size of the data table, and at the same time starts a background index rebuilding thread, and returns a message to the user that the index was successfully created.
[0042] Step 12: The background index rebuilding thread reads the value log offset address (LSN) of the first record from the data table, and reads the value of the first record from the value log using the LSN.
[0043] Step 13: Determine if this record belongs to the current table. If not, proceed to step 17; otherwise, proceed to step 14.
[0044] Step 14: Retrieve the Key from the read record to be inserted. Calculate the hash value using the corresponding hash function based on the Key's data type. Locate the array slot in memory using the hash value. Sequentially retrieve the LSN value log from the array slot collision list and read the corresponding record value. Compare the Key in the read record with the Key to be inserted.
[0045] Step 15: If a matching key is found and the current record operation is insert or update, then update the current LSN to the LSN of the new record. If the current record operation is delete, remove the LSN from the conflict chain.
[0046] Step 16: If no matching key is found and the current record operation is insert or update, insert the LSN of the new record into the conflict chain. If the current record operation is delete, do nothing.
[0047] Step 17: Read the next log entry. If it exists, proceed to step 13; otherwise, proceed to step 18.
[0048] Step 18: Hash index reconstruction complete.
[0049] like Figure 2 As shown, step 2, inserting, updating, or deleting hash indexes specifically includes the following steps:
[0050] Step 21: When a user performs insert, update, and delete operations on a data table, the database system based on value log technology will convert all operations into sequentially written logs and generate corresponding LSNs.
[0051] Step 22: Obtain the Key from the operation record, calculate the hash value using the corresponding hash function based on the Key's data type, locate the array slot in memory using the hash value, retrieve the corresponding record value from the LSN value log of the array slot collision list, and compare the Key in the record with the Key to be inserted.
[0052] Step 23: If a matching key is found and the current record operation is insert or update, then update the current LSN to the LSN of the new record. If the current record operation is delete, remove the LSN from the conflict chain.
[0053] Step 24: If no matching key is found and the current record operation is insert or update, insert the LSN of the new record into the conflict chain. If the current record operation is delete, do nothing.
[0054] like Figure 3 As shown, step 3, querying the hash index, specifically includes the following steps:
[0055] Step 31: The user initiates a query hash index request;
[0056] Step 32: When the database system based on value log technology receives a query request, it calculates the hash value using the corresponding hash function according to the data type of the query key; it locates the array slot in memory using the hash value, and retrieves the corresponding record value from the LSN value log from the array slot collision list in turn, and compares the key in the read record with the key currently being queried.
[0057] Step 33: If a matching key is found, return the current record to the user;
[0058] Step 34: If no matching key is found, no action is taken.
[0059] Let's take the asset table in the database as an example to explain the whole process. Assume there is a table T1 with three columns: UID, NAME, and PROPERTY, where UID is the primary key column:
[0060] Definition of Table 1T1
[0061] UID varchar(20) Asset ID, Primary Key NAME varchar(20) Asset Name PROPERTY int Asset value
[0062] The data in table T1 in the current log is as follows:
[0063] Table 2T1 Table Value Log Status
[0064]
[0065]
[0066] S101, the user initiates a request to create a new hash index on the UID column. After receiving the request, the system creates a fixed-length array A1 with a size of 2 in memory based on the actual size of table T1 (2 rows).
[0067] S102, after starting the reconstruction thread, returns a message indicating that the user was successfully created.
[0068] S103, the background reconstruction thread obtains the LSN=0 of the first record in the table, reads the record with Key=B002 through the LSN, and the operator for reading this record is the insertion operation. Then, the hash function of type varchar is used to calculate that Key=B002 belongs to slot 0 of A1. If the collision list of slot 0 is empty, then LSN=0 is inserted into the collision list.
[0069] S104, sequentially read the next record. The operator for reading this record is an update operation. The hash function of type varchar is used to calculate that Key = B002 belongs to slot 0 of A1. There is a value in the collision chain of slot 0. The record is read using LSN = 0, and the Key in the record is compared with the current Key. If they are found to be consistent, the LSN is updated to 10.
[0070] S105, sequentially read the next record. The operator for reading this record is an insertion operation. Then, use the varchar type hash function to calculate that Key = B003 belongs to slot 1 of A1. If the collision list of slot 1 is empty, then insert LSN = 20 into the collision list.
[0071] S106, sequentially read the next record. The operator for reading this record is a deletion operation. Then, the hash function of type varchar is used to calculate that Key = B003 belongs to slot 1 of A1. Since there is a value in the collision list of slot 1, the record is read using LSN = 20. The Key in the record is retrieved and compared with the current Key. It is found to be consistent, so the LSN value on the collision list is deleted.
[0072] S107, Hash index reconstruction complete.
[0073] S201, the user initiated a request to add a new data entry B001-pink-100.
[0074] S202, after receiving the request, the system first inserts the record into the value log, generating record LSN=40. A varchar hash function is then used to calculate the collision list for slot 1 of A1 where Key=B001 belongs.
[0075] S203, if the collision list is empty at this time, insert LSN=40 into the collision list.
[0076] S204, Insertion process ends.
[0077] S301, the user wants to query table T1, and the user initiates a query Key=B001.
[0078] S302, after receiving the request, the system uses a varchar type hash function to calculate that Key=B001 belongs to slot 1 of A1, retrieves LSN=40 from the collision chain of slot 1 of A1, and uses LSN=40 to read the record from the value log.
[0079] S303: Compare the key in the record with the key the user wants to query. If they match, return the record data to the user.
[0080] S304, search process ended.
[0081] This invention solves the write amplification problem of hash index persistent storage by combining the high-speed sequential write of value log technology with the advantages of key-value integrated disk writing. Furthermore, because value log technology can operate on inexpensive storage hardware, it addresses the cost issues associated with using expensive storage hardware.
Claims
1. A method for implementing hash index persistence based on value logs, characterized in that, Includes the following steps: Step 1: Create a hash index; When a user initiates a request to create a hash index, the database system, based on value log technology, internally traverses the data table records, calculates the hash value based on the key's data type, compares the key values, and rebuilds the hash index; specifically, this includes the following steps: Step 11: The user initiates a request to create a hash index. The database system based on value log technology internally constructs a fixed-size array in memory according to the actual size of the data table, and at the same time starts a background index rebuilding thread, and returns a message to the user that the index was successfully created. Step 12: The background index rebuilding thread reads the value log offset address (LSN) of the first record from the data table, and reads the value of the first record from the value log using the LSN. Step 13: Determine if this record belongs to the current table. If not, proceed to step 17; otherwise, proceed to step 14. Step 14: Retrieve the Key from the read record to be inserted. Calculate the hash value using the corresponding hash function based on the Key's data type. Locate the array slot in memory using the hash value. Sequentially retrieve the LSN value log from the array slot collision list and read the corresponding record value. Compare the Key in the read record with the Key to be inserted. Step 15: If a matching key is found and the current record operation is insert or update, then update the current LSN to the LSN of the new record. If the current record operation is delete, remove the LSN from the conflict chain. Step 16: If no matching key is found and the current record operation is insert or update, insert the LSN of the new record into the conflict chain. If the current record operation is delete, do nothing. Step 17: Read the next log entry. If it exists, proceed to step 13; otherwise, proceed to step 18. Step 18: Hash index reconstruction complete; Step 2: Perform insertion, update, or deletion operations on the hash index; When users perform insertion, update, and delete operations on the data table, the database system based on value log technology converts all operations into sequentially written value logs and generates corresponding LSNs. It calculates the hash value based on the key's data type and compares the key values to insert, update, or delete on the hash index; specifically, this includes the following steps: Step 21: When a user performs insert, update, and delete operations on a data table, the database system based on value log technology will convert all operations into sequentially written logs and generate corresponding LSNs. Step 22: Obtain the Key from the operation record, and calculate the hash value using the corresponding hash function based on the data type of the Key; The hash value is used to locate the array slot in memory. The corresponding record value is read from the LSN value log of the array slot collision chain in turn. The key in the read record is compared with the key to be inserted. Step 23: If a matching key is found and the current record operation is insert or update, then update the current LSN to the LSN of the new record. If the current record operation is delete, remove the LSN from the conflict chain. Step 24: If no matching key is found and the current record operation is insert or update, insert the LSN of the new record into the conflict chain. If the current record operation is delete, do nothing. Step 3: Query the hash index; When a user initiates a query request, the database system based on value log technology receives the request, calculates the hash value based on the data type of the query key, compares the key values, and performs the query; Querying the hash index specifically includes the following steps: Step 31: The user initiates a query hash index request; Step 32: When the database system based on value log technology receives a query request, it calculates the hash value using the corresponding hash function according to the data type of the query key. The hash value is used to locate the array slot in memory. The corresponding record value is read from the LSN value log of the array slot collision chain in turn. The key in the read record is compared with the key of the current query. Step 33: If a matching key is found, return the current record to the user; Step 34: If no matching key is found, no action is taken.
2. The method for implementing hash index persistence based on value logs as described in claim 1, characterized in that, The LSN pointing to the offset of the value log file is implemented as a physical address. During the search for the value log, the LSN is directly used to locate the physical offset address of the value log file, thereby reading the log.
3. The method for implementing hash index persistence based on value logs as described in claim 1, characterized in that, The LSN, which points to the offset of the value log file, is implemented as a logical address. By dividing the value log file into pages, the LSN value records the page number and the offset address within the page in the high and low bits, respectively. During the search of the value log, the page number is used to locate the physical block in the value log, and the offset address within the page is used to locate the offset address within the physical block, thereby reading the log.
4. A system for implementing hash index persistence based on value logs as described in claim 1, characterized in that, include: Hash index creation module, hash index operation module, and hash index query module; The three processes are respectively responsible for creating hash indexes, inserting, updating, or deleting hash indexes, and querying hash indexes.
5. The system for hash index persistence based on value logs as described in claim 4, characterized in that, In the hash index creation module, when a user initiates a hash index creation request, the database system, based on value log technology, internally traverses the data table records, calculates the hash value according to the data type of the key, compares the key values, and rebuilds the hash index.
6. The system for hash index persistence based on value logs as described in claim 4, characterized in that, In the hash index operation module, when users perform insert, update, and delete operations on the data table, the database system based on value log technology converts all operations into sequentially written logs and generates corresponding LSNs. It calculates the hash value based on the data type of the key and compares the key values to perform insert, update, or delete operations on the hash index.
7. The system for hash index persistence based on value logs as described in claim 4, characterized in that, In the hash index query module, when a user initiates a query request for records, the database system based on value log technology receives the query request, calculates the hash value according to the data type of the query key, compares the key value, and performs the query.