Value log memory mapping management method and system supporting concurrent index construction

Abstract

The invention discloses a value log memory mapping management method and system supporting concurrent index construction, and the method comprises the steps: carrying out the partitioning of a value log file, and obtaining a plurality of blocks with a fixed size; defining a block mapping array, wherein each element in the block mapping array records the mapping state of each block of the value log file; introducing a lock array corresponding to the block mapping array, wherein each lock in the lock array corresponds to the entry indexes of the plurality of blocks; when a task needs to read data of a certain offset of the value log file, calculating a value log block id according to the offset; searching whether a slot position corresponding to the block mapping array is empty or not according to the value log block id, and locking a corresponding position of the lock array only when the slot position is empty; after locking succeeds, file mapping operation is executed; when the block is not subjected to the offset, the data of the offset is not accessed within a period of time, that is, the state that the reference count of the block is 0 exceeds the preset duration, the timed cleaning mechanism can actively remove memory mapping at the corresponding position of the block mapping array, and related system resources are released.

Description

Technical Field

[0001] This invention belongs to the field of database storage and index management technology, specifically relating to a value log memory mapping management method and system that supports concurrent index construction. Background Technology

[0002] In graph databases or key-value stores, data can be stored as value logs in separate log files. To improve query efficiency, the system needs to build secondary indexes for tables or graphs. Index building typically involves full table scans and data reads, often improved through memory mapping. However, existing value log memory mapping mechanisms have the following drawbacks:

[0003] 1) Severe LRU conflicts and lock contention: Traditional solutions often use a global LRU cache to manage memory pages. Under high concurrency, when different threads access different areas of the log file, frequent cache replacement and lock operations are triggered, causing thread blocking and performance degradation.

[0004] 2) Duplicate mapping wastes memory: The lack of a unified mapping management mechanism means that different threads may repeatedly map the same region, leading to memory redundancy and increased management complexity.

[0005] 3) Delayed or frequent memory release: The mapped region is frequently loaded and unloaded in a short period of time, causing jitter, and there is a lack of an effective idle reclamation mechanism, which can easily lead to memory leaks.

[0006] Therefore, there is an urgent need for an improved solution that can efficiently manage value log mapping memory and reduce lock overhead and memory usage in high-concurrency scenarios. Summary of the Invention

[0007] Purpose of the Invention: To address the shortcomings of existing value log memory mapping mechanisms, this invention provides a value log memory mapping management method and system that supports concurrent index construction. By introducing block mapping arrays, fine-grained lock control, reference counting mechanisms, and timed garbage collection strategies, it achieves shared, dynamic management, and automatic release of mapped memory during high-concurrency database index construction, thereby improving overall system performance and memory utilization efficiency.

[0008] Technical solution: In a first aspect, the present invention provides a value log memory mapping management method that supports concurrent index construction, comprising:

[0009] The value log file is divided into blocks of fixed size. A block mapping array is defined, and each element in the block mapping array records the mapping status of each block of the value log file. The mapping status includes a mapping pointer, a reference count, and an idle timestamp. The mapping pointer is used to point to the starting address of the memory after the block is mapped. The reference count is used to indicate the number of active tasks currently using the block. The idle timestamp is used to record the timestamp when the block was last released from reference or the time when the reference count last dropped to 0.

[0010] A lock array corresponding to the block mapping array is introduced, wherein each lock in the lock array corresponds to an entry index of multiple blocks;

[0011] When a task needs to read data at a certain offset in the value log file, the value log block_id is calculated based on the offset. The value log block_id is then used to check if the corresponding slot in the block mapping array is empty. Only if the slot is empty is the corresponding position in the lock array locked based on the block_id modulo the lock array size. After successful locking, a file mapping operation is performed, mapping the block to the corresponding memory region. The mapped pointer is placed in the corresponding slot in the block mapping array, and the corresponding position in the lock array is unlocked. During mapping acquisition, the reference count is incremented by 1, indicating that the current task has acquired the right to use the block.

[0012] When the data at this offset is not accessed, the reference to the block is released, and the reference count is decremented by 1. When the reference count of the block is equal to 0, the idle timestamp is updated to record the time when the block was released from reference. If the block remains idle for more than a specified period of time, the timed cleanup mechanism will actively unmap the memory at the corresponding position in the block mapping array and release the relevant system resources.

[0013] Furthermore, the step of calculating the value log block_id based on the offset includes: taking the quotient of the offset and the block size to obtain the value log block_id.

[0014] Furthermore, when multiple tasks request the same block, only the first task performs the file mapping operation, and the other tasks wait for the mapping to complete before obtaining the right to use the block from the corresponding memory area and updating the reference count.

[0015] Furthermore, the process of releasing the reference to the block when the data at that offset is not accessed, decrementing the reference count by 1, and updating the block's free timestamp when the block's reference count equals 0, specifically includes:

[0016] Decrement the reference count by 1 and check if the block's reference count is greater than 0. If it is greater than 0, it means that there are still other references. If it is equal to 0, it means that the block is currently free. Record the start time of the block's free state as the free timestamp. The background periodic cleanup thread will automatically take over the free block and trigger the actual unmapping operation when the necessary conditions are met.

[0017] Furthermore, when the number of mapped blocks exceeds a set threshold, memory unloading operations are performed on blocks whose idle time exceeds a preset idle threshold.

[0018] Furthermore, the step of performing a memory unloading operation on blocks whose idle time exceeds a preset idle threshold when the number of mapped blocks exceeds a set threshold specifically includes:

[0019] Iterate through the entire block mapping array and perform the following checks and processing on each block:

[0020] Determine if the current block is in a reclaimable state. If so, while locking, release the memory mapping relationship of the block and set the corresponding mapping pointer to NULL.

[0021] The recyclable state is defined as a non-nullable mapped pointer, a reference count of 0, and an idle time exceeding a preset idle threshold.

[0022] Secondly, this invention provides a value log memory mapping management system that supports concurrent index construction, comprising:

[0023] The data structure module is used to divide the value log file into blocks, resulting in multiple fixed-size blocks. A block mapping array is defined, where each element records the mapping status of each block in the value log file. The mapping status includes a mapping pointer, a reference count, and an idle timestamp. The mapping pointer points to the starting memory address after the block is mapped. The reference count indicates the number of active tasks currently using the block. The idle timestamp records the timestamp when the block was last released from its reference or the time when the reference count most recently dropped to 0. A lock array corresponding to the block mapping array is introduced, where each lock in the lock array corresponds to an entry index of multiple blocks.

[0024] The lock / unlock module is used to perform memory mapping and demapping operations on block mapping array slots. When a task needs to read data at a certain offset in the value log file, the value log block_id is calculated based on the offset. The block mapping array slot is then checked for emptiness based on the block_id. Only if the slot is empty is the corresponding position in the lock array locked based on the block_id modulo the lock array size. After successful locking, a file mapping operation is performed, mapping the block to the corresponding memory region. The mapped pointer is placed in the corresponding slot of the block mapping array, and the corresponding position in the lock array is unlocked. During mapping acquisition, the reference count is incremented by 1, indicating that the current task has acquired the right to use the block. If the data at that offset is not accessed for a certain period (i.e., the reference count of the block is 0 for a preset time), a timed cleanup mechanism will actively manage the mapped block, locking the corresponding position in the lock array based on the block_id modulo the lock array size. After successful locking, a demapping operation is performed, the corresponding slot in the block mapping array becomes empty, and the corresponding position in the lock array is unlocked.

[0025] Furthermore, when multiple tasks request the same block, only the first task performs the file mapping operation, and the other tasks wait for the mapping to complete before obtaining the right to use the block from the corresponding memory area and updating the reference count.

[0026] Furthermore, the process of releasing the reference to the block when the data at that offset is not accessed, decrementing the reference count by 1, and updating the block's free timestamp when the block's reference count equals 0, specifically includes:

[0027] Decrement the reference count by 1 and check if the block's reference count is greater than 0. If it is greater than 0, it means that there are still other references. If it is equal to 0, it means that the block is currently free. Record the start time of the block's free state as the free timestamp. The background periodic cleanup thread will automatically take over the free block and trigger the actual unmapping operation when the necessary conditions are met.

[0028] Furthermore, when the number of mapped blocks exceeds a set threshold, memory unloading operations are performed on blocks whose idle time exceeds a preset idle threshold.

[0029] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0030] (1) Improve concurrent access performance: High-concurrency access is achieved based on block-level locks, and different threads accessing different blocks do not affect each other, reducing lock conflicts and context switching.

[0031] (2) Reduce lock management overhead: Locks are only applied during mapping and demapping, avoiding a large number of lock operations caused by maintaining a complex LRU structure and improving CPU utilization.

[0032] (3) Automated memory management: A timed recycling mechanism is introduced, which allows the system to automatically identify and unload idle memory blocks, thus avoiding memory leaks and long-term occupation.

[0033] (4) Simple implementation and strong adaptability: The data structure is simple and clear, and the interface calls are simple, making it easy to integrate and apply in existing systems and to extend and optimize them.

[0034] (5) The present invention can manage value log file memory mapping efficiently and with low memory usage. Attached Figure Description

[0035] Figure 1 This diagram illustrates the block array and lock array management structure of this invention. It shows the log file being divided into multiple blocks and their mapping in memory, with each lock corresponding to multiple blocks for concurrency control.

[0036] Figure 2 This is a schematic diagram of the mapping block acquisition process of the present invention. The process illustrates the logical steps of the system checking the mapping status, handling locks, executing mmap, and returning a pointer when a thread requests access to a specified file block.

[0037] Figure 3 This is a schematic diagram of the block release process of the present invention. The process illustrates how the system releases the reference to a specified file block after a thread accesses it.

[0038] Figure 4 This is a schematic diagram of the timer cleanup mechanism of the present invention. The process shows how the cleanup thread periodically scans the block mapping array, checks the reference count and idle time, and decides whether to execute munmap to release the memory mapping. Detailed Implementation

[0039] This invention provides a value log memory mapping management method that supports concurrent index construction. The database system employing value log technology converts all operations into sequentially written value logs and generates corresponding log sequence numbers (LSNs). When a user initiates an index creation request, the system internally traverses the data table records and rebuilds the index based on the key value, mainly including the following steps:

[0040] Step 1: Initialize two parallel arrays, block_array and lock_array, based on the value log file size and the predetermined block size. Specific operations include:

[0041] like Figure 1As shown, assuming the value log file size is N bytes and the predetermined block size is set to B bytes (e.g., 64MB), the value log file is divided into multiple blocks using the predetermined block size. A block mapping array `block_array` is defined with a length of approximately `ceil(N / B)`. Each `block_array[i]` corresponds to the i-th block of the value log file (byte offset range: [i*B, (i+1)*B)). The block mapping array (`block_array`) records the mapping status of each block, including the mapping pointer, reference count, and idle timestamp. Specifically, the elements of the block mapping array include:

[0042] mapped_ptr(mapped pointer) points to the starting address of the memory block after mapping; it is NULL if the block is not currently mapped.

[0043] readers (reference count): An integer counter representing the number of active tasks currently using this block.

[0044] last_unmap_time (last unreference time or free start time): Used to record the timestamp of the last time the block was released from reference or the time when the reference count most recently dropped to 0.

[0045] mapped_num (total number of mapped blocks): Used to record the total number of mapped blocks.

[0046] Introduce a lock array corresponding to the block mapping array, with each lock corresponding to multiple block entry indices.

[0047] The `block_id%L` rule maps blocks to the lock array `lock_array`, where `L` is the size of the lock array (e.g., 32). Each mapping or demapping only requires acquiring the lock at the corresponding block index; that is, only the slot of the corresponding block is locked. During system initialization, all block entries have an empty `mapped_ptr`, `readers` is 0, and all locks are set to an unoccupied state. Creating an index based on the value log does not change the log content; at this time, different threads can concurrently access the same block.

[0048] Step 2: When a task needs to read data at a specific offset in the value log file, the block containing that offset must first be mapped into memory. Specific operations include:

[0049] like Figure 2 As shown, the specific mapping process is as follows:

[0050] Calculate the target block index: Calculate the value log block_id based on the offset, which is the quotient of offset and block size.

[0051] Check mapping status: Check if the block_array slot is empty based on block_id to determine if the block has been mapped.

[0052] Lock and perform mapping: If the block is not yet mapped (i.e., the slot is empty), it needs to be locked (by taking the remainder of the block_id with respect to the size of the lock_array array and locking the corresponding position in lock_array). Under the premise of mutual exclusion (i.e., successful locking), the file mapping operation is performed (by calling the system mmap), and the mapped pointer is recorded, that is, the block pointer is placed in the corresponding slot of block_array; the number of mapped blocks is incremented by 1.

[0053] Each block maintains a `readers` field, recording the number of threads currently referencing it. The reference count is incremented by 1 when the block is acquired during mapping and decremented by 1 after use, ensuring thread-safe sharing and delayed unloading. Specifically, the initialization state is as follows: After mapping is complete, the system initializes the block's `readers` to 0 and unlocks the corresponding position in `lock_array`. The reference count is incremented: Regardless of whether this is the first mapping of the block, `readers` is incremented by 1, indicating that the current task has acquired the right to use the block.

[0054] In concurrent access scenarios, only the first thread requesting the same memory block will actually perform the mmap operation; other threads accessing the block around the same time will wait on the lock. Once the mapping is complete, these subsequent threads can directly use the mapped memory region, requiring only an update to the reference count, without needing to remap it.

[0055] Step 3: When a block of data is no longer needed, promptly release the reference to that block to facilitate potential subsequent memory reclamation. For example... Figure 3 As shown, the specific block reference release process is as follows:

[0056] First, check if the number of readers in the target block is greater than 0.

[0057] If a reference exists, decrement readers by 1.

[0058] When readers equals 0, it means that there are currently no users in the block, and the system will record the start time of the block becoming idle.

[0059] When a block's reference count reaches zero, its memory mapping is not immediately released. The system typically monitors the status of free blocks through a background timer, and only triggers the actual memory unloading operation when the proportion of mapped blocks and the idle time of free blocks exceed preset thresholds.

[0060] Step 4: This cleanup thread runs periodically. When the number of mapped blocks in the system exceeds a set threshold (e.g., 3 / 4 of the total number of blocks), a cleanup scan will be initiated.

[0061] like Figure 4 As shown, the cleanup process traverses the entire block_array, performing the following checks and processing on each block:

[0062] Determine if it is a free mapped block: If the block is still mapped (i.e. the mapped pointer is not null) and there are currently no active references (readers == 0), proceed to the next step.

[0063] Determine if the idle time has expired: Get the idle time of the block since the last time the reference was released. If the idle time exceeds the preset idle threshold (e.g., 3 seconds), the block is determined to be reclaimable.

[0064] Perform unmapping: While the lock is engaged, call the system munmap interface to unmap the memory of the block and set the corresponding mapping pointer to NULL to release the physical memory. After unmapping, release the corresponding lock.

[0065] This thread runs continuously, repeating the above process and periodically reclaiming idle mapping blocks.

[0066] To adapt to different system loads, it is recommended to dynamically adjust the following parameters to optimize performance:

[0067] Scan frequency t: The running cycle of the cleanup thread, which can be automatically adjusted according to memory pressure.

[0068] Trigger threshold k: Controls when to start cleaning up the mapped block.

[0069] The idle threshold UNMAP_THRESHOLD controls when the mapping is actually unloaded after the block is released, avoiding performance loss caused by frequent mapping and unloading.

[0070] Taking concurrent access as an example, the value log memory mapping management method supporting concurrent index construction provided by the above embodiments of the present invention will be further explained.

[0071] Concurrent index creation is performed, assuming that task A and task B need to read data located in the same block (e.g., block X) of the value log file almost simultaneously. According to the method of this embodiment, the specific process includes:

[0072] Time T0: Task A calls the block mapping acquisition function to obtain the mapping of block X. Since block X is initially unmapped, after acquiring the lock, A executes mmap to load it into memory, and the readers are incremented from 0 to 1. Task A then reads the required data through the returned pointer.

[0073] Time T1: Immediately following, task B also requests data from block X. When calling the block acquisition function, it finds the lock is held by task A and must wait. After task A releases the block lock shortly after T1, task B acquires the lock. At this point, block X is already mapped in memory (mapped_ptr is not empty). B does not need to execute mmap again; it simply increments readers from 1 to 2 and then obtains the mapping pointer to begin reading data. It can be seen that the memory for block X is shared by tasks A and B, avoiding redundant mapping.

[0074] Time T2: Task A completes the processing of data in block X, calls the map block release function to release the reference, and the system reduces readers from 2 to 1 (B is still in use, so it is still 1 and no idle time is recorded).

[0075] Time T3: Shortly after, task B also completes its reading and calls the map block release function, reducing readers from 1 to 0. At this point, the system records the start time of idle block X, last_unmap_time, as T3.

[0076] At time T4: During a subsequent periodic check, the background cleanup thread discovers that block X's readers are equal to 0 and has been idle for more than the threshold (assuming the threshold is less than T4-T3). Therefore, at time T4, it acquires the lock and executes munmap, removing the mapping of block X from memory and setting mapped_ptr to null. Block X is thus completely released and no longer occupies memory.

[0077] If a new task C accesses the block data at a future time T5 after block X is unloaded, mmap will be triggered again to load it when it calls the map block retrieval function, and the process is similar to that described above.

[0078] In the implementation of this invention, some engineering optimizations and fault tolerance measures can be incorporated. For example, consider handling exceptional situations: if a task crashes or times out while holding a block reference, the system should ensure that its reference is eventually released. This can be achieved by the task scheduling module iteratively cleaning up its list of unreleased block references when the task terminates. Furthermore, for very large log files with large block array lengths, segmented loading or multi-level structures can be considered to reduce initialization memory overhead. Moreover, this invention is also applicable to other types of large file memory mapping management, such as general large data file caching management. Similar memory and concurrency performance gains can be achieved simply by adjusting the block size strategy and cleanup timing.

[0079] This invention effectively solves the problems of high memory consumption and thread contention in value log mapping during traditional index creation by combining shared mapping, reference counting, and timed cleanup. It achieves automatic and efficient management of mapped memory while ensuring concurrent access performance. The solution presented in this invention has a simple and efficient structure, significant memory savings and performance improvements, and can be used in system environments that support concurrent index creation and have demanding memory requirements.

Claims

1. A value log memory mapping management method supporting concurrent index construction, characterized by: The method comprises the following steps: chunking a value log file to obtain a plurality of fixed-size blocks; defining a block mapping array, each element in the block mapping array recording a mapping state of each block of the value log file, the mapping state comprising a mapping pointer, a reference count and an idle timestamp; the mapping pointer is used to point to the starting address of the memory after mapping of the block; the reference count is used to represent the number of active tasks currently using the block; and the idle timestamp is used to record the timestamp of the last time the block is released or the timestamp of the last time the reference count is reduced to 0; a lock array corresponding to the block mapping array is introduced, each lock in the lock array corresponding to a plurality of block entry indexes; when a task needs to read data of a certain offset of the value log file, a value log block_id is calculated according to the offset; whether the slot corresponding to the value log block_id in the block mapping array is empty is found; only when the slot is empty, the corresponding position of the lock array is locked according to the block_id modulo the size of the lock array; after successful locking, a file mapping operation is performed to map the block to the corresponding memory area, the mapping pointer after mapping is placed in the corresponding slot of the block mapping array, and the corresponding position of the lock array is unlocked; when mapping is obtained, the reference count is increased by 1, indicating that the current task obtains the use right of the block; when the data of the offset is not accessed, the reference of the block is released, the reference count is reduced by 1, and when the reference count of the block is equal to 0, the idle timestamp of the block is updated to record the time when the block is released; after continuous idling for more than a specified time length, the memory mapping of the corresponding position of the block mapping array is actively released, and the related system resources are released.

2. The value log memory mapping management method supporting concurrent index construction according to claim 1, characterized in that: The value log block_id is calculated according to the offset, which comprises: obtaining the value log block_id by taking the quotient of the offset and the block size.

3. The method of claim 1, wherein: When there are multiple tasks requesting the same block, only the first task performs the file mapping operation, and other tasks wait for the completion of the mapping to obtain the use right of the block from the corresponding memory area and update the reference count.

4. The method of claim 1, wherein: When the data of the offset is not accessed, the reference of the block is released, the reference count is reduced by 1, and when the reference count of the block is equal to 0, the idle timestamp of the block is updated, which specifically comprises: reducing the reference count by 1, checking whether the reference count of the block at this time is greater than 0, if yes, indicating that there are still other references, if no, indicating that the block currently has no user, and recording the starting time of the block idling as the idle timestamp.

5. The method of claim 1, wherein: When the number of mapped blocks exceeds a set threshold, memory unloading operation is performed on the blocks with idle time exceeding a preset idle threshold.

6. The value log memory mapping management method of claim 5, wherein: When the number of mapped blocks exceeds a set threshold, memory unloading operation is performed on the blocks with idle time exceeding a preset idle threshold, which specifically comprises: traversing the entire block mapping array, and performing the following judgment and processing on each block: judging whether the current block is in a recyclable state, if yes, releasing the memory mapping relationship of the block under the premise of locking, and setting the corresponding mapping pointer to NULL; the recyclable state is that the mapping pointer is not empty, the reference count is 0, and the idle time exceeds the preset idle threshold.

7. A value log memory map management system that supports concurrent index construction, characterized by: ​ The data structure module is used for blocking the value log file to obtain a plurality of fixed-size blocks; a block mapping array is defined, each element in the block mapping array records the mapping state of each block of the value log file, and the mapping state includes a mapping pointer, a reference count and a free time stamp; the mapping pointer is used to point to the starting address of the mapped memory of the block; the reference count is used to represent the number of active tasks using the block; and the free time stamp is used to record the time stamp of the last time the block is released or the time of the last reference count reduction to 0; a lock array corresponding to the block mapping array is introduced, and each lock in the lock array corresponds to the entry index of a plurality of blocks; The locking and unlocking module is used for operating the block mapping array slot to perform memory mapping and demapping; when a task needs to read data of a certain offset of the value log file, the value log block_id is calculated according to the offset; whether the slot corresponding to the block mapping array is empty is found according to the value log block_id; only when the slot is empty, the corresponding position of the lock array is locked according to the block_id modulo the size of the lock array; after the locking is successful, the file mapping operation is performed, the block is mapped to the corresponding memory area, the mapping pointer after the mapping is placed in the corresponding slot of the block mapping array, and the corresponding position of the lock array is unlocked; when the mapping is obtained, the reference count is increased by 1, indicating that the current task obtains the use right of the block; when the data of the offset is not accessed, the reference of the block is released, the reference count is reduced by 1, and when the reference count of the block is equal to 0, the free time stamp of the block is updated to record the time when the block is released.

8. The value log memory mapping management system supporting concurrent index construction of claim 7, wherein: When there are a plurality of tasks requesting the same block, only the first task performs the file mapping operation, and other tasks wait for the completion of the mapping to obtain the use right of the block from the corresponding memory area and update the reference count.

9. The value log memory mapping management system that supports concurrent index construction of claim 7, wherein: When the data of the offset is not accessed, the reference of the block is released, the reference count is reduced by 1, and when the reference count of the block is equal to 0, the free time stamp of the block is updated, which specifically includes: The reference count is reduced by 1, and whether the reference count of the block at this time is greater than 0 is checked; if greater than 0, it indicates that there is still other reference; if equal to 0, it indicates that the block currently has no user, and the starting time of the block free at this time is recorded as the free time stamp.

10. The value log memory mapping management system that supports concurrent index construction of claim 7, wherein: When the number of mapped blocks exceeds a set threshold, the memory unloading operation is performed on the blocks with free time exceeding a preset free threshold.

Images