A memory table implementation of an LSM tree

Abstract

This invention relates to the field of LSM tree technology, specifically to a method for implementing an LSM tree in memory, comprising the following steps: initializing CSPPTrie, allocating the space currently needed for each node through MemPool, and optimizing node concurrent access based on a multi-core CPU architecture. In this invention, by introducing memory management and data synchronization technologies, the system's write speed and data consistency are significantly improved. The use of copy-on-write technology effectively reduces lock contention in multi-core environments. By creating data copies in memory and updating operations, more efficient concurrent writes are achieved, reducing lock requirements, increasing processing speed, and significantly reducing operation conflicts. The implementation of token and delayed release mechanisms optimizes memory resource management, prevents memory leaks, and enhances the system's resource reclamation efficiency. This strategy ensures full utilization of system resources, making memory reclamation more efficient. The application of memory-mapped files makes the data persistence process more direct and efficient.

Description

Technical Field

[0001] This invention relates to the field of LSM tree technology, and in particular to a method for implementing an LSM tree in memory. Background Technology

[0002] Log-Structured Merge-Tree (LSM) is a data structure used for persistent storage. It records data insertion, modification, and deletion operations in memory, and then writes these operations sequentially to disk in batches once a certain number of operations have occurred. This sequential write capability improves write performance, making it particularly suitable for applications where writes outnumber reads, such as big data storage systems. The core idea of ​​LSM is to sacrifice some read capacity to maximize write capacity, trading high-performance writes for a small amount of read performance. Each record in an LSM contains (key, version sequence number, type, value) and (Key, Seq, Type, Value). Key and Value are byte arrays; Seq is an integer; and Type indicates the data type, typically including ① Put and ② Delete. More complex LSMs include additional extended types.

[0003] A memtable is a mutable data structure, typically implemented using ordered data structures (such as balanced trees or skip lists) for fast insertion, deletion, and query operations. All write operations (insert, update, and delete) are first performed in the memtable. An immutable memtable, on the other hand, becomes immutable when it reaches a certain size or other conditions. Immutable memtables are read-only and await being flushed to disk. The LSM tree implementation of memtables is primarily used to improve the performance of database systems, file systems, or other applications requiring high-speed data writing, while ensuring persistent data storage and queryability.

[0004] Existing technologies suffer from inefficiency in high-concurrency environments. The conversion from conventional memory tables to disk tables involves complex data copying and physical writing, which not only consumes a lot of CPU resources but also increases operation latency. In addition, traditional locking mechanisms are prone to contention in multi-threaded environments, leading to inefficient write operations. This is particularly evident in applications requiring fast processing and high throughput, such as real-time data analysis and trading systems. Furthermore, data processing latency or slow system response directly affects the real-time nature of analysis results and the timeliness of transactions, resulting in missed opportunities or incorrect decisions. Moreover, inadequate memory management can also cause system stability issues, such as frequent memory overflows or excessive garbage collection. Summary of the Invention

[0005] To address the inefficiency issues inherent in existing technologies under high concurrency environments, the conversion from conventional in-memory tables to disk tables involves complex data copying and physical writing, consuming significant CPU resources and increasing operation latency. Furthermore, traditional locking mechanisms are prone to contention in multi-threaded environments, leading to inefficient write operations, particularly pronounced in applications requiring rapid processing and high throughput, such as real-time data analysis and trading systems. Moreover, data processing delays or slow system responses directly impact the real-time nature of analysis results and the timeliness of transactions, resulting in missed opportunities or incorrect decisions. In addition, inadequate memory management can cause system stability issues, such as frequent memory overflows or excessive garbage collection. This invention provides a method for implementing an LSM tree-based in-memory table. The technical solution is as follows:

[0006] On the one hand, a method for implementing an LSM tree in memory is provided, including the following steps:

[0007] S1: Initialize CSPPTrie, allocate the space needed by each node through MemPool, optimize node concurrent access based on multi-core CPU architecture, optimize lock contention using copy-on-write technology, and update the original node pointers through CAS operation to obtain a copy-on-write pointer set;

[0008] S2: Based on the copy-on-write pointer set, allocate access tokens, collect node addresses that need to be delayed in release, optimize the memory resource reclamation process, and avoid memory leaks and data races through tokens and delayed release mechanisms to obtain a memory reclamation queue;

[0009] S3: Based on the memory reclamation queue, periodically release unnecessary nodes in memory, record memory reclamation activities, and reallocate the reclaimed memory areas to refine memory reuse efficiency and obtain a memory reuse table;

[0010] S4: Based on the memory reuse table, adjust the thread-local storage configuration of the memory pool, adopt a memory pool with thread caching, optimize the speed of global memory requests and thread access to the memory pool, and implement integer offset addressing to replace pointer addressing method to obtain offset addressing mapping data.

[0011] S5: Based on the offset addressing mapping data, construct a memory-mapped file, match memory and file synchronous read and write, verify the consistency between memory and disk files through memory mapping, optimize file read and write paths, and generate data write-ahead logs;

[0012] S6: Based on the data write-ahead log, verify the integrity and security of the data write, adjust the data organization of the memory-mapped file, optimize data access latency and data access efficiency, and generate a data storage framework.

[0013] On the other hand, the copy-on-write pointer set includes a copy index, concurrent access metrics, and lock contention metrics; the memory reclamation queue specifically includes delayed release flags, token index records, and resource optimization metrics; the memory reuse table includes space reallocation records, memory release efficiency, and reuse rate metrics; the offset addressing mapping data includes thread configuration mapping, address translation efficiency, and addressing speed improvement results; the data write-ahead log includes operation integrity verification results and persistent disk writes; and the data storage framework specifically includes security verification parameters, latency optimization metrics, and access speed analysis results.

[0014] On the other hand, the specific steps for initializing CSPPTrie, allocating the space needed by each node through MemPool, optimizing concurrent access to nodes based on multi-core CPU architecture, optimizing lock contention using copy-on-write technology, and updating the original node pointers through CAS operations to obtain the copy-on-write pointer set are as follows:

[0015] S101: Initialize CSPPTrie, dynamically allocate memory space for each data node based on the current data size, match the multi-core CPU architecture, reduce memory usage, and obtain the node configuration table;

[0016] S102: Based on the node configuration table, create a data copy for each modification operation, optimize lock contention in concurrent operations, and use CAS to perform lock-free update of the original data pointer to obtain the copy synchronization record;

[0017] S103: Based on the aforementioned copy synchronization record, integrate the pointer updates of data modification operations, manage the pointers to data in memory, optimize the data access path, and obtain a copy-on-write pointer set.

[0018] On the other hand, based on the copy-on-write pointer set, access tokens are allocated, node addresses requiring delayed release are collected, and the memory resource reclamation process is optimized. The specific steps for obtaining the memory reclamation queue, using tokens and a delayed release mechanism to avoid memory leaks and data races, are as follows:

[0019] S201: Based on the copy-on-write pointer set, control data modification permissions, verify the order and security of modification operations, and allocate and manage access tokens for each data node to obtain a token management pool;

[0020] S202: Based on the token management pool, monitor the status of all nodes, mark nodes that are no longer active or need to be updated, put the node addresses into the memory reclamation processing queue, and trigger resource reclamation through a timed task to obtain resource reclamation scheduling information.

[0021] S203: Based on the resource reclamation scheduling information, periodically clean up nodes that are no longer needed, release associated memory resources, reallocate memory, and provide it to new or updated nodes to obtain a memory reclamation queue.

[0022] On the other hand, based on the memory reclamation queue, the steps of periodically releasing unnecessary nodes in memory, recording memory reclamation activities, and reallocating reclaimed memory areas to refine memory reuse efficiency and obtain a memory reuse table are as follows:

[0023] S301: Based on the memory reclamation queue, monitor the memory usage status, identify and mark inactive nodes, trigger the release process, and periodically reclaim the marked nodes to obtain a memory cleanup log;

[0024] S302: Based on the memory cleanup log, perform a memory release operation, record the release time and the amount of resources reclaimed for each node, update the memory management database, and obtain resource reclamation records;

[0025] S303: Based on the resource recycling records, analyze the recycling efficiency and space reuse, reallocate the released resources, allocate memory space for new data requests, and obtain a memory reuse table.

[0026] On the other hand, based on the memory reuse table, the thread-local storage configuration of the memory pool is adjusted, a memory pool with thread caching is adopted, the speed of global memory requests and thread access to the memory pool is optimized, and the integer offset addressing method is implemented to replace pointer addressing to obtain offset addressing mapping data. The specific steps are as follows:

[0027] S401: Based on the memory reuse table, adjust the memory pool configuration, match thread-local storage, optimize the resource allocation between threads, and reduce global memory access conflicts by adjusting memory access parameters to obtain a thread resource allocation table;

[0028] S402: Based on the thread resource allocation table, update the memory address mapping method, use integer offset instead of traditional pointer addressing, optimize address calculation speed and reduce processing latency, and obtain offset addressing optimization record;

[0029] S403: Based on the offset addressing optimization record, integrate the new mapping data into memory management, and verify whether each memory request can be located in the correct data area to obtain offset addressing mapping data.

[0030] On the other hand, by updating the mapping method of the memory addresses, integer offsets are used instead of traditional pointer addressing. Simultaneously, based on factors such as page alignment and cache line optimization, the following formula is adopted:

[0031] M = B + O × A

[0032] Calculate the new memory address M, where B is the base address of the memory block, O represents the integer offset from the base address, and A is the alignment adjustment parameter.

[0033] On the other hand, based on the offset addressing mapping data, the steps of constructing a memory-mapped file, matching memory and file synchronous read / write, verifying the consistency between memory and disk files through memory mapping, optimizing file read / write paths, and generating data write-ahead logs are as follows:

[0034] S501: Based on the offset addressing mapping data, configure the memory-mapped file, link memory operations and files, synchronize data changes in memory to disk, optimize read and write differences, and obtain a real-time data synchronization mechanism;

[0035] S502: Based on the real-time data synchronization mechanism, adjust the storage path of the CSPPTrie structure, perform data path mapping, and optimize the data processing latency by controlling the data flow in memory to obtain the optimized data path configuration;

[0036] S503: Based on the optimized configuration of the data path, perform data consistency monitoring, record the status of each data update, verify the security of the data, and obtain the data write-ahead log.

[0037] On the other hand, the integer offset of each piece of data written to the data write-ahead log is stored in CSPPTrie as the value associated with the key of that piece of data.

[0038] On the other hand, based on the data write-ahead log, the steps of verifying the integrity and security of data writing, adjusting the data organization of the memory-mapped file, optimizing data access latency and data retrieval efficiency, and generating the data storage framework are as follows:

[0039] S601: Based on the data write-ahead log, perform integrity checks on the data write operations, verify whether each data update meets the standard, identify and correct potential data errors, and obtain integrity check results;

[0040] S602: Based on the integrity check results, evaluate the organization structure of the memory-mapped file, adjust and optimize the data access method, and obtain the data access optimization results;

[0041] S603: Based on the data access optimization results, optimize the storage logic and data flow to match the needs of large-scale data processing and obtain a data storage framework.

[0042] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0043] By introducing memory management and data synchronization technologies, the system's write speed and data consistency are significantly improved. The copy-on-write technique effectively reduces lock contention in multi-core environments. By creating data copies in memory and updating operations, more efficient concurrent writes are achieved, reducing lock requirements, increasing processing speed, and significantly reducing operation conflicts. The implementation of token and delayed release mechanisms optimizes memory resource management, prevents memory leaks, and enhances the system's resource reclamation efficiency. This strategy ensures full utilization of system resources, making memory reclamation more efficient. The application of memory-mapped files makes the data persistence process more direct and efficient, supports real-time synchronization between memory and disk, reduces the risk of data inconsistency, and improves data access speed. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart of the main steps of the present invention;

[0046] Figure 2 This is a flowchart of steps S1 of the present invention;

[0047] Figure 3 This is a flowchart of steps S2 of the present invention;

[0048] Figure 4 This is a flowchart of steps S3 of the present invention;

[0049] Figure 5 This is a flowchart of step S4 of the present invention;

[0050] Figure 6 This is a flowchart of steps S5 of the present invention;

[0051] Figure 7 This is a flowchart of step S6 of the present invention. Detailed Implementation

[0052] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0053] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0054] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, their intended meanings are consistent. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, their intended meanings are consistent.

[0055] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0056] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0057] This invention provides a method for implementing an LSM tree in memory, such as... Figure 1 As shown, it includes the following steps:

[0058] S1: Initialize CSPPTrie, allocate the space needed by each node through MemPool, optimize node concurrent access based on multi-core CPU architecture to reduce memory consumption, optimize lock contention using copy-on-write technology, create a memory copy for each node modification, and update the original node pointer through CAS operation to obtain the copy-on-write pointer set;

[0059] S2: Based on the copy-on-write pointer set, allocate access tokens, collect the node addresses that need to be delayed in release, put them into the memory reclamation queue, optimize the memory resource reclamation process, and avoid memory leaks and data races through tokens and delayed release mechanisms to obtain the memory reclamation queue;

[0060] S3: Based on the memory reclamation queue, it releases unnecessary nodes in memory periodically, optimizes memory usage efficiency, records memory reclamation activities, and reallocates reclaimed memory areas to refine memory reuse efficiency and obtain a memory reuse table.

[0061] S4: Based on the memory reuse table, adjust the thread-local storage configuration of the memory pool, adopt a memory pool with thread caching, optimize the speed of global memory requests and thread access to the memory pool, and implement integer offset addressing to replace pointer addressing method to refine memory addressing efficiency and obtain offset addressing mapping data.

[0062] S5: Based on offset addressing mapping data, construct memory-mapped files, match memory and file synchronous read and write, verify the consistency between memory and disk files through memory mapping, store CSPPTrie structure in memory-mapped files, optimize file read and write paths, monitor the data persistence process, and generate data write-ahead logs;

[0063] S6: Based on the data write-ahead log, verify the integrity and security of the data write, record the data verification activities, adjust the data organization of the memory-mapped file, optimize data access latency and data access efficiency, and generate a data storage framework.

[0064] Using a 32-bit unsigned integer as a pointer for addressing within the memory pool, with 4 bytes as the addressing unit, the maximum addressable range is 16G: the largest 32-bit unsigned integer is (4G-1), so the largest 32-bit integer can be addressed in the range of 0 to (4G-1)*4 with 4-byte alignment.

[0065] A Trie is an automaton. The term "state" in an automaton corresponds to the term "node" in a data structure. Each Trie node can have a maximum of 256 children, but in reality, the vast majority of Trie nodes have no more than 3 children. Therefore, metadata is defined for each node (occupying exactly one 4-byte unit):

[0066]

[0067] When b_is_final is 1, it indicates that the node is in a terminating state (the terminating state of the automaton). A node in a terminating state represents a complete key, and thus corresponds to a key-value pair (Key-Value-Pay).

[0068] In fact, only the first byte contains metadata; the other three bytes serve other purposes. The most important metadata is n_cnt_type. Based on the value of n_cnt_type, nodes are classified into the following categories:

[0069] 1. Between 0 and 6: The value of n_cnt_type is the number of children of this node. Specifically:

[0070] 1, 0: At this time, the number of children of this node is 0, and b_is_final is definitely true;

[0071] 2, 1, 2: At this point, c_l abe l(label) stores the label of the corresponding child of this node;

[0072] 3.3-6: At this point, c_l abe l and the following 4 bytes store the label of the corresponding child of this node.

[0073] 2. 7: At this point, the number of children is between 7 and 16. The actual number of children is placed at the c_abel position and accessed using BigCount.n_children(u int16). The next 2 to 4 (4-byte) addressing units are placed in the abeel.

[0074] 3. 8: At this point, the number of children is between 17 and 64. A rank-select bitmap is used to store the index. When bits[ch] is 1, it indicates that a child corresponding to ch exists. The position of the 1st bit in the rank-select bitmap can be calculated from the bitmap, which is then used to retrieve the child pointer from the subsequent child array. The rank index occupies 4 bytes, and the rank bitmap occupies 32 bytes, divided into four 64-bit integers. The 4-byte rank index stores the prefix sum of the 64-bit integers popcnt from 0 to 3. Therefore, the rank of each ch is Rank(ch) = index[ch / 64] + popcnt(mask1(bitu64[ch / 64],ch%64)).

[0075] Therefore, the corresponding child = children[Rank(ch)].

[0076] IV. 15: At this point, an array of 256 child pointers is allocated for this node. The array item ch il dren[ch] is ni l, which means that there is no child corresponding to ch. Such a node will not be reallocated again due to the insertion of a new child.

[0077] The sequence number represents the sequence number of a "4-byte unit". Light gray (#1) represents zpath_len, blue (#2) is the number of children, gold (#3) and pink-orange (#4) are the label of the ABAL, dark green (#5) is the rank index, yellow (#6) is the rank bitmap, light green (#7) is the compressed path, red (#8) is the child pointer, light purple (#9) is the value, light blue (#10) is an array of pointers to 256 children (using 32-bit integers to represent pointers), and white (#0).

[0078] Among them, the pink-orange label, compressed path, and child pointer array are variable-length for different nodes, while the others are fixed-length:

[0079]

[0080] The first step is lookup. Given a key, starting from the root (root, initial_state), search for the child node corresponding to the current input character. If no child node is found, it means there is no corresponding node up to the current input character, and insertion is performed from there. There are several cases, but the implementation is not fundamentally different from ordinary pointer-based data structures (directly supported by the programming language). The important part is:

[0081] 1. Because insertion requires memory allocation, the nodes that were previously delayed in being released and are now ready for reclamation are reclaimed first. 2. Locks (MetaInfo.b_lock, equivalent to row locks) are applied to the nodes on which insertion will be performed.

[0082] 3. For nodes that need to be inserted, allocate memory according to the size after insertion and copy them. Perform the insertion on the copy.

[0083] enter

[0084] 4. Because the insertion increases the number of children, which in turn causes a change in cnt_type, handle the resulting change in storage method. 5. Initialize and copy the value corresponding to the key.

[0085] 6. Lock the parent node

[0086] 7. Use CAS to replace the pointer to the current node in the parent node.

[0087] Because of concurrent multi-threaded writes, when locking or CAS fails, the system immediately retryes several times (e.g., 5 times). If it still fails, it relinquishes the current thread's time slice and retryes several more times (e.g., 5 times). If it still fails, it sleeps for a random amount of time and then retryes again, until it succeeds. Intuitively, immediate retrying always seems to achieve success the fastest. Initially, this approach was used: considering the low probability of multi-threaded conflicts, CPU consumption was ignored, and immediate retrying was always performed (without relinquishing the time slice or sleeping). However, system timeouts occurred (although the probability was extremely low). Research and analysis revealed that "immediate retrying" caused thread "resonance": multiple threads alternately acquire locks and fail, retrying millions of times, until finally one thread succeeds by chance, breaking the resonance. Therefore, the number of immediate retries was strictly limited, and the system was then advised to relinquish the time slice and sleep for a random amount of time to resolve multi-threaded resonance.

[0088] CSPPTrie does not support deleting keys and their corresponding values ​​for two reasons: first, concurrent deletion would greatly increase the complexity of the implementation; and second, in the application scenario of in-memory tables, there is no deletion operation.

[0089] In Linux x86_64, CSPPTrie's write performance is 6 times that of SkipList, and its read performance is 8 times that of SkipList. In Linux ARM, the performance is 10 times and 11 times that of SkipList, respectively.

[0090] Based on the aforementioned mechanism, other index structures can also be used to replace CSPPTrie to implement memory tables, such as skip lists or red-black trees, provided that they use memory pools and integer offsets instead of pointers for addressing, just like CSPPTrie. In the prototype construction phase, the red-black tree used as a prototype does not support lock-free concurrency.

[0091] The copy-on-write pointer set includes replica indexes, concurrent access metrics, and lock contention metrics. The memory reclamation queue specifically includes delayed release flags, token index records, and resource optimization metrics. The memory reuse table includes space reallocation records, memory release efficiency, and reuse rate metrics. The offset addressing mapping data includes thread configuration mapping, address translation efficiency, and addressing speed improvement results. The write-ahead log includes operation integrity verification results and persistent disk writes. The data storage framework specifically includes security verification parameters, latency optimization metrics, and access speed analysis results.

[0092] like Figure 2 As shown, the steps to initialize CSPPTrie, allocate the space needed by each node through MemPool, optimize concurrent access to nodes based on multi-core CPU architecture to reduce memory consumption, optimize lock contention using copy-on-write technology, create a memory copy for each node modification, and update the original node pointer through CAS operations to obtain the copy-on-write pointer set are as follows:

[0093] S101: Initialize CSPPTrie, dynamically allocate memory space for each data node based on the current data size, match the multi-core CPU architecture, reduce memory usage, and obtain the node configuration table;

[0094] First, the actual memory space required for each node is calculated. Based on the number of children of a node, the specific offset address for dynamic allocation is determined by calculating the node's memory space utilization. The offset value is represented by a 32-bit unsigned integer. During initialization, a pointer set is created for each node. At the same time, according to the number of children (cnt_type) of a node, an appropriate storage structure is selected, and different strategies are used to configure the pointer set of child nodes, including direct storage, bitmap indexing, and fixed-length arrays. The memory pool location is addressed by integer offset values, and virtual memory space is allocated to unoccupied memory areas. CAS operations are used to avoid conflicts when multiple threads modify the memory. Meanwhile, the configuration table is dynamically updated to obtain a node configuration table optimized for multi-core CPU architecture.

[0095] S102: Based on the node configuration table, create a data copy for each modification operation, optimize lock contention in concurrent operations, and use CAS to perform lock-free update of the original data pointer to obtain the copy synchronization record;

[0096] By creating a copy-on-write replica for each node, when modifying node data, a new memory block is first allocated from the memory pool as a replica, and the original node data is copied to the new memory block. Thread-local storage (InstanceTLS) is used to cache the operation records of the current thread. At the same time, a delayed release mechanism is used to ensure that the old node data will not become invalid due to reading by other threads. A token mechanism is used to track all active threads and confirm node references. Then, a CAS operation is performed on the original node pointer that needs to be updated, replacing it with the replica pointer. After the replacement is completed, the old data is added to the delayed release queue. The version order is maintained through a doubly linked list. While ensuring the consistency of operations, the lock-free update of pointers is completed, and finally a replica synchronization record is formed.

[0097] S103: Based on the copy synchronization record, integrate the pointer update of data modification operation, manage the pointers of data in memory, optimize the data access path, and obtain the copy-on-write pointer set;

[0098] All pointer update operations on nodes are tracked and managed. First, the replication records of thread operations are collected and sorted by node version number. It is confirmed that all modified node pointers correctly point to the corresponding memory addresses. The data storage location of the modified node is located based on the offset value of the memory pool. The child node pointer set of each node is scanned, and pointer management is optimized using rank bitmap or other index structures. Finally, a delayed release mechanism is used to reclaim unreferenced old nodes, and the memory usage and offset address information in the node configuration table are updated. After integration, a new copy-on-write pointer set is formed for the next stage of data operations.

[0099] like Figure 3 As shown, based on the copy-on-write pointer set, access tokens are allocated, node addresses requiring delayed release are collected and placed into a memory reclamation queue, and the memory resource reclamation process is optimized. Memory leaks and data races are avoided through tokens and delayed release mechanisms. The specific steps to obtain the memory reclamation queue are as follows:

[0100] S201: Based on the copy-on-write pointer set, control data modification permissions, verify the order and security of modification operations, and allocate and manage access tokens for each data node to obtain a token management pool;

[0101] Each node is assigned a unique access token, which contains three parts: version number, thread identifier, and operation permission. The initial version number is generated by copying the version counter of the trie. Each token is inserted into a doubly linked list to record the state of all active threads. Each access is verified to check whether the token version of the accessing thread matches the current node version. The pointer of the node that needs to be modified is updated through a CAS operation. After the update is completed, the pointer of the old version node is marked as read-only to avoid data conflicts caused by concurrent operations. At the same time, the updated node pointer and the operation thread record are synchronized, and the used tokens are added back to the token pool for subsequent operation calls, forming a token management pool containing the operation state of all threads.

[0102] S202: Based on the token management pool, monitor the status of all nodes, mark nodes that are no longer active or need to be updated, put the node addresses into the memory reclamation processing queue, and trigger resource reclamation through a scheduled task to obtain resource reclamation scheduling information.

[0103] Monitoring node status requires obtaining the current active reference information for each node, traversing the thread activity recorded in the token chain, checking whether the node has been accessed or modified, and confirming whether the node needs to be updated or is no longer active by comparing the version number in the thread token with the node version number. The pointers of the confirmed inactive nodes are added to the delayed release queue. At the same time, a trigger based on a timed task is maintained to scan the delayed release queue at set time intervals, process the nodes in the queue that meet the release conditions, remove the memory references of the corresponding nodes, update the node status records in the management pool, and update the resource reclamation information in the scheduling log to form resource reclamation scheduling information for subsequent cleanup tasks.

[0104] S203: Based on resource reclamation scheduling information, regularly clean up nodes that are no longer needed, release associated memory resources, reallocate memory, and provide it to new or nodes that are to be updated, thereby optimizing the overall memory management efficiency and obtaining a memory reclamation queue.

[0105] A complete scan of the delayed release queue is performed to collect all nodes marked as freeable. By analyzing the node memory addresses and offsets recorded in the queue, the associated memory regions are located. It is checked for any incomplete reference operations or abnormal access records. Physical memory reclamation is performed on nodes that meet the release conditions. The free list of memory released through the memory pool is updated. The pointers are replaced with integer offsets to reallocate new nodes. The allocated memory block pointers are updated to the node configuration table. The updated memory allocation records are synchronized to the log file of the memory management module. At the same time, the reasons and status information of nodes that are not released are recorded, and finally a complete memory reclamation queue is formed.

[0106] like Figure 4As shown, based on a memory reclamation queue, unnecessary nodes in memory are released periodically to optimize memory usage efficiency. The memory reclamation activities are recorded, and reclaimed memory areas are reallocated to refine memory reuse efficiency, resulting in a memory reuse table. The specific steps are as follows:

[0107] S301: Based on the memory reclamation queue, monitor memory usage status, identify and mark inactive nodes, trigger the release process, and periodically reclaim the marked nodes to obtain memory cleanup logs;

[0108] The nodes in the recycling queue are periodically scanned, and the reference status and last access time of each node are checked in turn. By comparing the allocation records of the memory pool with the current thread activity status, it is determined whether the node has been marked as inactive. The latest version number of the node is checked to see if it matches the token version number of the referencing thread. The memory of inactive nodes is marked as reclaimable using a delayed release mechanism. The memory address and offset are recorded in the recycling task list. The task list is scanned by setting a timer trigger to confirm whether the memory block of the node to be released has met the release conditions. The relevant records are removed from the memory pool, and the memory cleanup log is updated.

[0109] S302: Based on the memory cleanup log, perform memory release operations, record the release time and the amount of resources reclaimed for each node, update the memory management database, and obtain resource reclamation records;

[0110] Extract the physical memory address and associated memory block offset of the node from the log records, locate the corresponding memory region and check if it contains active references. For memory blocks that are confirmed to have no references, call the memory pool's reclamation function to release the physical memory. At the same time, reinsert the released memory blocks into the memory free list. Calculate the size of the released memory, record the amount of resources reclaimed, and write the timestamp of the release operation to the log file. Update the resource allocation table and node status table in the memory management database, record the number of reclaimed memory blocks and their occupancy ratio, and form a complete resource reclamation record.

[0111] S303: Based on resource recycling records, analyze recycling efficiency and space reuse, reallocate released resources, allocate memory space for new data requests, and obtain a memory reuse table;

[0112] The system summarizes the resource release time and memory release amount in the recycling records, extracts information on all available memory blocks in the free list, including the starting address, memory block size, and alignment offset, sorts the memory blocks in ascending order of size, prioritizes the reallocation of larger memory blocks, matches the memory requirements of new data requests with free memory blocks, and if the size of a memory block exceeds the requirement, records the offset of the remaining portion in the free list as a new memory block, updates the memory pool's allocation table to record allocated and unallocated regions, records the allocation time and data type of the reallocated memory space, and generates a memory reuse table.

[0113] like Figure 5 As shown, based on the memory reuse table, the thread-local storage configuration of the memory pool is adjusted, a memory pool with thread caching is adopted, the speed of global memory requests and thread access to the memory pool is optimized, and the integer offset addressing method is implemented to replace pointer addressing to refine memory addressing efficiency. The specific steps to obtain the offset addressing mapping data are as follows:

[0114] S401: Based on the memory reuse table, adjust the memory pool configuration, match thread-local storage, optimize the resource allocation between threads, and reduce global memory access conflicts by adjusting memory access parameters to obtain the thread resource allocation table;

[0115] Extract the free memory block information corresponding to each thread from the current free memory list, divide the memory block size according to the needs of thread-local storage, allocate the memory blocks in the free list to the thread-independent cache area, adjust the resource allocation ratio between the thread and the global memory pool by calculating the thread access frequency and historical memory usage ratio, relocate the memory start address in the thread-local storage using integer offset values, update the memory block status of the free list to allocated, set access priority through thread scheduling, increase the memory allocation for threads with high-frequency access, and limit the competition of global access threads for local memory resources, generating an optimized thread resource allocation table.

[0116] S402: Based on the thread resource allocation table, the memory address mapping method is updated by using integer offsets instead of traditional pointer addressing, which optimizes the address calculation speed and reduces processing latency, resulting in offset addressing optimization records;

[0117] By updating the memory address mapping method and using integer offsets instead of traditional pointer addressing, the memory address calculation formula is as follows:

[0118] M = B + O × A

[0119] Calculate memory address M, where B is the base address of the memory block, O represents the integer offset from the base address, and A is the alignment adjustment parameter, which is usually 4 or 8, taking 4 as an example;

[0120] Given base address B = 1024 and integer offset O = 200, calculate memory address M:

[0121] M=B+O×A=1024+200×4=1824

[0122] The result indicates that the new memory address M is 1824.

[0123] S403: Based on offset addressing optimization records, the new mapping data is integrated into memory management, and each memory request is verified to be able to locate the correct data area, ensuring efficient execution of address translation and obtaining offset addressing mapping data;

[0124] Read each address mapping relationship in the optimization record, check whether the offset value is consistent with the memory pool allocation table, update inconsistent records, recalculate the actual offset value and overwrite the old mapping data, call the memory management verification function to simulate memory requests, match the corresponding offset value for each memory request, verify whether the data area is correct, mark the records that fail verification as abnormal and generate a repair task, write the verified records to the offset mapping data table, and update the global allocation record of the memory pool to form complete offset addressing mapping data.

[0125] like Figure 6 As shown, the specific steps for constructing a memory-mapped file based on offset-addressed mapping data, matching memory and file synchronous read / write, verifying the consistency between memory and disk files through memory mapping, storing the CSPPTrie structure in the memory-mapped file, optimizing the file read / write path, monitoring the data persistence process, and generating a data write-ahead log are as follows:

[0126] S501: Based on offset addressing mapping data, configure memory-mapped files, link memory operations and files, synchronize data changes in memory to disk, optimize read-write differences, and ensure that data is updated in real time between memory and disk;

[0127] The integer offset of each piece of data written to the data write-ahead log is stored in CSPPTrie as the value associated with the key of that piece of data;

[0128] S502: Based on the real-time data synchronization mechanism, adjust the storage path of the CSPPTrie structure, perform data path mapping, and optimize the data processing latency by controlling the data flow in memory to obtain the optimized data path configuration;

[0129] Extract the current memory mapping configuration and offset value table, calculate the actual address of each node in memory and its storage location on disk, update the node configuration table to map the data in the original storage path to the new disk file, modify the base address offset value of the memory mapping region, reallocate the corresponding address space in the memory pool, write the new storage path to the node metadata, control the flow of the memory pool to transfer all write operations to the new storage path, after completing the storage path adjustment, record the offset value of all data operations and the corresponding new storage address, and generate the data path optimization configuration.

[0130] S503: Based on data path optimization configuration, perform data consistency monitoring, record the status of each data update, verify data security, and obtain data write-ahead logs;

[0131] Extract the storage path and offset value of each node from the configuration, verify whether the node data can be correctly located in the new storage path one by one, check the validity of the storage address by reading the node metadata and offset value, compare the node data in memory with the storage record in the disk file to ensure that all update operations have been written to the correct disk area, record the node identifier, offset value and operation time of each update, write the verification results to the consistency log, generate repair tasks for nodes that fail verification and reallocate the storage path, forming a complete data write-ahead log.

[0132] like Figure 7 As shown, the specific steps for generating a data storage framework, based on the data write-ahead log, are as follows: verifying the integrity and security of data writes, recording data verification activities, adjusting the data organization of memory-mapped files, optimizing data access latency and data retrieval efficiency.

[0133] S601: Based on the data write-ahead log, perform integrity checks on the data write operations, verify whether each data update meets the standards, identify and correct potential data errors, and obtain the integrity check results;

[0134] Extract the offset value, target storage path, and operation time of each data update, verify whether the updated data has been written to the correct storage location, locate the data block in the memory-mapped file by reading the offset value, compare the content of the data block with the expected value in the update record, rewrite the data block with the discrepancy, record the specific time of the correction operation and the data difference before and after the correction, generate supplementary tasks for records that have not been corrected, mark the data status as pending repair, confirm that the repaired data is consistent with the original expected value by updating the consistency log, and generate the integrity verification result.

[0135] S602: Based on the integrity check results, evaluate the organizational structure of the memory-mapped file, adjust and optimize the data access method, and obtain the data access optimization results;

[0136] The system statistically analyzes the access and correction counts of all data blocks in the test results, extracts the offset ranges of high-frequency access regions and inefficient storage regions in the memory-mapped file, re-divides the region boundaries of the memory-mapped file, prioritizes the adjustment of high-frequency access data blocks to contiguous physical address spaces, updates the memory address mapping of regions by modifying the offset value table, performs memory compression operations on inefficient storage regions to reclaim excess free space, and records the new storage path and offset of each data block after adjustment, generating optimized data access results.

[0137] S603: Based on the data access optimization results, optimize the storage logic and data flow to match the needs of large-scale data processing and obtain the data storage framework;

[0138] The logical partitions of the memory pool are reconstructed, the adjusted access paths are mapped to the corresponding memory regions, the storage path table is updated to match the new data flow requirements, the capacity of the partitions and the distribution order of data blocks are adjusted by calculating the data traffic and memory usage ratio of each partition, the flow control module is configured to manage the transmission speed between data blocks, the priority of the data processing queue is reset, and the paths of high-frequency access data blocks are assigned to the preceding positions in the processing queue, forming an optimized data storage framework that supports large-scale data processing needs.

[0139] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0140] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0141] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0142] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0143] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0144] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0145] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0146] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0147] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0148] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for implementing an LSM tree in memory, characterized in that, The method includes: Initialize CSPPTrie, allocate the space needed by each node through MemPool, optimize node concurrent access based on multi-core CPU architecture, optimize lock contention using copy-on-write technology, and update the original node pointers through CAS operations to obtain a copy-on-write pointer set; The copy-on-write pointer set includes a replica index, concurrent access metrics, and lock contention metrics; Based on the copy-on-write pointer set, access tokens are allocated, node addresses that need to be delayed in release are collected, the memory resource reclamation process is optimized, and memory leaks and data races are avoided through tokens and delayed release mechanisms to obtain a memory reclamation queue. The memory reclamation queue specifically includes delayed release flags, token index records, and resource optimization metrics. Based on the memory reclamation queue, unnecessary nodes in memory are released periodically, memory reclamation activities are recorded, and the reclaimed memory areas are reallocated to refine memory reuse efficiency and obtain a memory reuse table. The memory reuse table includes space reallocation records, memory release efficiency, and reuse rate indicators. Based on the memory reuse table, the thread-local storage configuration of the memory pool is adjusted, a memory pool with thread caching is adopted, the speed of global memory requests and thread access to the memory pool is optimized, and the integer offset addressing method is implemented to replace pointer addressing to obtain offset addressing mapping data. The offset addressing mapping data includes thread configuration mapping, address translation efficiency, and addressing speed improvement results. Based on the offset addressing mapping data, a memory-mapped file is constructed to match synchronous read and write of memory and file. The consistency between memory and disk files is verified through memory mapping, the file read and write path is optimized, and a data write-ahead log is generated. The data write-ahead log includes operation integrity verification results and persistent disk write-to-disk data. Based on the data write-ahead log, verify the integrity and security of the data write, adjust the data organization of the memory-mapped file, optimize data access latency and data access efficiency, and generate a data storage framework. The data storage framework specifically includes security verification parameters, latency optimization metrics, and access speed analysis results.

2. The method for implementing an LSM tree in memory according to claim 1, characterized in that, The specific steps for initializing CSPPTrie, allocating the space needed by each node through MemPool, optimizing concurrent access to nodes based on multi-core CPU architecture, optimizing lock contention using copy-on-write technology, and updating the original node pointers through CAS operations to obtain the copy-on-write pointer set are as follows: Initialize CSPPTrie, dynamically allocate memory space for each data node based on the current data size, match the multi-core CPU architecture, reduce memory usage, and obtain the node configuration table; Based on the node configuration table, a data copy is created for each modification operation to optimize lock contention in concurrent operations, and lock-free update of the original data pointer is performed using CAS to obtain the copy synchronization record; Based on the aforementioned copy synchronization record, the pointer updates of data modification operations are integrated, the pointers to data in memory are managed, the data access path is optimized, and a copy-on-write pointer set is obtained.

3. The method for implementing an LSM tree in memory according to claim 1, characterized in that, Based on the aforementioned copy-on-write pointer set, access tokens are allocated, node addresses requiring delayed release are collected, and the memory resource reclamation process is optimized. The specific steps for obtaining the memory reclamation queue, using tokens and a delayed release mechanism to avoid memory leaks and data races, are as follows: Based on the copy-on-write pointer set, data modification permissions are controlled, the order and security of modification operations are verified, and access tokens are allocated and managed for each data node to obtain a token management pool. Based on the token management pool, monitor the status of all nodes, mark nodes that are no longer active or need to be updated, put the node addresses into the memory reclamation processing queue, and trigger resource reclamation through a scheduled task to obtain resource reclamation scheduling information. Based on the resource reclamation scheduling information, nodes that are no longer needed are periodically cleaned up, associated memory resources are released, memory is reallocated, and made available to new or updated nodes, thus obtaining a memory reclamation queue.

4. The method for implementing an LSM tree in memory according to claim 1, characterized in that, Based on the memory reclamation queue, the steps of periodically releasing unnecessary nodes in memory, recording memory reclamation activities, and reallocating reclaimed memory areas to refine memory reuse efficiency and obtain a memory reuse table are as follows: Based on the memory reclamation queue, the memory usage status is monitored, inactive nodes are identified and marked, the release process is triggered, and the marked nodes are reclaimed periodically to obtain a memory cleanup log; Based on the memory cleanup log, perform memory release operations, record the release time and the amount of resources reclaimed for each node, update the memory management database, and obtain resource reclamation records; Based on the resource recycling records, the recycling efficiency and space reuse are analyzed, the released resources are reallocated, memory space is allocated for new data requests, and a memory reuse table is obtained.

5. The method for implementing an LSM tree in memory according to claim 1, characterized in that, Based on the memory reuse table, the thread-local storage configuration of the memory pool is adjusted, a memory pool with thread caching is adopted, the speed of global memory requests and thread access to the memory pool is optimized, and the integer offset addressing replacement pointer addressing method is implemented to obtain the offset addressing mapping data. The specific steps are as follows: Based on the memory reuse table, the memory pool configuration is adjusted, thread-local storage is matched, the resource allocation between threads is optimized, and the global memory access conflict is reduced by adjusting the memory access parameters to obtain the thread resource allocation table. Based on the thread resource allocation table, the memory address mapping method is updated by using integer offsets instead of traditional pointer addressing, which optimizes address calculation speed and reduces processing latency, resulting in offset addressing optimization records. Based on the offset addressing optimization record, the new mapping data is integrated into memory management, and it is verified whether each memory request can be located in the correct data area to obtain offset addressing mapping data.

6. The method for implementing an LSM tree in memory according to claim 5, characterized in that, By updating the mapping method of the memory addresses, integer offsets are used instead of traditional pointer addressing. Furthermore, based on factors such as page alignment and cache line optimization, the following formula is adopted: ； Calculate memory address ,in, It is the base address of the memory block. Represents the integer offset starting from the base address. These are alignment adjustment parameters.

7. The method for implementing an LSM tree in memory according to claim 1, characterized in that, Based on the offset addressing mapping data, the steps of constructing a memory-mapped file, matching memory and file synchronous read / write, verifying the consistency between memory and disk files through memory mapping, optimizing file read / write paths, and generating data write-ahead logs are as follows: Based on the offset addressing mapping data, a memory-mapped file is configured, memory operations and files are linked, data changes in memory are synchronized to disk, read and write differences are optimized, and a real-time data synchronization mechanism is obtained. Based on the real-time data synchronization mechanism, the storage path of the CSPPTrie structure is adjusted, data path mapping is performed, and the data processing latency is optimized by controlling the data flow in memory to obtain the optimized data path configuration. Based on the optimized configuration of the data path, data consistency monitoring is performed, the status of each data update is recorded, and the security of the data is verified to obtain the data write-ahead log.

8. The method for implementing an LSM tree in memory according to claim 1, characterized in that, Based on the data write-ahead log, the steps for verifying the integrity and security of data writes, adjusting the data organization of memory-mapped files, optimizing data access latency and data retrieval efficiency, and generating a data storage framework are as follows: Based on the data write-ahead log, integrity checks are performed on the data write operations to verify whether each data update meets the standards, identify and correct potential data errors, and obtain integrity check results. Based on the integrity verification results, the organizational structure of the memory-mapped file is evaluated, and the data access method is adjusted and optimized to obtain the data access optimization results. Based on the data access optimization results, the storage logic and data flow are optimized to match the needs of large-scale data processing, resulting in a data storage framework.

Images