Memory table implementation method of LSM tree

Abstract

The invention relates to the technical field of LSM trees, in particular to a memory table implementation method of an LSM tree, which comprises the following steps of: initializing CSPPrie, distributing a currently required space for each node through MemPool, and optimizing node concurrent access according to a multi-core CPU (Central Processing Unit) architecture. According to the invention, by introducing the memory management and data synchronization technology, the write-in speed and data consistency of the system are remarkably improved, by adopting the copy-on-write technology, the lock competition in the multi-core environment is effectively reduced, and by creating the data copy in the memory and updating the operation, more efficient concurrent write-in is realized, the demand of the lock is reduced, and the user experience is improved. The processing speed is improved, operation conflicts are remarkably reduced, management of memory resources is optimized through implementation of a token and a delay release mechanism, memory leakage is prevented, the resource recovery efficiency of the system is improved, full utilization of the system resources is ensured through the strategy, memory recovery is more efficient, and the resource recovery efficiency of the system is improved. And the application of the memory mapping file enables the data persistence process to be more direct and efficient.

Description

Technical Field

[0001] The present invention relates to the technical field of LSM trees, and in particular to a method for realizing a memory table of an LSM tree. Background Art

[0002] LSM tree (Log-Structured Merge-Tree) is a data structure used to achieve persistent storage. It saves data insertion, modification, deletion and other operation records in memory. When these operations reach a certain number, they are written to disk in batches and in sequence, thereby using sequential writing to improve write performance. This structure is particularly suitable for application scenarios where writing is more than reading, such as big data storage systems. The core idea of ​​LSM tree is to give up some read capabilities in exchange for maximum write capabilities, and to sacrifice a small part of read performance in exchange for high-performance writing. Each record (Record) of LSM tree contains (key, version sequence number, type, value), English (Key, Seq, Type, Value). Among them, Key and Value are byte arrays; Seq is an integer; Type indicates the type of the data, generally including ① overwrite (Put) and ② delete (Delete). Complex LSM contains more extended types.

[0003] A memtable is a mutable data structure, usually implemented using an ordered data structure (such as a balanced tree or skip list) to facilitate fast insertion, deletion, and query operations. All write operations (insert, update, and delete) are first performed in the memtable. Immutable memtable: When a memtable reaches a certain size or other conditions, it becomes an immutable memtable. An immutable memtable is read-only and waits to be flushed to disk. The main purpose of the LSM tree memtable implementation method is to improve the performance of database systems, file systems, or other applications that require high-speed data writing, while ensuring persistent storage and queryability of data.

[0004] The existing technology has the problem of low efficiency in a high-concurrency environment. The conversion process from a conventional memory table to a disk table involves complex data copying and physical writing, which not only consumes a large amount of CPU resources, but also increases operation delays. In addition, the traditional lock mechanism is prone to competition in a multi-threaded environment, resulting in low efficiency of write operations, which is particularly prominent in application scenarios that require fast processing and high throughput, such as real-time data analysis and trading systems. In addition, data processing delays or slow system responses directly affect the real-time nature of analysis results and the timeliness of transactions, resulting in lost opportunities or wrong decisions. In addition, insufficient memory management can also cause system stability issues, such as frequent memory overflows or excessive garbage collection. Summary of the invention

[0005] In order to solve the problem of low efficiency in a high-concurrency environment in the prior art, the conversion process from a conventional memory table to a disk table involves complex data copying and physical writing, which not only consumes a large amount of CPU resources, but also increases the operation delay. In addition, the traditional lock mechanism is prone to competition in a multi-threaded environment, resulting in low efficiency of write operations, which is particularly prominent in application scenarios that require fast processing and high throughput, such as real-time data analysis and trading systems. In addition, data processing delays or slow system responses directly affect the real-time nature of the analysis results and the timeliness of transactions, resulting in lost opportunities or wrong decisions. In addition, insufficient memory management can also cause system stability problems, such as frequent memory overflows or excessive garbage collection. The embodiment of the present invention provides a method for implementing a memory table of an LSM tree. The technical solution is as follows:

[0006] On the one hand, a method for implementing a memory table of an LSM tree is provided, comprising the following steps:

[0007] S1: Initialize CSPPTrie, allocate the currently required space for each node through MemPool, optimize node concurrent access based on multi-core CPU architecture, optimize lock contention using copy-on-write technology, and update the original node pointer through CAS operation to obtain a set of copy-on-write pointers;

[0008] S2: Based on the write-time copy pointer set, an access token is allocated, the node addresses that need to be released delayed are collected, the memory resource recovery process is optimized, memory leakage and data contention are avoided through the token and delayed release mechanism, and a memory recovery queue is obtained;

[0009] S3: Based on the memory recycling queue, unneeded nodes in the memory are released periodically, memory recycling activities are recorded, and the recycled memory area is reallocated to refine the memory reuse efficiency and obtain a memory reuse table;

[0010] S4: Based on the memory reuse table, adjust the thread local storage configuration of the memory pool, adopt a memory pool with thread cache, optimize the global memory request and the access speed of the thread to the memory pool, and implement integer offset addressing to replace the pointer addressing method to obtain offset addressing mapping data;

[0011] S5: Based on the offset addressing mapping data, a memory mapping file is constructed, memory and file synchronous reading and writing are matched, the consistency of memory and disk files is verified through memory mapping, the file reading and writing path is optimized, and a data pre-write log is generated;

[0012] S6: According to the data pre-write log, verify the integrity and security of data writing, adjust the data organization of the memory-mapped file, optimize data access latency and data access efficiency, and generate a data storage framework.

[0013] On the other hand, the write-time copy pointer set includes a replica index, a concurrent access index, and a lock contention index; the memory recovery queue specifically includes a delayed release identifier, a token index record, and a resource optimization index; the memory reuse table includes a space reallocation record, memory release efficiency, and a reuse rate index; the offset addressing mapping data includes a thread configuration mapping, an address conversion efficiency, and an addressing speed improvement result; the data pre-write log includes an operation integrity verification result and a persistent disk; the data storage framework specifically includes security verification parameters, delay optimization indicators, and access speed analysis results.

[0014] On the other hand, initialize CSPPTrie, allocate the space currently required for each node through MemPool, optimize node concurrent access according to the multi-core CPU architecture, optimize lock contention using copy-on-write technology, and update the original node pointer through CAS operation. The specific steps to obtain the copy-on-write pointer set are as follows:

[0015] S101: Initialize CSPPTrie, dynamically allocate memory space for each data node according to the current data size, match the multi-core CPU architecture, reduce memory usage, and obtain the node configuration table;

[0016] S102: Based on the node configuration table, a data copy is created for each modification operation, lock contention in concurrent operations is optimized, and CAS is used to perform lock-free update of the original data pointer to obtain a copy synchronization record;

[0017] S103: Based on the replica synchronization record, the pointer update of the data modification operation is integrated, the pointing of the data in the memory is managed, the data access path is optimized, and a copy-on-write pointer set is obtained.

[0018] On the other hand, based on the write-time copy pointer set, access tokens are allocated, node addresses that need to be released delayed are collected, the memory resource recovery process is optimized, and memory leaks and data contention are avoided through token and delayed release mechanisms. The steps of obtaining a memory recovery queue are specifically as follows:

[0019] S201: Based on the write-time copy pointer set, control the data modification authority, verify the order and security of the modification operation, and allocate and manage the access token of each data node to obtain a token management pool;

[0020] S202: Based on the token management pool, monitor the status of all nodes, mark nodes that are no longer active or need to be updated, put the node addresses into the memory recycling processing queue, and trigger resource recycling through a scheduled task to obtain resource recycling scheduling information;

[0021] S203: Based on the resource recycling scheduling information, nodes that are no longer needed are cleaned up regularly, associated memory resources are released, memory is reallocated, and provided to new or to-be-updated nodes for use, thereby obtaining a memory recycling queue.

[0022] On the other hand, based on the memory recycling queue, the unnecessary nodes in the memory are released regularly, the memory recycling activities are recorded, and the recycled memory areas are reallocated to refine the memory reuse efficiency. The steps of obtaining the memory reuse table are specifically as follows:

[0023] S301: Based on the memory recycling queue, monitor the memory usage status, identify and mark inactive nodes, trigger the release process, and recycle the marked nodes regularly to obtain a memory cleanup log;

[0024] S302: Based on the memory cleanup log, perform a memory release operation, record the release time and amount of resources recovered for each node, update the memory management database, and obtain a resource recovery record;

[0025] S303: Based on the resource recovery record, the recovery efficiency and space reuse are analyzed, the released resources are reallocated, memory space is allocated for new data requests, and a memory reuse table is obtained.

[0026] On the other hand, based on the memory reuse table, the thread local storage configuration of the memory pool is adjusted, a memory pool with thread cache is adopted, the global memory request and the access speed of the thread to the memory pool are optimized, and integer offset addressing is implemented to replace the pointer addressing method. The steps of obtaining the offset addressing mapping data are specifically as follows:

[0027] S401: Based on the memory reuse table, adjust the memory pool configuration, match the thread local storage, optimize the resource allocation between threads, and reduce the global memory access conflict by adjusting the memory access parameters to obtain the thread resource allocation table;

[0028] S402: Based on the thread resource allocation table, the mapping mode of the memory address is updated, and integer offset is used to replace the traditional pointer addressing, so as to optimize the address calculation speed and reduce the processing delay, and obtain the offset addressing optimization record;

[0029] S403: Based on the offset addressing optimization record, the new mapping data is integrated into the memory management, and it is verified whether each memory request can be located in the correct data area to obtain the offset addressing mapping data.

[0030] On the other hand, by updating the mapping mode of the memory address, integer offset is used to replace the traditional pointer addressing, and according to the factors of page alignment and cache line optimization, the formula is used:

[0031] M=B+O×A

[0032] Calculate the new memory address M, where B is the base address of the memory block, O represents the integer offset from the base address, and A is the alignment adjustment parameter.

[0033] On the other hand, based on the offset addressing mapping data, a memory mapping file is constructed, memory and file synchronous reading and writing are matched, the consistency of memory and disk files is verified through memory mapping, the file reading and writing path is optimized, and the steps of generating a data pre-write log are specifically as follows:

[0034] S501: Based on the offset addressing mapping data, a memory mapping file is configured, memory operations are linked to files, data changes in the memory are synchronized to disk, read-write differences are optimized, and a real-time data synchronization mechanism is obtained;

[0035] S502: Based on the real-time data synchronization mechanism, the storage path of the CSPPTrie structure is adjusted to perform data path mapping, and the data processing delay is optimized by controlling the data flow in the memory to obtain the data path optimization configuration;

[0036] S503: Based on the data path optimization configuration, perform data consistency monitoring, record the status of each data update, and verify the security of the data to obtain a data pre-write log.

[0037] On the other hand, the integer offset of each piece of data written into the data pre-write log is stored in the CSPPTrie as the value associated with the key of the piece of data.

[0038] On the other hand, according to the data pre-write log, the integrity and security of data writing are verified, the data organization of the memory-mapped file is adjusted, and the data access latency and data access efficiency are optimized. The steps of generating a data storage framework are specifically as follows:

[0039] S601: Perform integrity check on the data writing operation according to the data pre-write log, verify whether each data update complies with the standard, identify and correct potential data errors, and obtain integrity check results;

[0040] S602: Based on the integrity check result, evaluate the organizational structure of the memory mapping file, adjust and optimize the data access method, and obtain a data access optimization result;

[0041] S603: Based on the data access optimization result, optimize storage logic and data fluidity to match large-scale data processing requirements and obtain a data storage framework.

[0042] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0043] By introducing memory management and data synchronization technology, the system's write speed and data consistency are significantly improved. The use of write-time copy technology effectively reduces lock contention in a multi-core environment. By creating data copies in memory and updating operations, more efficient concurrent writing is achieved, reducing the need for locks, improving processing speed, and significantly reducing operation conflicts. The implementation of token and delayed release mechanisms optimizes memory resource management, prevents memory leaks, and enhances the system's resource recovery efficiency. This strategy ensures full utilization of system resources and makes memory recovery more efficient. The application of memory-mapped files makes the data persistence process more direct and efficient, supports instant synchronization between memory and disk, reduces the risk of data inconsistency, and improves the speed of data access. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 It is a main step flow chart of the present invention;

[0046] Figure 2 is a flow chart of the steps of S1 of the present invention;

[0047] Figure 3 is a flow chart of the steps of S2 of the present invention;

[0048] Figure 4 is a flow chart of the steps of S3 of the present invention;

[0049] Figure 5 is a flow chart of the steps of S4 of the present invention;

[0050] Figure 6 is a flow chart of the steps of S5 of the present invention;

[0051] Figure 7 This is a flow chart of the steps of S6 of the present invention. DETAILED DESCRIPTION

[0052] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0053] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0054] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0055] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0056] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0057] The embodiment of the present invention provides a method for implementing a memory table of an LSM tree, such as Figure 1 As shown, the following steps are included:

[0058] S1: Initialize CSPPTrie, allocate the space currently required for each node through MemPool, optimize node concurrent access based on multi-core CPU architecture, reduce memory usage, optimize lock contention using copy-on-write technology, create a memory copy for each node modification, and update the original node pointer through CAS operation to obtain a copy-on-write pointer set;

[0059] S2: Based on the copy-on-write pointer set, access tokens are allocated, node addresses that need to be released later are collected, and the addresses are placed in the memory recycling queue to optimize the memory resource recycling process. Memory leaks and data contention are avoided through the token and delayed release mechanism to obtain the memory recycling queue.

[0060] S3: Based on the memory recycling queue, it releases unnecessary nodes in the memory regularly, optimizes memory usage efficiency, records memory recycling activities, and reallocates the recycled memory area to refine the memory reuse efficiency and obtain a memory reuse table;

[0061] S4: Based on the memory reuse table, adjust the thread local storage configuration of the memory pool, use a memory pool with thread cache, optimize the global memory request and thread access speed to the memory pool, and implement integer offset addressing to replace the pointer addressing method to refine the memory addressing efficiency and obtain offset addressing mapping data;

[0062] S5: Based on the offset addressing mapping data, build a memory-mapped file, match the memory and file synchronous reading and writing, verify the consistency of memory and disk files through memory mapping, store the CSPPTrie structure in the memory-mapped file, optimize the file reading and writing path, monitor the data persistence process, and generate data pre-write logs;

[0063] S6: Based on the data pre-write log, verify the integrity and security of data writing, record data verification activities, adjust the data organization of memory-mapped files, and optimize data access latency and data access efficiency to generate a data storage framework.

[0064] A 32-bit unsigned integer is used as a pointer to address inside the memory pool, and 4 bytes are used as the addressing unit, so the maximum addressing range is 16G: the largest 32-bit unsigned integer is (4G-1), so the largest 32-bit integer can be addressed in the range of 0 to (4G-1)*4 with 4-byte alignment.

[0065] Trie is an automaton. The term "state" of the automaton corresponds to the term "node" of the data structure. Each Trie node has a maximum of 256 children, but in fact, the number of children of most Trie nodes does not exceed 3, so meta information is defined for each node (occupying exactly one 4-byte unit):

[0066]

[0067] When b_is_final is 1, it means that the node is in a terminal state (the terminal state of the automaton). The node in the terminal state represents a complete key, thus corresponding to a key-value pair (Key-Value-Pai r).

[0068] In fact, only the first byte is meta information. The other three bytes have other uses. The most important meta information is n_cnt_type. According to the value of n_cnt_type, the nodes are divided into the following categories:

[0069] 1. Between 0 and 6: The value of n_cnt_type is the number of children of the node. Specifically:

[0070] 1, 0: At this time, the number of children of this node is 0, and b_is_fina l must be true;

[0071] 2. 1,2: At this time, c_labe l(label) saves the label of the corresponding child of the node;

[0072] 3. 3-6: At this time, c_labe l and the next 4 bytes store the label of the corresponding child of the node.

[0073] 2. 7: At this time, the number of children is between 7 and 16. The actual number of children is placed at the c_label position and accessed by BigCount.n_children(u int16). The next 2 to 4 (4-byte) addressing units are placed at the label position.

[0074] 3. 8: At this time, the number of children is between 17 and 64. The rank-select bitmap is used to store the index. When bits[ch] is 1, it means that there is a child corresponding to ch. The rank-select bitmap can be used to calculate which bit is 1 in ch, and use it to get the child pointer in the subsequent child array. The rank index occupies 4 bytes, and the rank bitmap occupies 32 bytes, which is divided into 4 64-bit integers. The 4-byte rank index stores the popcnt prefix sum of 64-bit integers 0 to 3, so that the rank of each ch Rank(ch) = index[ch / 64]+popcnt(mask1(bitu64[ch / 64],ch%64)).

[0075] Therefore, the corresponding child = children[Rank(ch)].

[0076] 4. 15: At this time, an array of 256 child pointers is allocated for the node. The array item children[ch] is ni l, indicating that there is no child corresponding to ch. Such a node will not be reallocated again due to the insertion of a new child.

[0077] The sequence number represents the sequence number of the "4-byte unit", light gray (#1) represents zpath_l en, blue (#2) is the number of children, gold (#3) and pink orange (#4) are l abe l labels, dark green (#5) is the rank index, yellow (#6) is the rank bitmap, light green (#7) is the compression path, red (#8) is the child pointer, light purple (#9) is the value (val ue), light blue (#10) is a pointer array containing 256 children (pointers are expressed using 32-bit integers), white (#0);

[0078] The pink-orange label, compression path, and child pointer array are variable length for different nodes, while the others are fixed length:

[0079]

[0080] First, we search. Given a key, we start from the root (root, initial_state) and search for the child corresponding to the current state on the current input character. If we can't find it, it means that there is no corresponding node until the current character is input. We insert it from here. There are many specific cases. The implementation method is essentially the same as the ordinary pointer-based data structure (directly supported by the programming language). The important part is:

[0081] 1. Because insertion requires memory allocation, the nodes that were previously delayed and can now be recycled are first recycled. 2. Lock the node on which insertion is required (MetaInfo.b_lock, equivalent to row lock)

[0082] 3. For the node that needs to be inserted, allocate memory according to the size after insertion and copy it, and perform the insertion on the copy.

[0083] enter

[0084] 4. Because the number of children increases due to insertion, which in turn causes cnt_type to change, handle the resulting change in storage method 5. Initialize and copy the value corresponding to the key

[0085] 6. Lock the parent node

[0086] 7. Use CAS to replace the pointer pointing to this node in the parent node

[0087] Because of multi-threaded concurrent writing, when locking or CAS fails, first immediately retry from the beginning several times (for example, 5 times). If it still fails, give up the time slice of the current thread and retry several times (for example, 5 times). If it still fails, sleep for a random period of time and try again until it succeeds. Intuitively, the immediate retry method can always achieve success the fastest. This method was used at the beginning: considering that the probability of multi-threaded conflicts is very low, the CPU consumption is ignored, and always retries immediately (do not give up the time slice, do not sleep). However, the system timed out (although the probability is extremely low). After research and analysis, it was found that "immediate retry" caused the thread "resonance": that is, multiple threads locked and failed alternately, and retried millions of times. Finally, a thread succeeded by chance and broke the resonance. Therefore, the number of immediate retries is strictly limited, and then try to give up the time slice and sleep for a random time to solve the multi-threaded resonance.

[0088] CSPPTrie does not support deleting keys and their corresponding values ​​for two reasons. First, concurrent deletion greatly increases the complexity of implementation. Second, there is no deletion operation in the application scenario of memory tables.

[0089] In Linux x86_64, CSPPTrie's write performance is 6 times that of SkipList, and its read performance is 8 times. In Linux ARM, the performance is 10 times and 11 times, respectively.

[0090] Based on the above mechanism, other index structures can also be used instead of CSPPTrie to implement memory tables, such as skip lists or red-black trees, provided that the memory pool is used like CSPPTrie and integer offsets are used instead of pointers for addressing. In the prototype stage, the red-black tree used as a prototype does not support lock-free concurrency.

[0091] The write-time copy pointer set includes replica index, concurrent access index, and lock contention index. The memory recycling queue includes delayed release identification, token index record, and resource optimization index. The memory reuse table includes space reallocation records, memory release efficiency, and reuse rate index. The offset addressing mapping data includes thread configuration mapping, address conversion efficiency, and addressing speed improvement results. The data pre-write log includes operation integrity verification results and persistent disk. The data storage framework includes security verification parameters, delay optimization indicators, and access speed analysis results.

[0092] like Figure 2 As shown in the figure, CSPPTrie is initialized, and the space currently required for each node is allocated through MemPool. Node concurrent access is optimized according to the multi-core CPU architecture to reduce memory usage. Copy-on-write technology is used to optimize lock contention. A memory copy is created for each node modification, and the original node pointer is updated through CAS operation. The specific steps to obtain the copy-on-write pointer set are as follows:

[0093] S101: Initialize CSPPTrie, dynamically allocate memory space for each data node according to the current data size, match the multi-core CPU architecture, reduce memory usage, and obtain the node configuration table;

[0094] Firstly, the actual required memory space is calculated for each node. According to the number of children of the node, the specific offset address of dynamic allocation is determined by calculating the node memory space utilization rate. The offset value is represented by a 32-bit unsigned integer. A pointer set is created for each node during initialization. At the same time, according to the number of children of the node (cnt_type), the corresponding storage structure is selected. Different strategies are used to configure the pointer set of the child node, including direct storage, bitmap index and fixed-length array. The memory pool location is addressed by the integer offset value, and virtual memory space is allocated to the unoccupied memory area. The conflict during multi-threaded modification is avoided through CAS operation. At the same time, the configuration table is dynamically updated to obtain the node configuration table optimized based on the multi-core CPU architecture.

[0095] S102: Based on the node configuration table, a data copy is created for each modification operation, lock contention in concurrent operations is optimized, and CAS is used to perform lock-free update of the original data pointer to obtain a copy synchronization record;

[0096] By creating a copy-on-write replica for each node, when performing a modification operation on the node data, first allocate a new memory block from the memory pool as a replica, copy the original node data to the new memory block, use thread local storage (InstanceTLS) to cache the operation records of this thread, and at the same time, use a delayed release mechanism to ensure that the old node data will not become invalid due to reading by other threads. A token mechanism is used to track all active threads and confirm node references. Then, a CAS operation is performed on the original node pointer that needs to be updated, replacing it with a replica pointer, and adding the old data to the delayed release queue after the replacement is completed. The version order is maintained through a bidirectional linked list. While ensuring the consistency of the operation, the lock-free update of the pointer is completed, and finally a replica synchronization record is formed.

[0097] S103: Based on the replica synchronization record, the pointer update of the data modification operation is integrated, the pointing of the data in the memory is managed, the data access path is optimized, and a copy-on-write pointer set is obtained;

[0098] Track and manage the pointer update operations of all nodes. First, collect the copy synchronization records of thread operations, sort them by node version number, confirm whether all modified node pointers correctly point to the corresponding memory address, locate the data storage location of the modified node according to the offset value of the memory pool, scan the child node pointer set of each node, use the ranked bitmap or other index structure to optimize pointer management, and finally use the delayed release mechanism to recycle unreferenced old nodes. At the same time, update the memory usage and offset address information in the node configuration table. After the integration is completed, a new write-time copy pointer set is formed for the next stage of data operations.

[0099] like Figure 3 As shown in the figure, based on the copy-on-write pointer set, access tokens are allocated, node addresses that need to be released delayed are collected, and the addresses are placed in the memory recycling queue to optimize the memory resource recycling process. Memory leaks and data contention are avoided through the token and delayed release mechanism. The specific steps of obtaining the memory recycling queue are as follows:

[0100] S201: Based on the copy-on-write pointer set, control the data modification authority, verify the order and security of the modification operation, and allocate and manage the access token of each data node to obtain a token management pool;

[0101] A unique access token is assigned to each node. The token contains three parts: version number, thread identifier and operation permission. The initial version number is generated by copying the version counter of the dictionary tree. Each token is inserted into a bidirectional linked list to record the status of all active threads. Each access is verified to check whether the token version of the accessing thread matches the current node version. The node pointer that needs to be modified is updated through CAS operation. After the update is completed, the old version of the node pointer is marked as read-only to avoid data conflicts caused by concurrent operations. At the same time, the updated node pointer and operation thread record are synchronized, and the used tokens are re-added to the token pool for subsequent operation calls, forming a token management pool containing the operation status of all threads.

[0102] S202: Based on the token management pool, monitor the status of all nodes, mark nodes that are no longer active or need to be updated, put the node addresses into the memory recycling processing queue, and trigger resource recycling through a scheduled task to obtain resource recycling scheduling information;

[0103] Monitoring the node status requires obtaining the current active reference information of each node, traversing the thread activities recorded in the token list, checking whether the node has been accessed or modified, and confirming whether the node needs to be updated or is no longer active by comparing the version number in the thread token with the node version number. The confirmed inactive node pointer is added to the delayed release queue. At the same time, a trigger based on a scheduled task is maintained to scan the delayed release queue at set time intervals, process the node marks in the queue that meet the release conditions, remove the memory references of the corresponding nodes, update the node status records of the management pool, and update the resource recovery information to the scheduling log to form resource recovery scheduling information for subsequent cleanup tasks.

[0104] S203: Based on the resource recycling scheduling information, regularly clean up nodes that are no longer needed, release associated memory resources, reallocate memory, and provide it to new or updated nodes for use, optimize the overall memory management efficiency, and obtain a memory recycling queue;

[0105] Perform a complete scan of the delayed release queue, collect all nodes marked as releasable, locate the associated memory area by analyzing the node memory address and offset value recorded in the queue, check whether there are any unfinished reference operations or abnormal access records, perform physical memory recovery on nodes that meet the release conditions, update the free list record of memory released by the memory pool, reallocate the new node by replacing the pointer with an integer offset, update the allocated memory block pointer to the node configuration table, synchronize the updated memory allocation record to the log file of the memory management module, and record the reasons and status information of the unreleased nodes, finally forming a complete memory recovery queue.

[0106] like Figure 4As shown in the figure, based on the memory recycling queue, the unnecessary nodes in the memory are released regularly, the memory usage efficiency is optimized, the memory recycling activities are recorded, and the recycled memory areas are reallocated to refine the memory reuse efficiency. The specific steps of obtaining the memory reuse table are as follows:

[0107] S301: Based on the memory recycling queue, monitor the memory usage status, identify and mark inactive nodes, trigger the release process, and recycle the marked nodes regularly to obtain the memory cleaning log;

[0108] Scan the nodes in the recycling queue regularly, check the reference status and the last access time of each node in turn, determine whether the node has been marked as inactive by comparing the allocation record of the memory pool and the activity status of the current thread, check whether the latest version number of the node matches the token version number of the referencing thread, use the delayed release mechanism to mark the memory of the inactive node as recyclable, record the memory address and offset in the recycling task list, scan the task list by setting a timer trigger, confirm whether the memory block of the node to be released has met the release conditions, remove the relevant records from the memory pool, and update the memory cleanup log.

[0109] S302: Based on the memory cleanup log, perform a memory release operation, record the release time and amount of resources recovered for each node, update the memory management database, and obtain a resource recovery record;

[0110] According to the log records, the physical memory address of the node and the associated memory block offset value are extracted, the corresponding memory area is located and checked whether it contains active references, and the memory pool's recycling function is called for the memory block that is confirmed to have no references to release the physical memory. At the same time, the released memory block is reinserted into the memory free list, and the amount of reclaimed resources is recorded by calculating the size of the released memory. The timestamp of the release operation is written into the log file, and the resource allocation table and node status table in the memory management database are updated to record the number of reclaimed memory blocks and the occupancy ratio to form a complete resource recovery record.

[0111] S303: Based on the resource recovery records, the recovery efficiency and space reuse are analyzed, the released resources are reallocated, memory space is allocated for new data requests, and a memory reuse table is obtained;

[0112] Summarize the resource release time and released memory amount in the recovery record, extract all available memory block information in the free list, including the starting address, memory block size and alignment offset value, sort the memory blocks from small to large in size, give priority to larger memory blocks for reallocation, match the memory requirements of new data requests with free memory blocks, if the size of the memory block exceeds the requirement, record the offset value of the remaining part in the free list as a new memory block, update the allocation table of the memory pool to record the allocated and unallocated areas, and record the allocation time and data type for the reallocated memory space to generate a memory reuse table.

[0113] like Figure 5 As shown, based on the memory reuse table, the thread local storage configuration of the memory pool is adjusted, a memory pool with thread cache is used, the global memory request and thread access speed to the memory pool are optimized, and integer offset addressing is implemented to replace the pointer addressing method to refine the memory addressing efficiency. The specific steps of obtaining the offset addressing mapping data are as follows:

[0114] S401: Based on the memory reuse table, adjust the memory pool configuration, match the thread local storage, optimize the resource allocation between threads, and reduce the global memory access conflict by adjusting the memory access parameters to obtain the thread resource allocation table;

[0115] Extract the free memory block information corresponding to each thread from the current memory free list, divide the memory block size according to the needs of thread local storage, allocate the memory blocks in the free list to the thread independent cache area, adjust the resource allocation ratio of the thread and the global memory pool by calculating the thread access frequency and the historical memory occupancy ratio, use the integer offset value to relocate the memory start address in the thread local storage, update the memory block status of the free list to allocated, set the access priority through thread scheduling, increase the memory allocation for high-frequency access threads, and limit the competition for local memory resources by global access threads to generate an optimized thread resource allocation table.

[0116] S402: Based on the thread resource allocation table, the mapping mode of the memory address is updated, and integer offset is used to replace the traditional pointer addressing, so as to optimize the address calculation speed and reduce the processing delay, and obtain the offset addressing optimization record;

[0117] By updating the mapping method of memory addresses, integer offsets are used instead of traditional pointer addressing. The memory address calculation formula is:

[0118] M=B+O×A

[0119] Calculate the memory address M, where B is the base address of the memory block, O represents the integer offset from the base address, and A is the alignment adjustment parameter, which is usually 4 or 8. Take 4 as an example.

[0120] Base address B = 1024, integer offset O = 200, calculate memory address M:

[0121] M=B+O×A=1024+200×4=1824

[0122] The result shows that the new memory address M is 1824.

[0123] S403: Based on the offset addressing optimization record, the new mapping data is integrated into the memory management, and it is verified whether each memory request can be located in the correct data area to ensure efficient execution of address conversion and obtain the offset addressing mapping data;

[0124] Read each address mapping relationship in the optimization record, check whether the offset value is consistent with the allocation table of the memory pool, update the inconsistent records, recalculate the actual offset value and overwrite the old mapping data, and call the verification function of the memory management to simulate the memory request, match the corresponding offset value for each memory request, verify whether the data area is correct, mark the records that fail the verification as abnormal and generate a repair task, write the records that pass the verification into the offset mapping data table, and update it to the global allocation record of the memory pool to form a complete offset addressing mapping data.

[0125] like Figure 6 As shown in the figure, based on the offset addressing mapping data, a memory mapping file is constructed, the memory and file are matched for synchronous reading and writing, the consistency of the memory and disk file is verified through memory mapping, the CSPPTrie structure is stored in the memory mapping file, the file reading and writing path is optimized, the data persistence process is monitored, and the steps of generating a data pre-write log are as follows:

[0126] S501: Mapping data based on offset addressing, configuring memory-mapped files, linking memory operations with files, synchronizing data changes in memory to disk, optimizing read-write differences, and ensuring that data is updated instantly between memory and disk;

[0127] The integer offset of each data entry written to the data pre-write log is stored in the CSPPTrie as the value associated with the key of the data entry;

[0128] S502: Based on the real-time data synchronization mechanism, the storage path of the CSPPTrie structure is adjusted to perform data path mapping, and the data processing delay is optimized by controlling the data flow in the memory to obtain the data path optimization configuration;

[0129] Extract the current memory mapping configuration and offset value table, calculate the actual address of each node in the memory and the storage location in the disk, map the data in the original storage path to the new disk file by updating the node configuration table, modify the base address offset value of the memory mapping area, reallocate the corresponding address space in the memory pool, write the new storage path to the node metadata, and transfer all write operations to the new storage path by controlling the flow direction of the memory pool. After completing the storage path adjustment, record the offset values ​​and corresponding new storage addresses of all data operations to generate a data path optimization configuration.

[0130] S503: Based on the data path optimization configuration, perform data consistency monitoring, record the status of each data update, and verify the security of the data to obtain a data pre-write log;

[0131] Extract the storage path and offset value of each node from the configuration, verify one by one whether the node data can be correctly located in the new storage path, check the validity of the storage address by reading the node metadata and offset value, compare the node data in the memory with the storage record in the disk file, ensure that all update operations have been written to the correct disk area, record the node identification, offset value and operation time of each update, write the verification results to the consistency log, generate repair tasks for nodes that fail the verification and reallocate the storage path to form a complete data pre-write log.

[0132] like Figure 7 As shown, according to the data pre-write log, the integrity and security of data writing are verified, data verification activities are recorded, the data organization of the memory-mapped file is adjusted, and data access latency and data access efficiency are optimized. The specific steps for generating a data storage framework are:

[0133] S601: Perform integrity check on the data writing operation according to the data pre-write log, verify whether each data update complies with the standard, identify and correct potential data errors, and obtain integrity check results;

[0134] Extract the offset value, target storage path and operation time of each data update, verify one by one whether the updated data is written to the correct storage location, locate the data block in the memory mapping file by reading the offset value, compare the content of the data block with the expected value in the update record, rewrite the data block with differences, record the specific time of the correction operation and the data difference before and after the correction, generate supplementary tasks for the records of unfinished corrections, mark the data status as pending repair, confirm that the repaired data is consistent with the original expected value by updating the consistency log, and generate integrity check results.

[0135] S602: Based on the integrity check result, evaluate the organizational structure of the memory mapping file, adjust and optimize the data access method, and obtain a data access optimization result;

[0136] The access and correction times of all data blocks in the statistical test results are counted, the offset ranges of high-frequency access areas and inefficient storage areas in the memory-mapped file are extracted, the regional boundaries of the memory-mapped file are redivided, and the high-frequency access data blocks are adjusted to the continuous physical address space in priority. The memory address mapping of the area is updated by modifying the offset value table, and the memory compression operation is performed on the inefficient storage area to reclaim excess free space. At the same time, the new storage path and offset of each data block after adjustment are recorded to generate optimized data access results.

[0137] S603: Based on the data access optimization result, optimize the storage logic and data liquidity to match the large-scale data processing requirements and obtain a data storage framework;

[0138] Reconstruct the logical partitions of the memory pool, map the adjusted access paths to the corresponding memory areas, update the storage path table to match the new data flow requirements, calculate the data flow and memory occupancy ratio of each partition, adjust the partition capacity and data block distribution order, configure the flow control module to manage the transmission speed between data blocks, reset the priority of the data processing queue, and assign the paths of high-frequency access data blocks to the front position in the processing queue, forming an optimized data storage framework that supports large-scale data processing needs.

[0139] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0140] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0141] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0142] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0143] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0144] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0145] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0146] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0147] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0148] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for implementing a memory table of an LSM tree, characterized in that: The method comprises: Initialize CSPPTrie, allocate the space currently required for each node through MemPool, optimize node concurrent access based on multi-core CPU architecture, optimize lock contention using copy-on-write technology, and update the original node pointer through CAS operation to obtain a set of copy-on-write pointers; Based on the write-time copy pointer set, access tokens are allocated, node addresses that need to be released delayed are collected, and the memory resource recovery process is optimized. Memory leaks and data contention are avoided through token and delayed release mechanisms to obtain a memory recovery queue; Based on the memory recycling queue, the unnecessary nodes in the memory are released regularly, the memory recycling activities are recorded, and the recycled memory areas are reallocated to refine the memory reuse efficiency and obtain a memory reuse table; Based on the memory reuse table, the thread local storage configuration of the memory pool is adjusted, a memory pool with thread cache is adopted, the global memory request and the access speed of the thread to the memory pool are optimized, and integer offset addressing is implemented to replace the pointer addressing method to obtain offset addressing mapping data; Based on the offset addressing mapping data, a memory mapping file is constructed to match memory and file synchronous reading and writing, the consistency of memory and disk files is verified through memory mapping, the file reading and writing path is optimized, and a data pre-write log is generated; According to the data pre-write log, the integrity and security of data writing are verified, the data organization of the memory-mapped file is adjusted, and the data access latency and data access efficiency are optimized to generate a data storage framework.

2. The method for implementing a memory table of an LSM tree according to claim 1, characterized in that: The write-time copy pointer set includes a replica index, a concurrent access index, and a lock contention index; the memory recovery queue specifically includes a delayed release identifier, a token index record, and a resource optimization index; the memory reuse table includes a space reallocation record, a memory release efficiency, and a reuse rate index; the offset addressing mapping data includes a thread configuration mapping, an address conversion efficiency, and an addressing speed improvement result; the data pre-write log includes an operation integrity verification result and a persistent disk; the data storage framework specifically includes security verification parameters, a delay optimization index, and an access speed analysis result.

3. The method for implementing a memory table of an LSM tree according to claim 1, characterized in that: Initialize CSPPTrie, allocate the space currently required for each node through MemPool, optimize node concurrent access based on multi-core CPU architecture, optimize lock contention using copy-on-write technology, and update the original node pointer through CAS operation. The specific steps to obtain the copy-on-write pointer set are as follows: Initialize CSPPTrie, dynamically allocate memory space for each data node according to the current data size, match the multi-core CPU architecture, reduce memory usage, and obtain the node configuration table; Based on the node configuration table, a data copy is created for each modification operation, lock contention in concurrent operations is optimized, and CAS is used to perform lock-free update of the original data pointer to obtain a copy synchronization record; Based on the replica synchronization record, the pointer update of the data modification operation is integrated, the pointing of the data in the memory is managed, the data access path is optimized, and a write-time copy pointer set is obtained.

4. The method for implementing a memory table of an LSM tree according to claim 1, characterized in that: Based on the write-time copy pointer set, access tokens are allocated, node addresses that need to be released delayed are collected, and the memory resource recovery process is optimized. Memory leaks and data contention are avoided through token and delayed release mechanisms. The steps of obtaining a memory recovery queue are specifically as follows: Based on the write-time copy pointer set, control the data modification authority, verify the order and security of the modification operation, and allocate and manage the access token of each data node to obtain a token management pool; Based on the token management pool, monitor the status of all nodes, mark nodes that are no longer active or need to be updated, put the node addresses into the memory recycling processing queue, and trigger resource recycling through scheduled tasks to obtain resource recycling scheduling information; Based on the resource recycling scheduling information, nodes that are no longer needed are regularly cleaned up, associated memory resources are released, memory is reallocated, and provided to new or to-be-updated nodes for use, thereby obtaining a memory recycling queue.

5. The method for implementing a memory table of an LSM tree according to claim 1, characterized in that: Based on the memory recycling queue, the unnecessary nodes in the memory are released regularly, the memory recycling activities are recorded, and the recycled memory area is reallocated to refine the memory reuse efficiency. The steps of obtaining the memory reuse table are specifically as follows: Based on the memory recycling queue, monitor the memory usage status, identify and mark inactive nodes, trigger the release process, and recycle the marked nodes regularly to obtain a memory cleanup log; Based on the memory cleanup log, a memory release operation is performed, the release time and the amount of resources recovered of each node are recorded, a memory management database is updated, and a resource recovery record is obtained; Based on the resource recovery record, the recovery efficiency and space reuse are analyzed, the released resources are reallocated, memory space is allocated for new data requests, and a memory reuse table is obtained.

6. The method for implementing a memory table of an LSM tree according to claim 1, characterized in that: Based on the memory reuse table, the thread local storage configuration of the memory pool is adjusted, a memory pool with thread cache is adopted, the global memory request and the access speed of the thread to the memory pool are optimized, and integer offset addressing is implemented to replace the pointer addressing method. The steps of obtaining the offset addressing mapping data are specifically as follows: Based on the memory reuse table, the memory pool configuration is adjusted to match the thread local storage, the resource allocation between threads is optimized, and the global memory access conflict is reduced by adjusting the memory access parameters to obtain the thread resource allocation table; Based on the thread resource allocation table, the mapping mode of the memory address is updated, and integer offset is used to replace the traditional pointer addressing, so as to optimize the address calculation speed and reduce the processing delay, and obtain the offset addressing optimization record; Based on the offset addressing optimization record, the new mapping data is integrated into the memory management, and it is verified whether each memory request can be located in the correct data area to obtain the offset addressing mapping data.

7. The method for implementing a memory table of an LSM tree according to claim 6, characterized in that: By updating the mapping mode of the memory address, integer offset is used to replace the traditional pointer addressing, and according to the factors of page alignment and cache line optimization, the formula is adopted: M=B+O×A Calculate the memory address M, where B is the base address of the memory block, O represents the integer offset from the base address, and A is the alignment adjustment parameter.

8. The method for implementing a memory table of an LSM tree according to claim 1, characterized in that: Based on the offset addressing mapping data, a memory mapping file is constructed, memory and file synchronous reading and writing are matched, the consistency of memory and disk files is verified through memory mapping, the file reading and writing path is optimized, and the steps of generating a data pre-write log are specifically as follows: Based on the offset addressing mapping data, a memory mapping file is configured, memory operations are linked to files, data changes in the memory are synchronized to disk, read-write differences are optimized, and a real-time data synchronization mechanism is obtained; Based on the real-time data synchronization mechanism, the storage path of the CSPPTrie structure is adjusted to perform data path mapping, and the data processing delay is optimized by controlling the data flow in the memory to obtain the data path optimization configuration; Based on the data path optimization configuration, data consistency monitoring is performed, the status of each data update is recorded, and the security of the data is verified to obtain a data pre-write log.

9. The method for implementing a memory table of an LSM tree according to claim 1, characterized in that: According to the data pre-write log, the integrity and security of data writing are verified, the data organization of the memory-mapped file is adjusted, and the data access latency and data access efficiency are optimized. The steps of generating a data storage framework are specifically as follows: According to the data pre-write log, the data write operation is checked for integrity, whether each data update complies with the standard, and potential data errors are identified and corrected to obtain an integrity check result; Based on the integrity check result, the organizational structure of the memory mapping file is evaluated, and the data access method is adjusted and optimized to obtain a data access optimization result; Based on the data access optimization results, the storage logic and data fluidity are optimized to match large-scale data processing requirements and obtain a data storage framework.

Images