Key value indexing method and device, electronic equipment and storage medium

By employing a multi-level adaptive optimization key-value storage system with dynamic tiering and collaborative management mechanisms, the problems of high query latency, read amplification, write amplification, and poor performance stability in LSM-Tree are solved, achieving high-performance and stable data storage.

CN121233591APending Publication Date: 2025-12-30CHINA TELECOM NETWORK SECURITY TECH CO LTD

Patent Information

Application Number
CN202511332060.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-09-17
Publication Date
2025-12-30

AI Technical Summary

Technical Problem

Existing LSM-Tree key-value storage systems suffer from high query latency and read amplification, write amplification and background overhead, difficulties in learning index integration, low utilization of data hotspots, and poor performance stability.

Method used

The key-value storage system adopts a multi-level adaptive optimization. It intelligently senses and divides data levels through a data dynamic layering module. Combined with dedicated storage and indexing strategies for hot data layers, warm data layers, and cold data layers, it utilizes a learning index management module and garbage collection to achieve global adaptive management and dynamically adjust strategies to optimize system performance.

Benefits of technology

Significantly reduces query latency and read amplification, reduces write amplification and background overhead, improves cache hit rate and resource utilization, and enhances system performance stability and predictability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121233591A_ABST
    Figure CN121233591A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses a key value indexing method and device, electronic equipment and a storage medium, and the method comprises the steps: responding to an access operation for a specific key, and determining heat information of the specific key; the popularity information comprises the latest access frequency and the latest access timestamp of a specific range to which the specific key belongs; the specific range is obtained by dividing the whole key value space; determining a target data layer to which the specific key belongs according to the popularity information and set hierarchical parameters; the target data layer comprises a hot data layer, a temperature data layer or a cold data layer; and indexing based on an indexing strategy of the target data layer, and determining a specific value corresponding to the specific key. By applying the key value indexing method provided by the embodiment of the invention, the performance of data storage can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data storage, and particularly relates to a key-value index method and device, electronic equipment and storage medium. BACKGROUND

[0002] Key-value index is an important part of data storage, and the log-structured merge tree has become a widely used core architecture of modern high-performance key-value storage systems (such as RocksDB, LevelDB, Cassandra, etc.) due to its excellent write performance. The basic principle is to convert random write into write to the memory buffer, and then sequentially flush to the disk to form a sorted string table when the buffer is full, and through a multi-layer merging process in the background to manage the data on the disk, maintain data order and recycle invalid space.

[0003] However, the inherent structural characteristics of the log-structured merge tree have the following problems: high query delay and read amplification, write method and background overhead, learning index integration difficulty, low data hotspot utilization, and poor performance stability.

[0004] Therefore, there is an urgent need for a key-value index method to overcome the problems in the log-structured merge tree and to achieve high-performance data storage. SUMMARY

[0005] The embodiments of the present application provide a key-value index method and device, electronic equipment and storage medium to improve the performance of data storage.

[0006] In a first aspect, an embodiment of the present application provides a key-value index method, which at least includes the following steps:

[0007] In response to an access operation on a specific key, determining the hotness information of the specific key; wherein the hotness information includes the latest access frequency and the latest access timestamp of a specific range to which the specific key belongs; the specific range is obtained by dividing the entire key-value space;

[0008] According to the hotness information and a set hierarchical parameter, determining a target data layer to which the specific key belongs; wherein the target data layer is a hot data layer, a warm data layer or a cold data layer;

[0009] Indexing based on the index strategy of the target data layer to determine the specific value corresponding to the specific key.

[0010] In this embodiment, firstly, upon receiving an access operation for a specific key, the popularity information of that key can be determined. This popularity information includes the latest access frequency and the latest access timestamp for a specific range to which the key belongs. Secondly, by integrating and setting hierarchical parameters, the target data layer to which the specific key belongs can be determined based on the popularity information, such as a hot data layer, a moderate data layer, and a cold data layer. Finally, indexing can be performed based on the target data's indexing strategy to determine the specific value corresponding to the specific key. This design offers relatively fast indexing speed and efficiency, thereby improving data storage performance.

[0011] Secondly, one embodiment of this application provides a key-value indexing device, including:

[0012] The response unit is used to: determine the popularity information of a specific key in response to an access operation to that key; wherein the popularity information includes the latest access frequency and the latest access timestamp of a specific range to which the specific key belongs; the specific range is obtained by dividing the entire key-value space;

[0013] The processing unit is used to: determine the target data layer to which a specific key belongs based on heat information and set hierarchical parameters; wherein the target data layer includes a hot data layer, a warm data layer, or a cold data layer;

[0014] The processing unit is used to: index based on the indexing strategy of the target data layer and determine the specific value corresponding to a specific key.

[0015] Thirdly, one embodiment of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the above methods.

[0016] Fourthly, one embodiment of this application provides a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implement the steps of any of the above methods.

[0017] Fifthly, one embodiment of this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above methods. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1A schematic diagram of the core framework of a key-value indexing method provided in an embodiment of this application;

[0020] Figure 2 This application provides an overall architecture diagram of a multi-level adaptive optimized LSM tree key-value storage system according to an embodiment of the present application.

[0021] Figure 3 A flowchart illustrating a key-value indexing method provided in an embodiment of this application;

[0022] Figure 4 A schematic diagram illustrating the core functions of a dynamic data layering module provided in an embodiment of this application;

[0023] Figure 5 A schematic diagram of a heuristic data hierarchical threshold decision logic flow provided in an embodiment of this application;

[0024] Figure 6 This is a schematic diagram illustrating a hierarchical storage and indexing data flow according to an embodiment of this application;

[0025] Figure 7 A schematic diagram of an AMM learning index task decision-making and scheduling process provided in an embodiment of this application;

[0026] Figure 8 A schematic diagram of a storage system GC triggering and scheduling optimization process provided in an embodiment of this application;

[0027] Figure 9 This is a schematic diagram of the structure of a key-value indexing device provided in an embodiment of this application;

[0028] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0030] For ease of understanding, the terms used in the embodiments of this application are explained below:

[0031] (1) Log-Structured Merge-Tree (LSM-Tree): A merge tree based on log structure, widely used in key-value storage systems. It optimizes write performance by converting random writes into sequential writes. It typically includes a memory buffer in memory and a sorted string table file on multiple layers of disk.

[0032] (2) KV Store: A database or storage system for storing, retrieving and managing key-value pairs of data.

[0033] (3) Read / Write Amplification: The amount of data that a read operation needs to access far exceeds the amount of data requested (read amplification); the amount of data actually written to physical storage by a write operation far exceeds the amount of data written by the user (write amplification), which is mainly caused by merging / compression in LSM trees.

[0034] (4) Compaction: The process of merging data from lower levels to higher levels in an LSM tree to clean up invalid data, maintain data order, and control the level size.

[0035] (5) Sorted String Table (SSTable): The file format in which LSM trees store data on disk, with internal key-value pairs sorted by key.

[0036] (6) Learned Index (LI): Use machine learning models (such as neural networks, linear regression, etc.) to learn the distribution of data keys and predict the location or existence of keys in the storage medium to replace or enhance traditional index structures (such as B+ trees, hash tables).

[0037] (7) Cumulative Distribution Function (CDF): A function that describes the probability of a value in the dataset being less than or equal to a certain key. The learning index predicts the location of the key by approximating the CDF.

[0038] (8) Recursive Model Index (RMI): A hierarchical / stage learning index architecture where the top-level model selects the next-level model based on the key, gradually narrowing the search range and improving the search accuracy of the "last mile".

[0039] (9) Key-Value Separation: An optimization technique that stores keys and pointers to values ​​in an LSM tree structure, while storing larger values ​​in a separate log structure (such as a Value Log) to reduce write amplification during compaction.

[0040] (10) Value Log (vLog): In key-value separation, a log file used to sequentially append and store actual value data.

[0041] (11) Garbage Collection (GC): In key-value separation, the process of reclaiming the space occupied by invalid or outdated values ​​in the Value Log.

[0042] (12) Hotspot: Key-value pairs or key ranges that are frequently accessed or updated in a dataset.

[0043] (13) Fine-grained cache: caches individual key-value pairs instead of caching the entire data block as in a block cache.

[0044] (14) Retraining: The process of retraining the model when the data distribution changes and the accuracy of the learning index model decreases.

[0045] (15) Cost-Benefit Model: A model used to evaluate the trade-off between the potential benefits (such as improved query performance) of performing an operation (such as retraining) and the required costs (such as computing resources and time overhead).

[0046] (16) Query Tiering: Based on the access frequency or "temperature" (cold / warm / hot) of the data, it is divided into different storage or index tiers and different optimization strategies are adopted.

[0047] (17) Adaptive Management: The ability of the system to dynamically adjust its internal strategies (such as cache replacement, index selection, and retraining scheduling) based on the workload characteristics and performance indicators monitored in real time.

[0048] The number of any elements in the accompanying drawings is for illustrative purposes only and not as a limitation, and any naming is for distinction only and has no limiting meaning.

[0049] In related technologies, among existing LSM-Tree key-value storage systems, the log structure merged tree, due to its excellent write performance, has become the core architecture widely adopted by modern high-performance key-value storage systems (such as RocksDB, LevelDB, Cassandra, etc.). Its basic principle is to transform random writes into writes to a memory buffer (Memtable). Once the buffer is full, it is sequentially flushed to disk to form a sorted string table (SSTable). A background multi-level compaction process manages the data on disk, maintaining data order and reclaiming invalid space.

[0050] However, the inherent structural characteristics of LSM-Trees also bring performance challenges, leading to the following optimization solutions:

[0051] (1) Key-Value Separation Technique: To address the severe write amplification issue in LSM-Trees when processing large values, some related technologies propose storing key and value pointers within the LSM-Tree structure, while appending the actual large value data to a dedicated Value Log file. This method significantly reduces the amount of data moved during compaction and minimizes write amplification. However, it introduces the need for garbage collection of the Value Log to reclaim the space occupied by invalid values.

[0052] (2) Learning Indexing Techniques: Given that the general design of traditional index structures (such as B+ trees) may not fully utilize the characteristics of data distribution, Kraska et al. proposed using machine learning models to replace or enhance indexes. Learning indexes predict the location of keys in the storage medium by learning the cumulative distribution function of the keys, theoretically reducing the search complexity from logarithmic to near constant level. Structures such as recursive model indexes further address the accuracy and "last-mile" search problems of single-model indexes.

[0053] (3) Preliminary integration of learned indexes with LSM-Tree: To optimize the query performance of LSM-Tree, some studies have applied learned indexes to LSM-Tree. Common applications include: using LI to replace the block index inside the SSTable file used to locate data blocks, thereby accelerating in-file lookups; proposing adaptive training strategies to try to balance model training overhead and query benefits; and exploring data layout strategies based on model predictions to reduce lookup errors.

[0054] (4) Optimization of caching mechanism: LSM-Tree systems generally use memory-based block cache to cache SSTable data blocks, index blocks, and filter blocks to reduce access latency of slow storage (such as disks and SSDs). In view of the limitations of standard block cache (such as coarse granularity and inability to accurately hit hot spots), AC-Key and other works have proposed more refined and adaptive cache management strategies.

[0055] (5) Exploration of Hotspot Awareness and Handling: Recognizing that access skew (hotspot) is a common phenomenon in real-world workloads, some studies have begun to explore how to utilize hotspot information in LSM-Trees. This includes attempting to introduce the concept of hotspots into cache management or compaction strategies, or performing data tiering based on hotspots.

[0056] Although the optimization schemes in the aforementioned related technologies have improved the performance of LSM-Tree key-value storage systems in certain aspects, the following technical problems still generally exist:

[0057] (1) Limitations of traditional LSM-Tree architecture and basic optimization schemes.

[0058] The read amplification problem persists: even with Bloom filters and block caching, the inherent multi-level lookup mechanism of LSM-Trees (especially in deep or wide-level structures) inevitably leads to high read amplification, making it difficult to meet the query latency requirements of low-latency applications. While key-value separation can reduce the size of LSM-Trees, it adds an extra Value Log read overhead when accessing values.

[0059] Write amplification and background overhead remain significant: the compaction of standard LSM-Trees introduces a large amount of data rewriting, leading to high write amplification and background I / O pressure. While key-value separation technology alleviates write amplification of values, it introduces a complex garbage collection (GC) mechanism. The GC process (scanning, verification, data copying, and metadata updating) itself consumes a large amount of CPU and I / O resources and may block foreground operations or cause performance fluctuations. Reinsertion of GC-collected data into the LSM-Tree further triggers intensive write I / O.

[0060] Coarse-grained caching is inefficient: Standard block caching mechanisms are coarse-grained and have difficulty accurately hitting access hotspots. They may cache a large number of cold data blocks, resulting in low memory utilization and an unsatisfactory cache hit rate under high skew load.

[0061] (2) Challenges of existing learning index integration schemes.

[0062] Staticity and update difficulties: The learned index is essentially a static model built based on the data distribution at the training time, which makes it difficult to efficiently and cost-effectively adapt to the continuous data changes and distribution drift caused by insertion, update, deletion and compaction in LSM-Tree.

[0063] Retraining Overhead and the Balancing Dilemma: When the accuracy of a query index (LI) decreases due to data changes or its associated SSTable becomes invalid (e.g., due to compaction or deletion), retraining is necessary. The retraining process itself is computationally intensive and may require reading large amounts of data, introducing significant background overhead. Existing adaptive strategies (such as those based on fixed time windows or simple cost models) are insufficient in achieving the optimal balance between the benefits of query acceleration and the costs of retraining, especially failing to fully consider data popularity or the actual use value of the LI.

[0064] The tight coupling with LSM compaction presents a problem: building LIs directly onto volatile SSTable files (such as replacement index blocks) binds the LI's lifecycle to the SSTable. Frequent LSM compaction operations lead to frequent LI failures and rebuilds, significantly shortening the effective service time of LIs and limiting overall benefits.

[0065] (3) Insufficient utilization of data hotspot characteristics.

[0066] Lack of systematic hotspot management: Existing optimization solutions often consider hotspots in isolation at a single stage (such as caching or compaction), lacking a cross-level, collaborative, and dynamic hotspot awareness and management mechanism. Standard key-value separation and garbage collection treat cold and hot data equally, resulting in resource waste.

[0067] The cold and hot data processing strategy is too simplistic: the failure to adopt differentiated storage, indexing and caching strategies based on the access frequency or "temperature" (cold, warm, hot) of the data results in the inability to provide the optimal processing path for different types of data.

[0068] (4) Poor performance stability and predictability.

[0069] Background task interference: Background tasks such as compaction, GC, and LI retraining may consume a lot of system resources and compete with foreground read and write operations, causing drastic performance fluctuations or even short pauses, affecting the quality of service (QoS).

[0070] Large-scale deletion impacts (Read-After-Delete problem). LSM-Tree deletes by writing tombstone markers. Large-scale deletion operations generate a large number of tombstones, which continue to affect query performance (especially range queries) and consume storage space until they are completely cleaned up by compaction.

[0071] Therefore, the main technical problem solved by the embodiments of this application is how to systematically integrate learning indexes, dynamic hierarchies and adaptive management mechanisms into a KV-separated LSM-Tree to collaboratively optimize read and write performance, reduce various amplification and background task overheads, and efficiently utilize data hotspots to achieve high-performance and stable key-value storage.

[0072] Therefore, the embodiments of this application aim to optimize several core technical bottlenecks commonly found in existing LSM-Tree-based key-value storage systems and their optimization schemes, and make improvements in the following aspects:

[0073] High query latency and read amplification: The inherent multi-level lookup mechanism of LSM-Tree causes queries to need to access files at multiple levels. Even with caching and filters, read amplification remains severe, making it difficult to meet the needs of low-latency applications.

[0074] Write amplification and background overhead: Background compaction operations introduce significant data rewriting to maintain data order and reclaim space, leading to write amplification and consuming substantial I / O bandwidth and CPU resources. For systems employing key-value separation (KVSeparation), the garbage collection (GC) process also introduces significant background overhead and potential performance fluctuations.

[0075] Learning index integration challenges: When applying learning indexes to dynamically changing LSM-Trees, problems arise such as the contradiction between model staticity and data updates, the high cost of retraining and the difficulty in balancing it, and the frequent failures caused by the tight coupling between the LI lifecycle and LSM merging operations.

[0076] Inefficient use of data hotspots: The existing system lacks an effective mechanism for identifying and differentiating data access hotspots (frequently accessed data), resulting in low caching efficiency and failure to concentrate optimization resources on the most critical data.

[0077] Poor performance stability: Background maintenance tasks (Compaction, GC, LI retraining) may compete for resources with foreground operations, leading to drastic performance fluctuations and affecting service predictability and user experience.

[0078] After introducing the design concept of the embodiments of this application, the following is a brief introduction to the application scenarios to which the technical solutions of the embodiments of this application can be applied. It should be noted that the application scenarios described below are only for illustrating the embodiments of this application and are not intended to limit the scope. In specific implementation, the technical solutions provided by the embodiments of this application can be flexibly applied according to actual needs.

[0079] To further illustrate the technical solutions provided in the embodiments of this application, a detailed description is provided below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiments of this application provide method operation steps as shown in the following embodiments or drawings, the method may include more or fewer operation steps based on conventional or non-inventive methods. In steps where there is no logically necessary causal relationship, the execution order of these steps is not limited to the execution order provided in the embodiments of this application.

[0080] This invention proposes a multi-level, adaptively optimized key-value storage system based on LSM-Tree. This system is a novel storage architecture that integrates multiple optimization mechanisms. Figure 1 A schematic diagram of the core framework of a key-value indexing method provided in this application embodiment is shown below.Figure 1 As shown, the core architecture consists of the following key collaborative modules and mechanisms:

[0081] (1) Data Dynamic Tier Module (DDTM).

[0082] DDTM's core function is intelligently sensing and classifying data into hierarchical levels. It continuously monitors the access frequency and relevance of data keys or key ranges (key_range), and dynamically categorizes data into three logical levels—hot, warm, and cold—based on system policies and real-time status, enabling differentiated data processing. Ensuring that the most frequently accessed data utilizes the most efficient paths, while matching the most suitable storage and indexing mechanisms to data with different activity levels, is the foundation for the efficient operation of the entire system. The hierarchical classification criteria and capacity can be adaptively adjusted to cope with workload changes.

[0083] (2) Hierarchical storage and indexing module.

[0084] This module adopts a hierarchical design, matching specially optimized storage and indexing mechanisms for data with different activity levels to maximize overall system performance:

[0085] Hot Data Tier (HDT): This layer directly serves read and write requests for top-level hot data through a high-performance, fine-grained key-value (KV) memory cache. Its core function is to bypass the slower persistence path, providing near-memory speed access with extremely low latency, significantly reducing the response time for hot queries. The cache management strategy (such as replacement and write-back) is intelligently controlled by the system to achieve high hit rates and high efficiency.

[0086] Warm Data Tier (WDT): Utilizes a learned indexing model to provide efficient persistent indexes for data with moderate and relatively stable access frequency. Its core function lies in directly indexing the read-only archive partition of the Value Log generated during garbage collection, rather than the constantly changing SSTable files in the LSM tree. This design fundamentally solves the problem of tight coupling between the LI lifecycle and LSM compaction in existing LI integration schemes. Since the archive partition is read-only and relatively stable, its data will not change unless another GC occurs. This significantly extends the effective service time of the LI and substantially reduces LI failures and retraining overhead caused by frequent LSM compaction.

[0087] Cold Data Tier (CDT): Maintains a sizable and compact standard LSM tree, responsible for handling newly written data (before it becomes warm or is garbage collected) and truly infrequent cold data. Its core functions are serving as the entry point for new data writes, the final persistent storage of all data, and providing fallback query capabilities. Because it only manages cold data, the size of this LSM tree and the background compaction overhead can be effectively controlled.

[0088] This module employs customized storage and indexing strategies for data at different temperatures: high-speed caching for hot data, decoupled persistent learning indexes for warm data, and compact LSM trees for cold data. This optimizes resource allocation and significantly shortens query paths, forming the core foundation for the high performance and efficiency of this application's embodiments.

[0089] (3) Learning Index Management Module (LIMM)

[0090] This module integrates garbage collection with learning index maintenance. While performing Value Log garbage collection (reclaiming invalid space and organizing valid data), it leverages the ordered data stream generated during GC to efficiently trigger and assist in the construction or updating of the warm data layer learning index. This transforms the necessary background maintenance task of GC into a low-cost opportunity to generate or update the learning index, significantly reducing the overhead of introducing and maintaining high-performance learning indexes. It is a key innovative mechanism for achieving efficient indexing of the warm data layer.

[0091] (4) Global Adaptive Management Module (AMM)

[0092] To achieve system intelligence, self-adaptation, and self-optimization. As the central controller of the system, it continuously monitors the performance indicators, resource usage, and workload characteristics of the entire system, and makes global decisions and dynamic adjustments.

[0093] AMM is not a simple parameter tuner, but rather the central intelligent decision-making and control hub of a storage system. Its core vision is to endow the system with true self-adaptation, self-learning, and self-optimization capabilities, enabling it to continuously and dynamically adapt to changes in workload patterns, data distribution characteristics, and the underlying hardware environment without human intervention, maintaining the system in a pre-set, near-optimal operating state. Through a closed-loop process of "global monitoring -> deep analysis and prediction -> intelligent decision-making -> precise control," it coordinates the behavior of various components within the system (data dynamic hierarchical management module, hierarchical storage index module, learning index management module, etc.) to achieve a dynamic balance between multiple objectives such as performance, overhead, and stability.

[0094] AMM continuously and with low overhead collects performance metrics, resource usage, data characteristics, and workload patterns from all parts of the system. This provides a comprehensive, real-time, and accurate data foundation for subsequent analysis and decision-making. Examples of monitoring metrics are as follows:

[0095] Front-end interface: QPS, average latency / P99, error rate.

[0096] Hot data layer: hit rate, eviction rate, dirty data ratio, average dwell time, and capacity utilization.

[0097] Warm data layer: LI query hit rate, average LI query latency, model prediction error, Key List lookup time.

[0098] Cold data layer: size of each LSM level, compaction frequency / duration / IO volume, read / write amplification factor.

[0099] Value Log: Active vLog fill speed, number / size of archive partitions, and disk space utilization.

[0100] Learn about index management: GC trigger frequency, GC duration, effective data reclamation rate, and generated Key List size.

[0101] Background task executor: task queue length, average waiting / execution time of LI training tasks, CPU / IO utilization.

[0102] Dynamic data stratification: hotspot key / range distribution, stratification conversion frequency.

[0103] OS level: CPU load, memory usage, disk I / O queue, network bandwidth.

[0104] This application proposes a performance optimization technique for a key-value storage system based on an LSM-Tree architecture. It further involves the application of learning index and cache management techniques in the storage system. Primarily based on monitoring data and a built-in cost-benefit analysis model, it intelligently adjusts the operating strategies of each module, including dynamic data tiering thresholds, hot cache management methods, GC triggering timing and priority, and the selection and scheduling of LI training / updates. This ensures the system tends towards optimal operating conditions under various operating circumstances, maintaining high performance and high stability. It effectively solves the following key problems in existing technologies:

[0105] To address the issues of high query latency and read amplification: hot caching is used to directly serve hot data, while warm data uses learned indexes to accelerate lookups (bypassing most LSM layers), effectively reducing average query latency and suppressing read amplification.

[0106] To address write amplification and background overhead issues: By caching the hot data layer, most frequent updates to hot data are absorbed and merged, significantly reducing the effective write operation frequency injected into the LSM-Tree. This fundamentally slows down the accumulation and change rate of its internal files, minimizing the triggering frequency and resource consumption of background compaction. GC efficiency is optimized and LI training costs are reduced through GC-LI collaborative optimization. Unnecessary background tasks are avoided through adaptive management, comprehensively reducing write amplification and background resource consumption.

[0107] To address the challenges of learning index integration: by deploying LI on vLog partitions decoupled from compaction, the LI lifecycle is significantly extended and the retraining frequency is reduced; the LI data preparation challenge is solved through GC and LI collaboration; and the cost-benefit model is used to intelligently balance the overhead and benefits of LI.

[0108] To address the issue of low efficiency in utilizing data hotspots: dynamic stratification and dedicated optimization mechanisms at each stratum ensure efficient identification and targeted processing of data hotspots, improving cache hit rate and resource utilization.

[0109] To address the issue of poor performance stability: Decoupling design reduces conflicts between background tasks, and adaptive management smooths resource contention, significantly improving the system's performance stability and predictability.

[0110] This invention aims to provide a novel key-value storage solution that is high-performance, low-overhead, highly stable, and intelligently adaptable to workload changes through an innovative system architecture and collaborative optimization mechanism.

[0111] Figure 2 A diagram illustrating the overall architecture of a multi-level adaptive optimized LSM tree key-value storage system provided in this application embodiment is shown below. Figure 2 User requests enter the system through the front-end interface. The global adaptive management module plays the role of the core coordinator, responsible for monitoring the entire system, scheduling resources, and optimizing adaptive strategies to ensure that the system maintains optimal performance under different workloads and data characteristics.

[0112] The main data processing flow involves the following key modules:

[0113] Data Dynamic Hierarchical Management Module: This module performs real-time heat monitoring and intelligent classification of data, and manages the dynamic migration and placement strategies of data between different storage tiers based on preset or adaptively adjusted thresholds.

[0114] Tiered storage and indexing module: This is the core data access area, specifically including:

[0115] Hot data layer: Provides high-performance memory caching for the most frequently accessed data, achieving extremely low access latency.

[0116] Warm data layer: Utilizes decoupled learning indexes to efficiently manage and access warm data, which is typically stored persistently.

[0117] Cold data layer: Manages newly written data and infrequently accessed cold data through a scaled LSM tree structure.

[0118] Value storage module: As Value Log Storage, it is responsible for the persistent storage of actual data values, separated from the key, to optimize write performance and space utilization.

[0119] Learning Index Management Module: This module closely integrates the garbage collection process of the value log with the construction and updating process of the learning index, improving overall efficiency through data stream reuse and collaborative scheduling.

[0120] In addition, the background task execution module is responsible for asynchronously executing various system maintenance tasks, such as LSM tree compaction, VLog garbage collection scheduling, and index maintenance. These tasks are triggered and coordinated by modules such as global adaptive management or learning index management to ensure the performance and stability of the system's front-end services.

[0121] refer to Figure 3 This application provides a key-value indexing method, including the following steps:

[0122] S301: In response to an access operation to a specific key, determine the hotness information of the specific key.

[0123] S302: Determine the target data layer to which a specific key belongs based on heat information and set hierarchical parameters.

[0124] The target data layer includes a hot data layer, a warm data layer, or a cold data layer.

[0125] S303: Index based on the indexing strategy of the target data layer to determine the specific value corresponding to a specific key.

[0126] In this embodiment, firstly, upon receiving an access operation for a specific key, the popularity information of that key can be determined. This popularity information includes the latest access frequency and the latest access timestamp for a specific range to which the key belongs. Secondly, by integrating and setting hierarchical parameters, the target data layer to which the specific key belongs can be determined based on the popularity information, such as a hot data layer, a moderate data layer, and a cold data layer. Finally, indexing can be performed based on the target data's indexing strategy to determine the specific value corresponding to the specific key. This design offers relatively fast indexing speed and efficiency, thereby improving data storage performance.

[0127] The core responsibility of DDTM is to continuously sense data popularity and assign hierarchical labels (hot / warm / cold) to the data. This provides a basis for decision-making in other parts of the system. Figure 4 A schematic diagram illustrating the core functionality of a dynamic data layering module provided in this application embodiment is shown below. Figure 4 As shown, the core functionality of this module can be broken down into two main processes:

[0128] Popularity information update process: Respond to access operations and update the frequency and timestamp within the statistical range.

[0129] The process of determining the hierarchy of keys: Based on the latest popularity information, system parameters, and learning index status, the keys are labeled with hierarchical tags.

[0130] The following sections will explain these two main processes:

[0131] Regarding S301, a specific key can be represented by k. An access operation to a specific key can be a Get(k) read request or a Put(k,value) write / update request. In response to an access operation to a specific key, a popularity update process is automatically triggered, thereby obtaining the popularity information for that specific key. This popularity information includes the latest access frequency and the latest access timestamp for the specific range to which the specific key belongs.

[0132] Optionally, the process of determining heat information can be achieved through steps A1-A3:

[0133] A1: Determines the specific range to which a specific key belongs.

[0134] To reduce memory overhead, systems typically don't maintain a complete set of statistics for every individual key. Instead, they divide the entire key-value space into several consecutive key ranges. Therefore, it's possible to determine the specific range to which a particular key belongs from these ranges. Keys within the same range exhibit similar access characteristics.

[0135] A2: Update the value of the access frequency counter corresponding to a specific range to obtain the latest access frequency.

[0136] Optionally, find the frequency counter corresponding to the specific range and increment the value of this frequency counter by one (HeatStats.frequency[range]++). Each visit will increase the "heat score" of the corresponding range, thus obtaining the latest visit frequency, which can be used as an important indicator to measure activity.

[0137] A3: Update the access timestamp corresponding to a specific range to the current time to obtain the latest access timestamp.

[0138] For example, obtain the current system time and update the last access timestamp corresponding to that specific range to the current time (HeatStats.lastAccess[range] = Now()).

[0139] Once the popularity information is updated, this latest popularity information will be available for the "determine the hierarchy of keys" process and the global adaptive management module to make more accurate hierarchy division and optimization decisions. Based on this real-time updated popularity information, the "determine the hierarchy of keys" process will be further executed to support the dynamic allocation of the logical hierarchy to which a specific key belongs.

[0140] Regarding S302, once the heat information is updated, the target data layer to which a specific key belongs can be determined based on the heat information and the set hierarchical parameters. The target data layer here can be a hot data layer, a warm data layer, or a cold data layer.

[0141] Optionally, the process of determining the target data layer to which a specific key belongs can be implemented through steps B1-B2:

[0142] B1: Calculate the score for a specific key based on the latest access frequency, the latest access timestamp, and the set hierarchical parameters.

[0143] The hierarchical parameters AMM.tieringParams include a time decay factor, access frequency weight, and access relevance weight. Optionally, both the time decay factor and access frequency weight are pre-set. The access relevance weight indicates the importance of access frequency, and the score indicates the current overall popularity level of a specific key. Specifically, relevance is another key indicator for determining whether data is still "hot"; for example, recently accessed data is more likely to be accessed again in the short term.

[0144] Optionally, the system can call an internal function CalculateHeatScore, which takes a specific key k of the level to be judged, calculates a comprehensive data score heatScore, and uses this data score as the score of the specific key.

[0145] B2: Determine the target data layer to which a specific key belongs based on the score and the relationship between the pre-set hot layer threshold, warm layer threshold and cold layer threshold.

[0146] The process of determining the target data layer to which a specific key belongs typically includes the following scenarios:

[0147] Case 1: If the score is greater than the hot layer threshold, the target data layer to which the specific key belongs is determined to be the hot data layer.

[0148] Optionally, the calculated score `heatScore` is compared with the hot layer threshold `AMM.hotThreshold`. If the score is significantly greater than the hot layer threshold, it indicates that the specific key has high activity and should enjoy the most efficient processing path. In this case, the function immediately determines that the specific key belongs to the hot data layer and returns this result. Subsequent system operations (such as GET requests) should first attempt to find the specific key in the hot data layer.

[0149] Case 2: If the score is less than the hot layer threshold and greater than the warm layer threshold, and the learning index status indicates that a specific key has been managed by the learning index in the warm data layer, then the target data layer to which the specific key belongs is determined to be the warm data layer.

[0150] Optionally, if a specific key does not meet the hot layer criteria (i.e., its score is less than the hot layer threshold), it can be further evaluated whether the score is greater than the warm layer threshold. If so, it is necessary to query whether the specific key has already been covered by the learned index. If so, the specific key is determined to belong to the warm data layer. The function returns this result, indicating that the specific key has a certain level of activity and that the system has prepared an efficient persistent index for it. Furthermore, even if a key's access frequency reaches the warm data layer standard, if it is very new data that has not yet undergone GC and LI construction processes, it will not be immediately classified as warm data, but will be temporarily assigned to the cold data layer. This ensures that the LI of the warm data layer is built for relatively stable data.

[0151] Case 3: If neither of the above two cases is met, the target data layer to which the specific key belongs is determined to be the cold data layer.

[0152] For example, if a specific key neither meets the criteria for a hot data layer nor fully meets the criteria for a warm data layer, then that specific key will be classified as a cold data layer. The function returns this result. All newly written data before it is identified as a hot data layer or processed by the garbage collector and a new LI is created, as well as all long-term inactive old data, will be classified as cold data. The system primarily manages this data through a compact LSM-Tre.

[0153] In this way, the function will eventually return a clear hierarchical label (Tier.HOT, Tier.WARM, or Tier.COLD) for use by other parts of the system (such as query routing logic).

[0154] In summary, in this embodiment, the system continuously updates heat information, providing a dynamic data foundation for subsequent level determination. Using this latest heat information, system-set (adaptively adjustable) thresholds, and the status of whether a specific key has been covered by the warm data layer's learning index, the system intelligently assigns each specific key to an appropriate processing level, thereby providing a decision-making basis for subsequent level-specific optimizations (such as hot cache access, LI query, or LSM lookup).

[0155] Furthermore, in the above embodiments, the hot layer threshold, warm layer threshold, and cold layer threshold can be dynamically adjusted. Figure 5 A schematic diagram of a heuristic data hierarchical threshold decision logic flow provided in an embodiment of this application is shown below. Figure 5 It illustrates the initialization and parameter acquisition process, the thermal layer threshold adjustment decision, the temperature layer threshold adjustment decision, and the threshold boundary check and smoothing application process.

[0156] The hot zone threshold adjustment strategy includes the following cases:

[0157] Case C1: If the hit rate of the hot data layer is lower than the hit rate threshold and the capacity utilization is greater than the utilization threshold, then reduce the hot layer threshold.

[0158] This indicates insufficient hot cache performance; that is, the cache is almost full but the effect is poor, suggesting that the current hot caching threshold may be too high, preventing the cache from performing the required number of hot and cold cycles. In this case, the hot caching threshold can be lowered, for example, by subtracting an adjustment parameter from the original hot caching threshold to obtain a new hot caching threshold.

[0159] Case C2: If the capacity utilization of the hot data layer is lower than the utilization threshold or the eviction rate is greater than the eviction rate threshold, and the warm data layer has the ability to accept new index instances, then increase the hot layer threshold.

[0160] This indicates that hot cache resources are being wasted or can be optimized. When the capacity utilization of the hot data layer is too low, it means there is a lot of memory space. When the eviction rate is too high, it means there is frequent swapping in and out. However, the warm data layer still has the ability to accept new LIs, which may mean that the current warm layer threshold is too low. In the end, too much data that is not hot enough occupies the valuable cache. At this time, the hot layer threshold can be increased.

[0161] Case C3: If the average query latency of the cold data layer is greater than the maximum tolerable value, then reduce the threshold of the hot layer;

[0162] This indicates that the cold data layer is under too much pressure, meaning it is overloaded. If the hot data layer still has free capacity, it may be due to an improper hot data layer threshold setting, causing data that should have been cached to fall into the cold data layer. In this case, the hot data layer threshold can be lowered.

[0163] If the above significant conditions are not met, the thermal threshold can remain unchanged.

[0164] The adjustment strategies for the temperature spherical threshold include the following:

[0165] Case D1: If the actual average return on investment of the warm data layer learning index is lower than the return on investment threshold, or the average effective lifespan of the index model is less than the lifespan threshold, then increase the warm layer threshold.

[0166] This situation illustrates the inefficiency of warm data layer LI (Library Injection). In this case, if the actual average ROI of the warm data layer's learning index is lower than the ROI threshold, or if the average effective lifespan of the LI is too short, it indicates that the vLog partition it covers is garbage collected immediately after being built. This means that building LI for some data may be counterproductive. The decision in this situation is to increase the warm layer threshold and reduce the amount of data entering the warm data layer and triggering LI construction.

[0167] Case D2: If the number of new cold data in the cold data layer that are waiting for garbage collection and index model processing and whose access popularity is higher than the popularity threshold is greater than the quantity threshold, and the warm data layer has the ability to accept and manage more index instances, then the warm layer threshold is lowered.

[0168] This situation represents a backlog of high-value warm data candidates. If a large amount of new cold data with high access frequency is backed up in the cold data layer awaiting GC and LI processing, and the warm data layer currently has the capacity to accept and manage more LI instances, it indicates that the current warm layer threshold may be too high, hindering the conversion of valuable data to the warm data layer. In this case, the decision is to lower the warm layer threshold.

[0169] If the conditions for significant adjustment mentioned above are not met, the temperature threshold will remain unchanged for the time being.

[0170] In addition, the calculated thresholds for the new and old thermospheres need to undergo boundary checks to ensure they are within a reasonable minimum / maximum range. The boundary check process is as follows:

[0171] Typically, it's necessary to ensure that the threshold for the new temperature layer is always lower than the threshold for the new hot layer (and maintains a certain interval) to uphold the logical correctness of the tier definition. If the calculated new threshold differs from the current threshold, AMM will apply the new threshold to the dynamic tier management module through a smooth update mechanism (DDTM.UpdateThresholdsSmoothly). Smooth updates are used to avoid performance fluctuations caused by drastic threshold changes. If the old and new thresholds are the same, no update is performed in this tuning cycle. This completes one dynamic tier threshold adjustment process, and AMM will repeat this process periodically.

[0172] By dynamically adjusting data stratification thresholds, the system ensures that the data distribution and capacity allocation across the hot, warm, and cold strata match the current workload in real time, thereby achieving optimal overall system performance. This adjustment process is based on continuous monitoring of key indicators at each system stratum, such as hit rate, average access latency, actual capacity utilization, and the distribution of data hotspots. The AMM comprehensively analyzes this monitoring data to determine whether the current stratification strategy deviates from the ideal state.

[0173] Based on the above analysis, AMM uses a set of decision-making logic (initially based on expert rules, and potentially evolving into a more advanced machine learning model in the future) to calculate and adjust the hot and warm layer thresholds. For example, when the actual capacity of the hot data layer is lower than the ideal value and the hit rate is not up to standard, AMM may appropriately lower the hot layer threshold to guide more hot data into the cache to improve the hit rate; conversely, if the hot data layer is overloaded or the eviction rate is too high, the threshold will be increased. Similarly, for the warm data layer, if it covers insufficient data and there is a large amount of high-value new data waiting to be processed, or if the ROI or lifespan of its learning index is too low, AMM will adjust the warm layer threshold accordingly to optimize the efficiency of learning index construction and resource allocation.

[0174] Finally, the AMM will send the calculated new threshold to the dynamic hierarchical management module, which will then perform subsequent data hierarchical determinations based on the updated threshold. Through this closed-loop "monitoring-analysis-decision-control" mechanism, the system can continuously and adaptively optimize its data hierarchical strategy, ensuring that data is placed in the most appropriate processing level under various operating conditions, thereby maximizing overall performance and resource utilization efficiency.

[0175] Regarding S303, after determining the target data layer, indexing can be performed based on the indexing strategy of the target data layer to determine the specific value corresponding to a specific key.

[0176] This process can be implemented using a hierarchical storage index module, which is the core data access area. It is not a single storage structure, but rather a dynamic data hierarchy (hot, warm, cold) based on the data layering module. Data is guided to its respective optimized sub-layer for storage and indexing to achieve optimal performance and efficiency. Optionally, the process of determining a specific value corresponding to a specific key can be categorized into the following cases:

[0177] Case 1: If the target data layer is a hot data layer, then the specific value corresponding to a specific key is determined based on the mapping relationship of key-value pairs represented by the hot data layer in the MAP structure.

[0178] In this case, which is the case of KV high-speed caching, the hot data layer is the topmost and fastest storage layer in the system. The hot data layer is specifically responsible for directly storing and serving the keys and complete values ​​of data that are determined to be "hot" in the layer. It adopts a fine-grained key-value caching mechanism in memory.

[0179] To support high concurrency access, a thread-safe sharded hash table structure is adopted, and each shard can be configured with an efficient replacement strategy (such as LRU, LFU, or an adaptive strategy dynamically specified by AMM). For update operations, dirty bit marking and reliable write-back logic (if configured in write-back mode) must be implemented to ensure that data can be safely persisted to the lower layer (usually the LSM-Tree of the cold data layer) before being evicted.

[0180] Its core function is to bypass all slow persistent storage accesses and complex index lookup paths, providing an extremely low-latency (typically in the microsecond or even nanosecond) read / write experience for the most frequently accessed data. Its capacity, admission policy (when to promote data from the lower layer to the cache), replacement policy, and write policy (write-through or write-back) can all be dynamically adjusted by a globally adaptive management module to maximize cache hit rate and effectively manage memory resources. When cached data is evicted and is dirty, it interacts with the write interface of the cold data layer.

[0181] Case 2: If the target data layer is a warm data layer, then the specific key is input into the pre-trained index model so that the index model indexes the location of the key stored in the first-value log read-only archive partition and determines that the value stored at the location is a specific value; wherein, the first-value log read-only archive partition is generated by the garbage collection process.

[0182] This scenario refers to the learning indexing of the vLog. The warm data layer provides efficient persistent indexing services for data classified as "warm". Its innovation lies in its use of a learning indexing model to directly index the positions (i.e., vLog offsets) of keys generated during the garbage collection process and stored in the read-only archive partition of the Value Log.

[0183] The system maintains a metadata structure (e.g., a key-range-based B+ tree or range hash table) to manage and locate multiple LI instances. Each LI instance contains a loaded machine learning model (or a reference to its persistent representation), a handle to the corresponding ordered Key List file, and associated Value Log archive partition information. During a query, the appropriate LI instance is quickly located using the metadata structure. The LI model is then used to predict the key's position in the Key List, and an optimized search (e.g., Model Biased Search) is performed within a small predicted range to obtain the accurate vLog offset.

[0184] Traditional indexing (LI) integration schemes, such as TridentKV, often build the learned index within the SSTable file of the LSM tree as an enhanced block index. However, the compaction operation of the LSM tree frequently rewrites and merges the SSTable file, causing the internal LI model to frequently fail and requiring costly retraining, thus limiting the overall benefits of LI. The decoupling scheme proposed in this invention perfectly avoids this problem by shifting the indexing target of LI to the longer-lived, read-only Value Log archive partition. This allows LI to provide efficient query services stably and consistently over the long term, significantly improving its return on investment.

[0185] The core function of the warm data layer is to completely decouple the learning index from the LSM compaction process. By using a relatively stable and long-lived Value Log archive partition, the effective service time of the LI is significantly extended, and the need for LI retraining due to LSM compaction is significantly reduced. This provides query speeds far faster than traditional multi-level LSM lookups for data with moderate access frequency, while ensuring data persistence. The creation and destruction of LI instances are driven by the GC-driven learning index module based on GC progress and AMM decisions. The LI training task is also submitted to the background task executor by the AMM based on its cost-benefit assessment and priority, and it also needs to provide interfaces (such as IsIndexed(k)) for other modules to query whether a key or range has been indexed.

[0186] Case 3: If the target data layer is a cold data layer, then based on the LSM tree structure of the cold data layer, determine the location of a specific key and determine the value stored at that location to be a specific value; where each node of the LSM tree structure stores a key-value pair pointing to the second value log, the second value log refers to the active log and the value log in the archive partition not covered by the warm data layer, and the key-value pair of the second value log is a key-location pair.

[0187] This scenario represents a compact LSM-Tree. The cold data layer handles data deemed "cold" and all newly written data that hasn't yet been explicitly stratified. It employs a standard LSM tree structure, but its size and number of layers are expected to be effectively controlled, storing pointers to the Value Log (including active logs and archive partitions that may not yet be covered by the warm data layer).<Key,vLog_Offset> Yes. It can be implemented based on existing mature LSM libraries (such as simplified or customized versions of RocksDB). Its configuration parameters, such as Memtable size, level factor, and threshold for triggering compaction, should be designed to be dynamically adjusted by the global adaptive management module (AMM) to maintain its compact structure and low backend maintenance overhead.

[0188] The cold data layer is the primary entry point for new data (referring to the key and offset portions; values ​​are directly entered into the Value Log). It also serves as the final persistent storage guarantee for all data and provides a fallback query capability for cold data and data that has not yet been optimized by the upper layers. Because it primarily manages cold data, its active data volume is expected to be much smaller than that of a traditional LSM tree, thus effectively controlling the frequency and overhead of its background compaction. It also receives dirty data written back from the hot data layer. Its compaction strategy and triggering timing are also influenced by AMM instructions (for example, the AMM may instruct it to more aggressively merge stabilizing data into the lower layers so that this data can enter the GC-driven learning index module's processing scope more quickly, and thus may be covered by the warm data layer index more rapidly).

[0189] Optional, Figure 6 The diagram illustrates the hierarchical storage and indexing data flow in the embodiments of this application, following the core design principles and processes:

[0190] 1. Write to the priority cold data layer:

[0191] All new write (Put) values ​​are first appended to the active Value Log, while their keys and vLog offsets are primarily entered into the Memtable of the compact LSM tree (CNLT) in the cold data layer. This is to ensure fast data persistence and consistency.

[0192] 2. Optimize reading layer by layer:

[0193] The read (Get operation) will attempt to find data in each layer according to the priority order of "hot -> warm -> cold", and return once a match is found, so as to make use of the most efficient path.

[0194] 3. Popularity drives tier upgrades:

[0195] DDTM continuously monitors data popularity. When the access frequency and relevance of data in the cold or warm data layer reach a certain threshold, it will be promoted to a more popular layer (e.g., from CDT / WDT to HDT).

[0196] 4. Cooling and Learning Index Management:

[0197] Data in the HDT will be evicted according to the replacement policy if it cools down (dirty data is written back to the CDT).

[0198] Data in CDT (especially newly written data) will be consolidated into the Value Log archive partition over time and with the progress of GC, and may be overwritten by WDT's LI, thus transforming from "cold" to "warm".

[0199] The Value Log archive partition covered by WDT will eventually be reclaimed by an older GC process, and its LI will become invalid, with the data completely "cooled down" or disappearing.

[0200] AMM Global Control: The thresholds, strategies, and priorities of the entire data flow and hierarchical transformation are dynamically controlled by the global adaptive management module (AMM).

[0201] Optionally, the process of determining the index model in the above embodiments is explained through steps E1-E3:

[0202] E1: In response to the garbage collection process, scan the log of values ​​to be reclaimed in the active log to determine valid key-value pairs.

[0203] When GC is triggered (determined by AMM based on space, load, or expected LI gains), the algorithm first ensures that new writes are switched to the new active log. Then, it scans the Value Log (vLog_old_info) awaiting reclamation and identifies still valid key-value pairs by querying the current authority state of the system (including cold, warm, and hot data layers). These valid key-value pairs are collected and sorted by key.

[0204] E2: Writes valid key-value pairs sequentially to a new value log archive partition, generating a file containing the key-value pairs and recording the key range covered by the new value log archive partition. The file containing key-value pairs is a file containing key-position pairs.

[0205] The sorted valid key-value pairs are sequentially written to a new, read-only Value Log archive partition (vLog_archive). Meanwhile, a...<Key,new_vLog_offset> An ordered key list file (key_list_file) is also generated. This key list file is the foundation for subsequent index training. The key range (key_range) covered by this new archive partition is also recorded.

[0206] E3: Create an indexing task that trains the indexing model based on the metadata of the new value log archive partition and the current popularity of the key range.

[0207] The Learning Index Management (LI) module is a collaborative mechanism deeply integrated into the Value Log's garbage collection process. Traditional GC only focuses on reclaiming invalid space and migrating valid data. This module, however, builds upon this by intelligently reusing the data processing results generated during GC (i.e., sorted valid key-value pairs), using them as a high-quality input source for the warm data layer's learning index. Based on the decisions of the global adaptive management module, it triggers and assists in the construction or incremental updates of the LI. In short, it cleverly transforms the necessary background maintenance overhead of GC into an opportunity to improve the performance of warm data queries.

[0208] Optionally, the complete algorithm flow for this process is shown below:

[0209]

[0210] Referring to the algorithm described above, the metadata of the newly generated archive partition (key_range, key_list_file.size()) and the current heat (range_heat) of this key range obtained from the dynamic hierarchical management module are submitted to the global AMM. The AMM internally performs complex cost-benefit analysis (such as the logic of ShouldTrainLI in Algorithm 2, considering LI lifespan, contribution to overall query performance, etc.) to decide whether it is worthwhile to build / update the learning index for this new archive partition, and what method to use (full training or incremental update), and determines the task priority.

[0211] LI Task Submission: If the AMM's decision is to execute the LI task (task_info.should_execute is true), the algorithm will create a specific LI task (li_task) based on the decision information (whether it's incremental update or full training, the target LI instance, etc.). This task will then be submitted to the background task executor (BackgroundTaskExecutor) along with the priority calculated by the AMM, so that the background executor can reasonably allocate resources for asynchronous LI model training.

[0212] Resource cleanup and subsequent optimization: At appropriate times (e.g., after ensuring the new LI has been built and activated, or according to the AMM's strategy), the module will schedule the cleanup of old, no longer needed resources, such as old Value Log partitions, their associated Key List files, and old LI instances that may be replaced by the new LI. Simultaneously, it may notify the cold data layer that data for certain key ranges is now being efficiently managed by the new LI in the warm data layer, allowing the cold data layer to perform internal optimizations (e.g., more aggressive compaction or cleanup of redundant entries for these ranges).

[0213] Through the algorithm described above, the GC-driven learning index module transforms GC, a background task primarily focused on space reclamation, into a crucial enabling process that proactively builds and maintains the learning index for the warm data layer. By combining this with the intelligent decision-making of the AMM, it ensures that the construction of the LI is valuable and cost-effective, thus making a significant contribution to the overall system performance and efficiency.

[0214] Furthermore, in the cost-benefit analysis and intelligent scheduling of LI training (AMM regulating the GC-driven learning index module), the primary goal of AMM is to ensure that the construction and maintenance of LIs bring tangible net benefits and avoid unnecessary waste of background resources. To this end, AMM relies on comprehensive monitoring inputs, including metadata (key range, size) of newly generated Value Log archive partitions, historical or predicted access frequencies from the dynamic data tiering management module, coverage and performance records of existing LIs in the warm data layer, and query performance data from the cold data layer. After completing GC and generating candidate vLog archive partitions, the GC-driven learning index module is only responsible for reporting relevant information to AMM; the final decision-making power regarding whether to train an LI for that partition, when to train it, and how to train it rests entirely with AMM.

[0215] The core of AMM's decision-making lies in its internally maintained and continuously optimized cost-benefit analysis model. This model relies on a series of key parameters, such as the estimated latency savings from LI queries, the expected effective lifespan of LI (this is no longer a simple prediction, but rather considers factors such as current LSM compaction pressure, disk space availability, and future GC scheduling), LI training time (which can be predicted based on historical training data and model complexity), and related resource cost factors and the minimum acceptable ROI threshold. Crucially, these model parameters are not static but can be self-learned and adjusted based on the actual effects of historical optimizations and the current operating state of the system, allowing AMM's decision-making capabilities to continuously evolve and become more accurate.

[0216] Furthermore, the decision-making model of AMM is not static. After each execution of the LI training task, AMM monitors and collects actual performance feedback, such as the actual query latency savings brought by LI, the actual time and resource consumption of GC and LI training, and the actual effective lifespan of the LI. AMM's internal self-learning mechanism (e.g., through simple heuristic algorithms or future more complex machine learning models) dynamically adjusts the weight factors and internal parameters in the cost-benefit model based on this feedback data, such as `amm_config.cpu_cost_per_unit_time` or `amm_config.min_acceptable_roi`. This allows AMM's decision-making capabilities to continuously evolve, becoming increasingly accurate and intelligent, ensuring that the maintenance of the learned index always serves to improve the overall system performance and maximize the utilization efficiency of background resources.

[0217] After making a decision, the AMM assigns dynamic priorities to LI training or update tasks with positive ROI (generally, the higher the ROI, the higher the priority), and sends the task along with priority information and the currently available background resources (such as CPU cores and I / O quotas) to the background task executor for scheduling. In addition, the AMM queries the warm data layer to intelligently determine whether to perform a low-cost incremental update (if existing LIs support it and have similar coverage) or perform full training to further optimize resource utilization. Through this series of sophisticated analysis, decision-making, and scheduling mechanisms, the AMM ensures that the maintenance of the learning index always serves to improve the overall system performance and maximizes the efficiency of background resource utilization.

[0218] Figure 7 This is a schematic diagram of an AMM learning index task decision-making and scheduling process provided in an embodiment of this application. The specific algorithm is shown in Algorithm 2 below.

[0219]

[0220]

[0221]

[0222] The core objective of Algorithm 2 is that, for a newly generated Value Log archive partition created by the garbage collection process, the global adaptive management module uses precise quantitative analysis to weigh the potential benefits and costs of building or updating the learning index. The ultimate goal is to intelligently decide whether to execute the LI build / update task and assign it a reasonable execution priority, thereby ensuring the effective utilization of background resources and truly improving the overall system performance.

[0223] The algorithm steps are described below:

[0224] (1) Profit estimation

[0225] The core task at this stage is to quantify the expected total performance gains from building a learning index for candidate partitions. This requires accurately assessing the time saved by using LI compared to not using LI (i.e., directly querying the LSM layer), and considering the number of times the data may be accessed in the future as well as the effective service duration of LI itself.

[0226] Obtain performance benchmarks and estimated LI performance:

[0227] First, the algorithm obtains the average latency for querying the current range of candidate partition keys from the cold data LSM layer. This represents the performance baseline (avg_latency_without_li) without additional LI optimization. Next, based on the data characteristics of the candidate partition (such as data volume) and the target model complexity set by the AMM, the algorithm estimates the expected average latency (avg_latency_with_li_expected) for querying the same range after building a learned index for that partition.

[0228] If the estimated query latency after using LI is not better than (i.e., greater than or equal to) the query latency of the current LSM layer, then building this LI obviously will not bring any performance savings. In this case, the algorithm will directly determine not to execute the LI task and record the reason as "LI does not save latency", thus avoiding subsequent invalid cost calculations.

[0229] Calculate unit query savings and predict future visits:

[0230] If it is initially determined that LI (Latency Saving) can bring latency savings, the precise time saved per query due to using LI is calculated (latency_saving_per_query). This forms the basis for subsequent calculations of the total benefit. Then, the algorithm needs to predict the total access frequency of the key range within the future effective lifecycle of LI. This typically relies on historical access frequency information provided by the dynamic data tiering management module and may combine it with the frequency prediction model (PredictAccessFrequency) within the AMM (Automatic Data Management Model) to more accurately predict future access trends.

[0231] Simultaneously, the algorithm also needs to predict the effective lifespan of the learned index (predicted_li_lifespan). The lifespan of an LI is not permanent; it is affected by various factors such as earlier future GC processes (which may reclaim the Value Log partition of its index) and large-scale data updates causing model failure. The AMM's prediction model comprehensively considers factors such as current GC pressure and data stability.

[0232] Calculate the total expected delayed revenue:

[0233] Finally, multiplying the latency saved per unit query by the total expected number of queries for that LI over its effective lifetime yields the total expected latency benefit (total_latency_benefit). This value is the core basis for subsequent comparisons with costs.

[0234] (2) Cost estimation

[0235] The goal of this stage is to fully quantify the resource costs required to build or update the learning index, including computing resources (CPU, I / O) and storage resources.

[0236] Model complexity determination and training cost estimation:

[0237] First, based on the amount of data in the candidate partitions and the target model complexity set by the AMM (for example, a simpler model may be used for partitions with a large amount of data to control training time), determine the specific complexity (model_complexity_actual) of the LI model to be built.

[0238] Based on the model complexity and the training time records of similar historical LI models obtained from the Warm Data Layer (WDT), the algorithm estimates the training time required for this training (training_time). The estimated training time is then multiplied by the unit CPU cost factor and unit I / O cost factor defined in the AMM configuration to obtain the CPU cost (training_cpu_cost) and I / O cost (training_io_cost) during the training process.

[0239] Storage cost estimates:

[0240] The algorithm estimates the storage size (li_storage_size) required for the LI model itself and its associated metadata (such as the Key List file). Then, it multiplies this storage size by the storage cost factor per unit time and unit space defined in the AMM configuration, and then by the previously predicted effective lifespan of the LI, thereby calculating the total storage cost (storage_cost_per_lifespan) occupied by the LI throughout its entire lifespan.

[0241] Incremental decision updates and cost adjustments (key optimization):

[0242] To further reduce maintenance costs, the algorithm queries the state information of the warm data layer to determine whether a learned index instance already exists for the candidate key range, and whether this instance supports an efficient incremental update mechanism.

[0243] If such an LI exists and supports incremental updates, the algorithm estimates the cost of performing the incremental update (incremental_update_cost). If this incremental update cost is significantly lower than the cost of full retraining, the system will prioritize the incremental update. In this case, the actual training cost (actual_training_cost) will be updated to the incremental update cost, the task will be marked as an incremental update, and the target LI instance to be updated will be recorded. If no LI can be incrementally updated, or the cost of incremental updates is not more advantageous, the actual training cost will remain the cost of full training.

[0244] Calculate the total task cost:

[0245] Finally, the adjusted actual training cost is added to the total storage cost over the lifetime to obtain the total expected cost (total_task_cost) for executing this LI build / update task.

[0246] (3) ROI Calculation and Decision Execution

[0247] This stage is the final decision-making process. Based on the total revenue and total cost calculated earlier, a return on investment (ROI) analysis is used to determine whether to execute the LI task and to assign it a reasonable priority.

[0248] Cost-effectiveness verification:

[0249] First, a simple validity check is performed. If the calculated total task cost is less than or equal to zero (which should not happen under normal circumstances and is considered abnormal), then the LI task is directly determined not to be executed, and the reason is recorded as "invalid task cost".

[0250] Calculate Return on Investment (ROI):

[0251] Dividing the total expected latency benefit calculated in the first phase by the total task cost calculated in the second phase yields the return on investment (ROI) for the LI task. A higher ROI means a greater performance gain per unit of cost investment.

[0252] Final decision and prioritization:

[0253] The AMM compares the calculated ROI with the minimum acceptable ROI threshold set in the system configuration (amm_config.min_acceptable_roi). Simultaneously, to avoid allocating resources to tasks with high ROI but negligible absolute benefits, it also checks whether the total expected latency benefit exceeds a minimum absolute benefit threshold (MIN_ABSOLUTE_BENEFIT). Only when both ROI and absolute benefit meet the preset criteria will the AMM make the final decision to execute the LI task (li_task_decision.should_execute = true).

[0254] Once the decision to execute is made, AMM will also calculate a dynamic priority (li_task_decision.priority) for the LI task based on the calculated ROI (and may also take into account factors such as the current system background task queue's busyness and available resources).

[0255] Finally, a decision result (li_task_decision) containing information such as whether to execute, task type (full / incremental), and priority is returned. If the decision is to execute, the task will be submitted to a background task executor for asynchronous processing. If the ROI or absolute return does not meet the target, the decision is not to execute the LI task.

[0256] (c) GC triggering and scheduling optimization (AMM controlling LIMM)

[0257] The Adaptive Management Module (AMM) controls garbage collection (GC), which is one of the core components of this invention for achieving system intelligence and performance optimization. It changes the traditional GC model, which passively triggers GC based solely on space thresholds, and introduces a dynamic, goal-driven "GC demand" scoring and decision-making mechanism. Its core objective is not only to promptly reclaim disk space occupied by invalid data, but more importantly, to proactively create opportunities for building high-quality learning indexes (LI) for warm data layers, while minimizing the impact of the GC process itself on the system's front-end services, thus achieving a balance among multiple objectives.

[0258] To achieve this goal, AMM continuously monitors and comprehensively evaluates information from three key dimensions to quantify the current "demand" for performing garbage collection:

[0259] 1. Space Pressure (Score_Space): Assess the current scarcity of available disk space. The less available space, the higher the score, indicating a higher priority for space reclamation. This is the fundamental driver of Garbage Collection (GC), ensuring the healthy operation of the system's storage.

[0260] 2. LI Construction Opportunity (Score_LI_Opportunity): This assesses how much high-value (i.e., frequently accessed but not yet covered by LI) cold data is currently backed up in the Value Log partition awaiting garbage collection (GC). If this data can be cleaned up by GC and subsequently used to build LIs, query performance will be significantly improved. The more abundant and frequently accessed the high-value data awaiting indexing, the higher this score, representing a greater value in creating performance improvement opportunities through GC. This is a key aspect of the GC-LI collaboration in this invention, enabling GC to serve proactive performance optimization.

[0261] 3. System Load (Score_System_Load): This assesses the overall system workload, particularly the background CPU and disk I / O usage. A higher score (as a penalty) indicates a greater potential negative impact of garbage collection (GC) on foreground performance.

[0262] AMM uses a configurable weighted formula (Overall_GC_Need = w1*Score_Space + w2*Score_LI_Opportunity - w3*Score_System_Load) to merge the scores of these three dimensions into an overall GC need (Overall_GC_Need). The weighting factors w1, w2, and w3 can be adaptively adjusted based on the long-term performance of the system to optimize the decision model.

[0263] Based on this quantified overall GC demand, Figure 8 The diagram below illustrates the GC triggering and scheduling optimization process of the storage system provided in this embodiment of the application. Specific details are as follows:

[0264] Intelligent Triggering Decision: The AMM periodically or event-drivenly calculates Overall_GC_Need. When this score exceeds a dynamically adjusted trigger threshold, the AMM issues a GC instruction to the learning index management module. This threshold itself can also be adjusted based on system conditions (such as lowering the threshold when disk space is extremely scarce).

[0265] Targeted GC Scope Recommendations: When GC is triggered, AMM does more than just initiate a general garbage collection process. Based on the analysis results of Score_LI_Opportunity, it can recommend to the LIMM to prioritize GC of Value Log partitions containing the most "high-value indexable" data. This transforms GC from blindly "starting from the oldest" to a strategic "value-driven" approach, thereby maximizing the opportunities and efficiency of subsequent LI construction.

[0266] Dynamic Priority Setting and Resource Scheduling: AMM assigns a dynamic priority to triggered GC tasks (and subsequent potential LI training tasks). This priority considers factors such as the strength of Overall_GC_Need, the size of Score_LI_Opportunity, and the current system load. The background task executor schedules GC tasks based on this priority, ensuring that high-priority GC (such as GC that resolves severe space pressure or generates extremely high-value LI) is processed first, while low-priority GC is performed when the system is relatively idle.

[0267] Continuous learning and model optimization (advanced features): AMM can monitor the actual effects of GC and LI construction (such as space reclamation rate, performance improvement brought by LI, and the impact of GC on the front end), and based on this feedback data, adaptively adjust the weight factors and internal parameters in the "GC demand" scoring model through machine learning or heuristic algorithms. This allows AMM's GC decision-making ability to continuously evolve, becoming increasingly accurate and intelligent.

[0268] To make the technical solution of this application easier to understand, a specific example will be used below:

[0269] Assuming the system is running, the global adaptive management module provides the current operating parameters: hotThreshold is 100.0, warmThreshold is 50.0, the tiering parameters (tieringParams) are set including access frequency weight W_freq is 1.0, access recency weight W_recency is 15.0, and time decay factor decay_factor is 0.005 (decays per second).

[0270] (1) Storage procedure for monitoring data:

[0271] For statistics on key ranges, design an in-memory data structure to store this statistical information:

[0272] Sharded Hash Map: To support high concurrency, a thread-safe sharded hash map is used. Key: The key of the hash table is range_id. For simplicity, assume the key space is divided into groups of 1000, so range_id = floor(key / 1000).

[0273]

[0274] heatScore = 4.12 + 15.0 = 19.12

[0275] Decision: 19.12 is still less than warmThreshold(50.0). During these accesses, k=8888 was consistently classified as Tier.COLD.

[0276] c. Third stage: GC and LI construction, completing the advancement to "warm data".

[0277] 1. Data Maturity: Suppose that after some time, the old Value Log containing data with range_id=8 is garbage collected (GC). The GC process organizes this data into a new, read-only vLog archive partition.

[0278] 2. AMM Decision: After GC is completed, the Learning Index Management Module (LIMM) reports the information of this new archive partition (including the range_id=8 it covers) to AMM 4.

[0279] The AMM query retrieved the current popularity of range_id=8 from DDTM and found that its frequency was very high (assuming it had accumulated to hundreds of times). The calculated popularity score had exceeded the temperature threshold of 50.0.

[0280] AMM conducted a cost-benefit analysis and determined that it was worthwhile to build an LI for this high-popularity range, so it approved and created a background training task.

[0281] 3. LI construction complete: The background task executor has completed the training of the LI model and notified the warm data layer that range_id=8 has now been covered by LI.

[0282] 4. A decisive visit:

[0283] The system received another Get(k=8888) request.

[0284] Determination level:

[0285] At this point, the frequency is very high, assuming the calculated heatScore is 58.0.

[0286] 58.0<hotThreshold(100.0)-> This is not hot data.

[0287] 58.0>warmThreshold(50.0) -> Satisfies the warm data heat condition.

[0288] The system checks the second condition: "Has k=8888 been covered by the learned index?". The warm data layer returns yes.

[0289] Final decision: Both conditions are met, and k = 8888 is ultimately determined to be Tier.WARM.

[0290] Operation performed: The Get(8888) request is routed to the warm data layer, where the query is completed through an efficient learning index.

[0291] In summary, the embodiments of this application have the following beneficial effects:

[0292] (1) Dynamically adaptive three-layer heterogeneous key-value storage architecture and its data management.

[0293] The architecture comprises a fine-grained key-value in-memory cache for top-level hot data (hot data layer), a read-only archive partition of the Value Log that directly indexes using learned indexes (warm data layer), and a scalable compact LSM tree (cold data layer). Data is dynamically assigned and transferred between these layers based on real-time monitored access frequency, under the control of a globally adaptive management module.

[0294] Advantages compared to existing technologies:

[0295] Effectively improve query performance: By matching the optimal storage and access path for data at different "temperatures" (direct caching of hot data, efficient LI for warm data, and centralized LSM for cold data), the average query latency is significantly reduced, effectively suppressing the read amplification problem of traditional LSM trees.

[0296] Optimize resource allocation: Concentrate limited cache and computing resources (used for learning indexes) to serve the most active data, thereby improving overall resource utilization efficiency.

[0297] Enhanced system scalability: The controlled size of the LSM tree in the cold data layer reduces its compaction overhead, which is beneficial to the overall stability and scalability of the system.

[0298] Differences from existing technologies:

[0299] Unlike simple LSM enhancements or single caching / learning index optimizations, this invention proposes a novel, dynamically collaborative three-layer heterogeneous architecture that achieves more thorough separation of concerns and performance optimization. In particular, the decoupling of the warm data layer learning index from LSM compaction is a key innovation.

[0300] (2) Deeply integrate Value Log garbage collection with learning index construction

[0301] In the garbage collection process of the Value Log, this invention transforms a background task that originally only focused on space reclamation into a crucial enabling process that actively builds and maintains the learning index for the warm data layer. Specifically, it utilizes the sorted, valid key-value pair data stream generated during GC as the primary input source for the warm data layer's learning index. The global adaptive management module intelligently decides whether and how (full or incremental) to build or update the learning index for the newly generated Value Log archive partition based on a cost-benefit analysis, and asynchronously submits the task to the background for execution.

[0302] Advantages compared to existing technologies:

[0303] Significantly reduces the cost of learning index maintenance: The GC process naturally completes the validity screening and sorting of data, greatly reducing the data preparation overhead required to build a high-quality learning index, thus significantly reducing the cost of introducing and maintaining a high-performance learning index.

[0304] Increase the added value of the GC process: Transform the GC task, which originally focused primarily on space reclamation, into a process that simultaneously creates conditions for performance optimization (building a learning index), thereby improving the overall efficiency of background tasks.

[0305] Facilitate timely updates of the learning index: As the data evolves and GC proceeds, the LI can be updated or rebuilt periodically and at low cost, maintaining its adaptability to the current data distribution.

[0306] Differences from existing technologies:

[0307] Existing technologies typically treat GC and LI maintenance as independent processes. Even with initial integration, LI construction often relies on volatile SSTable files within the LSM, and its lifecycle is affected by compaction operations. This invention achieves synergistic efficiency by deeply integrating the GC and LI maintenance processes and introducing intelligent decision-making. This original "turning waste into treasure" concept cleverly transforms the necessary background overhead of GC into an opportunity to improve warm data query performance, which is the essential difference between this invention and existing technologies.

[0308] (3) Adaptive storage system management based on global monitoring and cost-benefit analysis.

[0309] A global adaptive management module is established to continuously monitor the performance metrics, resource usage, and workload characteristics of each system component. Based on this real-time data and a built-in (self-learning and adjustable) cost-benefit analysis model, AMM can dynamically and collaboratively adjust the system's key operational strategies, including: dynamic data stratification thresholds, hot data caching management methods (such as size, replacement strategy, and write strategy), GC triggering timing and priority, and the selection, priority, and resource allocation for learning index training / update.

[0310] Advantages compared to existing technologies:

[0311] Achieve system-level intelligence and self-optimization: enable the system to proactively adapt to dynamically changing workloads and data distributions without human intervention, continuously maintaining the system in a near-optimal operating state.

[0312] Balancing multiple optimization objectives: It can intelligently weigh and make decisions among multiple possible optimization objectives based on preset strategies (such as prioritizing latency, prioritizing throughput, or balancing resource consumption).

[0313] Improve performance stability and predictability: By intelligently scheduling background tasks (GC, Compaction, LI training) and dynamically adjusting resource allocation, the interference of background operations on foreground services is minimized, thereby improving the overall stability and performance predictability of the system.

[0314] Differences from existing technologies:

[0315] Unlike traditional system management methods based on static configuration or local optimization, this invention proposes a global, closed-loop, data-driven adaptive control mechanism that endows the storage system with a certain degree of intelligence and flexibility.

[0316] like Figure 9 As shown, based on the same inventive concept as the key-value indexing method described above, this application embodiment also provides a key-value indexing device, including a response unit 91 and a processing unit 92.

[0317] The response unit 91 is used to: determine the popularity information of a specific key in response to an access operation to that specific key; wherein the popularity information includes the latest access frequency and the latest access timestamp of a specific range to which the specific key belongs; the specific range is obtained by dividing the entire key-value space;

[0318] Processing unit 92 is used to: determine the target data layer to which a specific key belongs based on heat information and set hierarchical parameters; wherein the target data layer includes a hot data layer, a warm data layer, or a cold data layer;

[0319] Processing unit 92 is used to: index based on the indexing strategy of the target data layer and determine the specific value corresponding to a specific key.

[0320] In one alternative implementation, the response unit 91 is specifically used for:

[0321] Determine the specific range to which a particular key belongs; where keys within the same range have similar access characteristics;

[0322] Update the access frequency counter value corresponding to a specific range to obtain the latest access frequency;

[0323] Update the access timestamp corresponding to a specific range to the current time to obtain the latest access timestamp.

[0324] In one optional implementation, the processing unit 92 is specifically used for:

[0325] The score for a specific key is calculated based on the latest access frequency, the latest access timestamp, and the set hierarchical parameters. The set hierarchical parameters include a time decay factor, access frequency weight, and access relevance weight. The access relevance weight is used to indicate the importance of access frequency, and the score is used to indicate the current overall popularity level of the specific key.

[0326] Based on the score and the relationship between the pre-set hot layer threshold, warm layer threshold and cold layer threshold, the target data layer to which a specific key belongs is determined.

[0327] In one optional implementation, the processing unit 92 is specifically used for:

[0328] If the score is greater than the hot layer threshold, the target data layer to which the specific key belongs is determined to be the hot data layer;

[0329] If the score is less than the hot layer threshold and greater than the warm layer threshold, and the learning index status indicates that a specific key has been managed by the learning index in the warm data layer, then the target data layer to which the specific key belongs is determined to be the warm data layer.

[0330] Otherwise, determine that the target data layer to which the specific key belongs is the cold data layer.

[0331] In one optional implementation, the processing unit 92 is specifically used for:

[0332] If the target data layer is a hot data layer, then the specific value corresponding to a specific key is determined based on the mapping relationship of key-value pairs represented by the hot data layer in the MAP structure.

[0333] If the target data layer is a warm data layer, then a specific key is input into the pre-trained index model so that the index model indexes the location of the key stored in the first-value log read-only archive partition and determines that the value stored at the location is a specific value; wherein, the first-value log read-only archive partition is generated by the garbage collection process;

[0334] If the target data layer is a cold data layer, then based on the LSM tree structure, the location of a specific key is determined, and the value stored at that location is determined to be a specific value. Here, each node of the LSM tree structure stores a key-value pair pointing to the second value log. The second value log refers to the active log and the value log in the archive partition that is not covered by the warm data layer. The key-value pair of the second value log is a key-location pair.

[0335] In an optional implementation, the processing unit 92 is further configured to:

[0336] In response to the garbage collection process, scan the log of values ​​to be collected in the active log to determine valid key-value pairs;

[0337] Valid key-value pairs are written sequentially to a new value log archive partition, generating a file containing key-value pairs and recording the key range covered by the new value log archive partition; the file containing key-value pairs is a file containing key-position pairs.

[0338] Create an indexing task that trains the indexing model based on the metadata of the new value log archive partition and the current popularity of the key range.

[0339] In an optional implementation, the processing unit 92 is further configured to:

[0340] The strategies for adjusting the thermal threshold include:

[0341] If the hit rate of the hot data layer is lower than the hit rate threshold and the capacity utilization is greater than the utilization threshold, then the hot layer threshold is reduced.

[0342] If the capacity utilization of the hot data layer is lower than the utilization threshold or the eviction rate is greater than the eviction rate threshold, and the warm data layer has the ability to accept new index instances, then the hot layer threshold is increased.

[0343] If the average query latency of the cold data layer exceeds the maximum tolerable value, then the threshold for the hot layer should be reduced.

[0344] The strategies for adjusting the temperature threshold include:

[0345] If the actual average return on investment of the warm data layer learning index is lower than the return threshold, or the average effective lifespan of the index model is less than the lifespan threshold, then the warm layer threshold will be increased.

[0346] If the number of new cold data in the cold data layer that are waiting for garbage collection and index model processing has an access popularity higher than the popularity threshold exceeds the quantity threshold, and the warm data layer has the ability to accept and manage more index instances, then the warm layer threshold is lowered.

[0347] The key-value indexing device proposed in this application adopts the same inventive concept as the key-value indexing method described above, and can achieve the same beneficial effects, so it will not be described again here.

[0348] Based on the same inventive concept as the key-value indexing method described above, this application also provides an electronic device, which may specifically be a desktop computer, portable computer, smartphone, tablet computer, personal digital assistant (PDA), server, etc. Figure 10 As shown, the electronic device may include a processor 1001 and a memory 1002.

[0349] The processor 1001 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0350] Memory 1002, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. Memory is any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. Memory 1002 in the embodiments of this application may also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0351] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned computer storage medium can be any available medium or data storage device that a computer can access, including but not limited to: mobile storage devices, random access memory (RAM), magnetic storage (e.g., floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (e.g., CDs, DVDs, BDs, HVDs, etc.), and semiconductor storage (e.g., ROMs, EPROMs, EEPROMs, non-volatile memory (NAND flash), solid-state drives (SSDs)) and other media capable of storing program code.

[0352] Alternatively, if the integrated units described above in this application are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods of the various embodiments of this application. The aforementioned storage medium includes: mobile storage devices, random access memory (RAM), magnetic memory (e.g., floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical memory (e.g., CDs, DVDs, BDs, HVDs, etc.), and semiconductor memory (e.g., ROMs, EPROMs, EEPROMs, non-volatile memory (NAND flash), solid-state drives (SSDs), etc.) and other media capable of storing program code.

[0353] Based on the same inventive concept, this application also provides a computer program product, which includes computer program code that, when run on a computer, causes the computer to execute any of the key-value indexing methods discussed above. Since the principle by which the above computer program product solves the problem is similar to that of the key-value indexing method, the implementation of the above computer program product can be referred to the implementation of the method, and repeated details will not be elaborated further.

[0354] The above embodiments are only used to provide a detailed description of the technical solutions of this application. However, the description of the above embodiments is only for the purpose of helping to understand the methods of the embodiments of this application and should not be construed as a limitation on the embodiments of this application. Any changes or substitutions that can be easily conceived by those skilled in the art should be covered within the protection scope of the embodiments of this application.

Claims

1. A key-value indexing method, characterized by, The method comprises the following steps: In response to an access operation on a specific key, determining the hotness information of the specific key; wherein the hotness information comprises the latest access frequency and the latest access timestamp of a specific range to which the specific key belongs; the specific range is obtained by dividing the entire key-value space; According to the hotness information and a set hierarchical parameter, determining a target data layer to which the specific key belongs; wherein the target data layer is a hot data layer, a warm data layer, or a cold data layer; Based on the index strategy of the target data layer, indexing is performed to determine the specific value corresponding to the specific key.

2. The method of claim 1, wherein, The determination of the hotness information of the specific key comprises: Determining the specific range to which the specific key belongs; wherein the keys in the same range have similar access characteristics; Updating the value of the access frequency counter corresponding to the specific range to obtain the latest access frequency; Updating the access timestamp corresponding to the specific range to the current time to obtain the latest access timestamp.

3. The method according to claim 1 or 2, characterized in that, The determination of the target data layer to which the specific key belongs according to the hotness information and the set hierarchical parameter comprises: According to the latest access frequency, the latest access timestamp, and the set hierarchical parameter, calculating the score of the specific key; wherein the set hierarchical parameter comprises a time decay factor, an access frequency weight, and an access recency weight; the access recency weight is used to indicate the importance of the access frequency, and the score is used to indicate the overall hotness level of the specific key at present; According to the relationship between the score and the pre-set hot layer threshold, warm layer threshold, and cold layer threshold, determining the target data layer to which the specific key belongs.

4. The method of claim 3, wherein, The determination of the target data layer to which the specific key belongs according to the relationship between the score and the pre-set hot layer threshold, warm layer threshold, and cold layer threshold comprises: If the score is greater than the hot layer threshold, it is determined that the target data layer to which the specific key belongs is the hot data layer; If the score is less than the hot layer threshold and greater than the warm layer threshold, and the learning index state indicates that the specific key has been managed by the learning index in the warm data layer, it is determined that the target data layer to which the specific key belongs is the warm data layer; Otherwise, it is determined that the target data layer to which the specific key belongs is the cold data layer.

5. The method of claim 1, wherein, The indexing based on the index strategy of the target data layer to determine the specific value corresponding to the specific key comprises: If the target data layer is the hot data layer, the mapping relationship between the key-value pairs represented by the MAP structure of the hot data layer is determined to determine the specific value corresponding to the specific key; If the target data layer is the warm data layer, the specific key is input into a pre-trained index model, so that the index model indexes the position of the key stored in the first value log read-only archive partition, and determines the value stored at the position as the specific value; wherein the first value log read-only archive partition is generated by a garbage collection process. If the target data layer is a cold data layer, determining the position of the specific key based on the cold data layer of the LSM tree structure, and determining that the value stored at the position is the specific value; wherein each node of the LSM tree structure stores a key-value pair pointing to a second value log, the second value log being a value log in an active log and an archived partition not covered by the warm data layer, the key-value pair of the second value log being a key-position pair.

6. The method of claim 5, wherein, The method further comprises: In response to a garbage collection process, scanning the value log to be recycled in the active log to determine valid key-value pairs; sequentially writing the valid key-value pairs to a new value log archived partition, generating a file containing key-value pairs, and recording the key range covered by the new value log archived partition; wherein the file containing key-value pairs is a file containing key-position pairs; creating an index task, training the index model based on the meta information of the new value log archived partition and the current heat of the key range.

7. The method according to any one of claims 1 to 2, 4 to 6, characterized in that, The method further comprises: If the hit rate of the hot data layer is lower than a hit rate threshold and the capacity utilization rate is greater than a utilization rate threshold, then the hot layer threshold is reduced; If the capacity utilization rate of the hot data layer is lower than the utilization rate threshold or the eviction rate is greater than an eviction rate threshold, and the warm data layer has the ability to accommodate new index instances, then the hot layer threshold is increased; If the average query delay of the cold data layer is greater than a maximum tolerable value, then the hot layer threshold is reduced; If the actual average return on investment of the warm data layer learning index is lower than a return rate threshold, or the average effective life cycle of the index model is less than a life cycle threshold, then the warm layer threshold is increased; If the number of new cold data with access heat higher than a heat threshold in the cold data layer waiting for the garbage collection process and the index model processing is greater than a number threshold, and the warm data layer has the ability to accommodate and manage more index instances, then the warm layer threshold is reduced.

8. A key-value index apparatus, characterized by, comprises: a response unit configured to determine heat information of a specific key in response to an access operation on the specific key; wherein the heat information comprises a latest access frequency and a latest access timestamp of a specific range to which the specific key belongs; the specific range being obtained by dividing the entire key-value space; a processing unit configured to determine a target data layer to which the specific key belongs according to the heat information and a set hierarchical parameter; wherein the target data layer is a hot data layer, a warm data layer, or a cold data layer. The processing unit is configured to index based on an index strategy of the target data layer to determine a specific value corresponding to the specific key.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the method of any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon computer program instructions, wherein, The computer program instructions are executed by the processor to implement the steps of the method of any one of claims 1 to 7.

Citation Information

Patent Citations

  • Cloud storage system based on user collaboration cloud

    CN103442057A

  • Persistent memory key value storage system

    CN111309270A

  • Data storage method and device

    CN117149100A

  • Data processing method and device, electronic equipment and readable medium

    CN119166630A

  • Multi-NDP system read-write acceleration method based on key value separation technology

    CN119271666A

Cited By

  • Key value data processing method, database platform, equipment, medium and product

    CN121501836A