An Adaptive Optimization Method for State-Aware Hierarchical Storage Index Structure

Abstract

This invention relates to an adaptive optimization method for a state-aware hierarchical storage index structure, belonging to the field of computer storage and data indexing. It consists of a base-layer metadata index table and a hot-layer accelerated index table, both sharing the same contiguous shared data area. An in-band hotness encoding area is constructed using reusable bit fields of existing metadata fields in the base-layer buckets, embedding access hotness information into redundant bit fields of the bucket metadata for in-band encoding. On the query path, a probability sampling method is used to trigger base-layer hotness state updates, reducing the number of atomic writes to shared cache lines in high-frequency access scenarios. A state vector is constructed, and the sampling probability and hot-layer admission threshold are dynamically adjusted accordingly to achieve hot object identification, index promotion, and hierarchical accelerated access. This invention improves the access efficiency and stability of dense hash tables under complex load conditions without requiring the construction of an external counting structure that grows linearly with the bucket size.

Description

Technical Field

[0001] This invention relates to a state-aware, hierarchical storage index structure adaptive optimization method, and more particularly to a dense hash index optimization method and system that achieves in-band heat awareness and hierarchical adaptive acceleration for access skew without introducing an external counting array of linear size to the bucket size. It belongs to the field of computer storage and data indexing technology. Background Technology

[0002] With the rapid development of large-scale data processing and storage systems, efficient key indexing and lookup capabilities have become critical infrastructure for key-value stores, memory caching systems, file system metadata management, and high-performance computing. Dense open-address hash tables, due to their contiguous memory array-based bucket structure and metadata, offer high space utilization efficiency and good cache locality, and are therefore widely used in these systems. In engineering practice, to reduce memory consumption and improve efficiency per unit space, hash tables are often run for extended periods under high load factors.

[0003] However, in real-world operating environments, access requests often exhibit a significant non-uniformity, meaning a small number of keys (hot keys) are accessed frequently, while a large number of keys (cold keys) are accessed less frequently, resulting in a long-tailed or power-law distribution (e.g., Zipf distribution). When high load factors and access skewed concurrency exist, the performance of open-address hash tables is more easily affected: due to the increasingly crowded bucket structure, collisions are more frequent, and clusters formed during probing are more pronounced, leading to longer probing sequences for some keys. For hot keys, these longer probing paths are repeatedly executed, easily causing increased access latency, aggravated tail latency (such as high quantile latency) fluctuations, and further amplifying cache misses and synchronization contention in concurrent scenarios, thus causing a decrease in system throughput and a deterioration in service stability.

[0004] To address collision and clustering issues in open-address hash tables, existing technologies often employ linear probing, RobinHood hashing, and their improved strategies. These methods improve element distribution and reduce lookup failure probability by maintaining metadata information such as probe distances. These methods typically achieve good average performance under low to medium load or near-uniform access scenarios. However, the core logic of existing rearrangement algorithms like RobinHood is based on fairness judgment using "physical displacement," favoring elements with greater physical distances for preemption. This mechanism is essentially "hotness-independent." When access distribution is skewed, it cannot distinguish between high-frequency hotspots and long-tail cold data. It might even push frequently triggered hotspot keys deeper into the probe sequence to optimize rarely accessed cold data, causing the probe overhead for hotspot keys to increase rather than decrease, significantly increasing cache access pressure under overall load.

[0005] To mitigate the impact of access skew on system performance, some existing technologies attempt to introduce key access frequency or popularity information to guide data rearrangement, position swapping, or the construction of hierarchical caches. However, existing hotspot acceleration solutions typically face a dual dilemma: on the one hand, in-place rearrangement operations disrupt the probe monotonicity of mechanisms such as Robin Hood hashing, leading to a significant increase in retrieval logic complexity and performance fluctuations; on the other hand, popularity statistics mechanisms based on auxiliary structures usually rely on additional counters or auxiliary indexes, which, in addition to increasing storage usage, more importantly introduce additional random memory accesses into the tight loop of query and update paths, easily causing cache pollution, increased cache miss rates, and increased memory bandwidth consumption. In multi-threaded concurrent environments, if popularity updates involve atomic operations or shared counters, they can also easily lead to increased cache consistency traffic, cache row contention, and false sharing, making it difficult to achieve low-interference, low-overhead online popularity awareness under high load and high concurrency conditions.

[0006] Furthermore, many existing optimization strategies employ fixed parameters or static configurations, lacking the ability to dynamically adapt to changes in hash table operation status and access patterns. When load factors, access skewness, or concurrency levels change, fixed strategies struggle to adjust the trade-off between heat statistics overhead and optimization trigger frequency in a timely manner, potentially limiting optimization effectiveness and even leading to performance degradation in some scenarios.

[0007] Therefore, under complex operating conditions such as high load, high access skew, and concurrent access, existing technologies still suffer from problems such as large overhead for heat perception, insufficient coordination between hot spot identification and access optimization, lack of state-aware adjustment capabilities, and heavy hierarchical maintenance logic. There is an urgent need for a state-aware, hierarchical storage index structure adaptive optimization method to achieve low contention heat tracking, hierarchical acceleration of hot objects, and runtime adaptive access optimization without significantly increasing additional storage overhead, thereby improving the access efficiency and stability of dense open addressing hash indexes under real load. Summary of the Invention

[0008] To address the problems of existing dense open-address hash tables, such as lengthened probe paths, increased access latency (especially high quantile latency fluctuations), and decreased throughput due to additional storage structures and concurrency contention introduced by online hotspot statistics, this invention provides a state-aware, hierarchical storage index structure adaptive optimization method. This method achieves low-interference awareness of hotspot keys and runtime adaptive acceleration without requiring the construction of an external counting structure that grows linearly with the bucket size, thereby improving the access efficiency and stability of dense hash tables under complex load conditions.

[0009] The present invention adopts the following technical solution:

[0010] An adaptive optimization method for a state-aware hierarchical storage index structure includes:

[0011] S1, initialize the storage layout, configure three contiguous storage spaces in memory, including the base layer metadata index table for carrying the full index, the hot layer accelerated index table for carrying the hot spot bypass accelerated index, and the shared data area;

[0012] S2, during the execution of the target key query or update operation, the hot layer accelerated index table is searched first. If a match is found, the shared data area is located directly using the storage index. If a match is not found, the process falls back to the base layer metadata index table to perform standard probing and executes step S3. The target key is the identification information used for hash mapping and locating the target data object.

[0013] S3: Access and parse the base layer metadata index table. If no match is found, terminate; if a match is found, record the state vector V of the base layer metadata index table operation for the matched object, including the global load factor. Access skewness index Average detection distance and performance degradation detection signals;

[0014] S4, Adaptively adjust sampling probability based on state vector V And update the admission threshold of the thermal acceleration index table;

[0015] S5, based on sampling probability Update popularity level ;

[0016] S6, based on actual detection distance and popularity level Calculate promotion score When promotion score When the threshold is greater than the updated admission threshold, the hit object and its storage index in the shared data area are marked as hotspot promotion candidates; when When the threshold value is less than or equal to the updated admission threshold, the system does not trigger index level changes and still performs standard probe access through the base layer metadata index table, without triggering index insertion in the hot layer accelerated index table.

[0017] S7. For hot promotion candidates, perform hot layer promotion operation and insert the fingerprint of the hit object and the storage index of the hit object in the shared data area into the hot layer acceleration index table.

[0018] The core technical idea of ​​this invention is as follows: the system consists of a base layer metadata index table (Base Layer, or base layer for short) and a hot layer acceleration index table (Hot Tier, or hot layer for short), both sharing the same contiguous shared data area. An in-band hotness encoding area is constructed using reusable bit fields of existing metadata fields in the base layer buckets, embedding access hotness information into redundant bit fields of the bucket metadata for in-band encoding. On the query path, base layer hotness status updates are triggered through probability sampling to reduce the number of atomic writes to shared cache lines in high-frequency access scenarios, thereby reducing concurrency contention.

[0019] Meanwhile, the system dynamically adjusts the sampling probability and admission threshold based on the operating status (including global load factor, average detection distance, and access skewness index). When performance degradation is detected, high-value hot keys and their corresponding data area indexes are promoted to the hot layer based on a comprehensive promotion score composed of the base layer detection distance and in-band heat. Through a two-level indexing mechanism of "prioritizing hot layer retrieval and directly accessing data area based on index," the system achieves logical skipping and on-demand acceleration of the redundant detection chain of the base layer while maintaining the monotonicity of the physical distribution of the base layer and the semantics of conflict resolution.

[0020] Preferably, in step S1, the three consecutive memory storage spaces establish a mapping relationship through pointers or offsets;

[0021] The starting addresses of the base layer metadata index table and the hot layer acceleration index table are aligned with the processor cache line size (preferably 64 bytes) to improve cache utilization and reduce the risk of false sharing in a multi-core environment;

[0022] The shared data area is shared by the base layer metadata index table and the hot layer acceleration index table. The base layer metadata index table consists of multiple buckets, which are the basic storage units of the base layer metadata index table. Each bucket has a W-bit wide metadata field, which is divided into a function bit field and a popularity bit field. The popularity bit field occupies N bits, with the optimal selection... Used to encode access popularity level, the function bit field occupies WN bits, used to store hash fingerprint, occupancy status and control information used for conflict resolution and early stop criteria;

[0023] An in-band heat encoding region is constructed within the heat bit field. The heat encoding uses an N-bit saturation counter. When the count reaches 2... N The system reaches saturation at -1 to prevent overflow from affecting the functional bit field; the system uses bit mask isolation operations to ensure that atomic updates only affect the hot bit field and keep the functional bit field unchanged; in a preferred embodiment, the hot bit field can be right-shifted and decayed or graded back-off according to the time window or operation window to reflect the timeliness of hot spots, and does not introduce an external counting array of the same size as the base layer storage bucket.

[0024] During initialization, the functional fields of the base layer metadata index table are set to empty bucket status, and the heat field is set to a preset initial value (preferably 0). The hot layer acceleration index table is initialized to empty to await subsequent hot spot promotion insertion. A shared data area is configured to store target data objects or key-value pairs, and a mapping relationship with the base layer is established. The system establishes empty bucket semantic protection rules: during empty bucket determination, fingerprint comparison, and conflict resolution, only the functional fields are judged. For example, FUNC_MASK=~HOT_MASK is used to mask the heat field before performing occupancy status or fingerprint matching to ensure that any heat encoding status will not change the empty bucket determination and retrieval semantics; where =~ indicates the bitwise inversion result.

[0025] The base layer metadata index table maintains the original conflict resolution and early stop criteria, the hotness field is set to a preset initial value, and the hot layer acceleration index table is put into a pending admission state.

[0026] Preferably, in step S3, the global load factor = , This indicates the number of buckets currently occupied by the base layer metadata index table. This indicates the total number of buckets in the base layer metadata index table;

[0027] Detection range This represents the number of probe steps from the initial hash position to the target bucket, and the average probe distance. By measuring the detection distance We get the average.

[0028] To avoid cache pollution caused by a global scan of the base layer, an access skewness metric is constructed. As an approximate representation of the degree of heat concentration. Access skewness index. The frequency of buckets that reach a high heat level during the in-band heat update process is obtained by sampling along the path, which is achieved by synchronously recording the frequency of buckets that reach a high heat level, thus approximately reflecting the concentration of hotspots.

[0029] To reduce the impact of instantaneous fluctuations within the window on the degradation determination results, the system can adjust the average detection distance. and access skewness index Smoothing can be performed using exponential moving average, sliding average, or other low-overhead time-series filtering methods.

[0030] current Performance degradation detection signal at any moment From average detection distance and access skewness index The decision, specifically:

[0031] Average detection range Preferably, the statistics are obtained by the thread-local accumulator within a preset window, when the following conditions are met. and When the query cost degradation in the base layer metadata index table is determined to be caused by access skew, and the degradation continues at K... on When a statistical window occurs, Setting it to 1 indicates that the current state is degenerate; when the query cost degenerates continuously at K... off If no statistical window occurs, Setting it to 0 indicates that the current state has exited the degenerate state and entered a non-degenerate state; other cases will... Set as ;in, This represents the threshold value for the average detection distance index. This indicates the threshold for the access skewness index; express Performance degradation detection signal at any time; K on K represents the number of degradation confirmation windows. off This indicates the number of confirmation windows that have been restored.

[0032] Preferred, Using loading factor The associated adaptive threshold mechanism can be expressed as:

[0033]

[0034] in, Indicates the current load factor The baseline expected detection range can be obtained by calibrating based on historical statistical results from the system initialization phase or normal operation phase. This indicates a tolerance margin, used to allow for a certain degree of normal fluctuation;

[0035] Access skewness index To characterize the concentration of access popularity in the base layer, it can be defined as:

[0036]

[0037] in, This indicates the number of times the bucket in the high-heat level state within the window has been triggered. This indicates the total number of visits within the window; A fixed percentage threshold can be selected, such as 0.1 to 0.4.

[0038] Preferably, in step S4, the sampling probability of updating the in-band heat value is based on the state vector V. Gating control and adaptive computation are performed to control the triggering frequency of heat update, thereby achieving a balance between heat perception accuracy and concurrent write overhead. This process can be implemented through a feedback gating module, which is an independent control module set in the system control path, independent of the base layer metadata index table, the heat layer acceleration index table, or the shared data area.

[0039] when =0, meaning that when no performance degradation occurs, the sampling probability P is set to 0 or limited to a lower range to disable or weaken the hot bit field update, thereby maintaining near-zero additional overhead for the basic layer query path;

[0040] when =1, meaning that when performance degradation occurs, the intensity of heat tracking is increased, and the adaptive calculation phase of sampling probability is initiated. First, candidate values ​​of sampling probability are calculated. :

[0041]

[0042] in, As the baseline sampling probability, This is the load suppression coefficient, which can be taken as 1 to 4. This is the skew enhancement factor, which can be taken as 1 to 3. As the global load factor, To access the skewness metric; this function satisfies the following adjustment trend: when the global load factor When it rises, Decrease to suppress atomic write conflicts and cache consistency overhead under high concurrency; when the skewness index When it rises, Increase the size to accelerate hotspot capture and heat convergence.

[0043] To ensure the usability and stability of the sampling probability, candidate values ​​for the sampling probability are... By performing interval truncation, the final sampling probability is obtained. :

[0044]

[0045] in, This represents the interval cutoff function, used to truncate candidate sampling probabilities. Limited to a preset range [ Within ], to obtain the final sampling probability. .

[0046] in, The meaning is: if < Then take = ;like ≤ ≤ ,Pick = ;like > Then take = .

[0047] , These represent the lower and upper bounds of the sampling probability, respectively; and satisfy 0 ≤ ≤1.

[0048] Preferably, in step S4, during initialization, a thermal layer access threshold baseline value is set. ( This is the baseline reference value for the thermal layer admission threshold. During initialization, the current thermal layer admission threshold is set. Set as ,Right now = Degradation trigger threshold and recovery trigger threshold , Less than Degradation trigger threshold Used to determine whether the current detection cost has entered the degradation range. To determine whether the current detection cost has fallen back to the recovery range, it can be set or calibrated based on the baseline statistical results of the detection cost during normal system operation, offline stress test results, or historical operating data.

[0049] The system continuously monitors the detection cost statistics within the monitoring window. , Average detection distance can be taken ;

[0050] When performance degradation detection signal =1, meaning that when performance degradation occurs, a downward update is performed on the threshold, and the amount of downward adjustment is... for:

[0051]

[0052] This is the threshold down-adjustment coefficient under the degradation state. This is the skew enhancement coefficient, used to characterize the amplification effect of the degree of skew on the threshold reduction amplitude;

[0053] Updated admission thresholds = , The previous hotspot admission threshold;

[0054] When performance degradation detection signal =0, and the current detection cost statistic < When the current base layer is determined to have exited the degradation zone and entered the recovery zone, the entry threshold is adjusted upwards to gradually tighten the hotspot entry conditions, reduce the risk of excessive expansion of the hotspot layer and frequent maintenance, and the adjustment amount is increased. for:

[0055]

[0056] in, The threshold adjustment coefficient is used in the recovery state.

[0057] Updated admission thresholds ;

[0058] When the current detection cost statistics satisfy When the system determines that the current basic layer metadata index table is in a maintenance interval, the admission threshold remains unchanged within this maintenance interval. = .

[0059] Meanwhile, to avoid performance fluctuations caused by frequent changes in the thermal layer under high load or high concurrency conditions, the system sets a thermal layer admission threshold. Implement stability protection strategies, including but not limited to: improving The lower limit, reduction The adjustment range, or the introduction of a minimum trigger interval (at least a preset number of operations or a time window between two adjacent layout maintenances), can be adjusted to avoid jitter under high load while ensuring that the detection path is substantially shortened.

[0060] Preferably, the implementation process in step S5 is as follows:

[0061] S51, when the target key is matched in the base layer metadata index table during a query or update operation, based on the current sampling probability... The probability of triggering this event is determined to suppress redundant hot write operations in high-frequency access scenarios, thereby reducing atomic write contention and cache consistency traffic in concurrent environments. Sampling probability. Restricted to [ , Within the range; when the system is in a non-degradable state or low-intervention mode, Can be reverted to near A smaller value is used to reduce hot write overhead.

[0062] When a pseudo-random number R is accessed and a hit occurs in the base layer metadata index table, a value R < 0 is generated. If a trending status update is triggered, step S52 is executed; otherwise, the current trending status update is ignored.

[0063] S52, after triggering the heat status update, the system uses atomic read-write operations to perform a saturated incremental update only on the heat bit field of the hit bucket, causing the heat bit field to undergo state evolution and obtain the updated heat level. And the functional bit fields remain unchanged during the update process through bit masking operations;

[0064] S53, in the updated popularity ranking Update the value of the metadata field based on The system uses a Compare-and-Swap (CAS) instruction to attempt to change the value of the old metadata field in memory. Updated to If the update fails due to concurrency competition, the value of the latest metadata field is obtained and steps S51-S52 are repeated. To avoid the hot bucket spinning for a long time under high concurrency, the hot update is abandoned directly after the number of repetitions reaches the preset limit, so as to ensure the throughput and tail delay stability of the query path.

[0065] In step S51, the pseudo-random number R is a random number uniformly distributed in the interval [0,1). To reduce overhead, a thread-local pseudo-random sequence or a lightweight hash function based on thread identifier and operation sequence number is used to generate it.

[0066] Preferably, in step S52, to avoid disrupting the original conflict resolution semantics and early retrieval stopping criteria of the basic layer metadata index table, the system follows the following isolation principle:

[0067] The heat index bit field does not participate in empty bucket determination, fingerprint matching, or early stopping criteria of the probe chain; in logic involving conflict resolution and early stopping judgment, the system uses bitmask extraction function bit field for judgment, ensuring that changes in the heat index bit field value do not affect the correctness of the base layer retrieval; specifically:

[0068] The heat field mask is HOT_MASK, and the function field mask is FUNC_MASK, where FUNC_MASK = ~HOT_MASK, where = ~ indicates the bitwise inversion result. The displacement of the heat field is defined as SHIFT. For example, when the heat field is located in the highest N bits of the metadata field, we have: SHIFT = WN.

[0069] Perform an atomic read on the metadata field of the base layer hit bucket to obtain the value of the old metadata field. The current popularity level is extracted in the register through bit masking and bit shifting operations. :

[0070]

[0071] Here, & represents a bitwise AND operation. This indicates a bitwise right shift operation;

[0072] The updated heat level is calculated using saturation increment. :

[0073]

[0074] In step S53, the updated metadata value is constructed. :

[0075] |( )

[0076] Where | represents a bitwise OR operation. This indicates a bitwise left shift operation;

[0077] In step S53, the preset upper limit is 1 to 2 times. In the event of contention or conflict, the system may abandon the current hot layer promotion operation or perform only a limited number of lightweight retries to maintain the low-blocking characteristics of the query path.

[0078] To prevent the heat value from overflowing and interfering with the functional bit field under extremely high frequency access, heat encoding employs saturation protection; when the count reaches 2... N After -1, subsequent increases are suppressed, and the highest heat level is maintained.

[0079] To prevent historical hotspots from maintaining high popularity levels for extended periods and affecting the timeliness of hotspot access, the system can periodically perform attenuation processing on the popularity bit domain. Preferably, attenuation is achieved through periodic right shifts or window-triggered tiered declines, for example... := 1. Upcoming popularity level A one-bit logical right shift is performed to approximately halve the heat value, thereby achieving an exponential time decay effect. The decay can be triggered by window boundaries or gating control signals, and there is no need to introduce an external counting array of linear size to the bucket size.

[0080] Preferably, in step S6, based on the actual detection distance of the current visit... Heat level extracted from heat domain Calculate the hot topic promotion score according to the preset scoring function. This is used to quantify the potential performance benefits that an object may bring after entering the hot layer. The potential performance benefits are mainly reflected in the fact that if the object is covered by the hot layer index, subsequent accesses can preferentially hit through the hot layer, thereby reducing the number of times the probe path is repeatedly traversed in the base layer and reducing access latency and probe overhead caused by long probes. Therefore, the scoring function preferably reflects two types of factors simultaneously: the probe cost corresponding to the current access and the activity level of the object being repeatedly accessed within the current time range. Specifically:

[0081]

[0082] in, Indicates about detection range A monotonically increasing mapping function; This indicates the popularity level. A monotonically increasing mapping function; The weighting coefficient represents the detection range. The weighting coefficient represents the popularity level.

[0083] When system degradation is mainly manifested as a significant increase in detection range, the system increases the weighting coefficient of detection range. And correspondingly reduce the weighting coefficient of the popularity level. This makes the scoring more inclined to prioritize covering keys with high access costs; when system degradation is mainly caused by increased access concentration, the system increases the weighting coefficient of the popularity level. Prioritize coverage of frequently accessed hot keys.

[0084] Preferably, in step S7, the hotspot promotion operation refers to introducing the index information of the hotspot promotion candidate into the hotspot acceleration index table, thereby providing a fast access path for subsequent queries. Preferably, the promotion operation is only started when the feedback gating module allows hotspot maintenance and meets the minimum trigger interval or hotspot capacity budget constraint, so as to reduce the additional maintenance overhead during high load or jitter phases.

[0085] The process of constructing and inserting hotspot index entries is as follows:

[0086] During the hot layer promotion operation, the system constructs a corresponding hot layer index entry for the target hot object; the index entry includes at least: the fingerprint or simplified identifier of the target object (for fast matching), the storage index pointing to the actual object in the shared data area, and optional status information (such as access timestamp, heat mirror value or version number).

[0087] For objects already residing in the hot layer acceleration index table, the access timestamp, heat mirror value, or local access count of the hot layer entry is updated upon hot layer hit. If necessary, the base layer heat bit field can be asynchronously written back according to a preset sampling probability, or the detection distance of the object's most recent base layer metadata index table can be cached in the hot layer entry as a static cost item. Preferably, the hot layer index array is implemented using a capacity-constrained data structure, whose size is significantly smaller than the base layer bucket array, and can be set according to the number of concurrent threads, cache capacity, or preset maintenance budget.

[0088] Concurrency consistency protection is implemented when inserting index entries, including: using atomic compare-and-swap (CAS) to complete the occupancy of index slots; using fine-grained locks or segmented locks to protect hot-layer local areas; and introducing version number verification to avoid concurrent insertions and reads being interleaved.

[0089] In subsequent query operations, a hierarchical retrieval strategy is adopted:

[0090] The system prioritizes searching the hot-layer accelerated index table and performs fast matching on the target object. If the hot-layer accelerated index table is matched, the system first performs fast matching based on the fingerprint in the hot-layer index entry, and then performs full key verification and / or version verification on the target key in the shared data area. If the verification passes, the target object is returned; if the verification fails, it is considered a hot-layer false hit, and the system falls back to the base layer metadata index table to perform standard probing. The corresponding hot-layer entry can also be invalidated or refreshed. When the hot-layer accelerated index table is not matched, standard probing is performed according to the original conflict resolution strategy and early stopping criteria of the base layer metadata index table. The physical layout or retrieval invariants of the base layer metadata index table are not changed due to the hot-layer accelerated index table mechanism.

[0091] Preferably, the hot-layer accelerated index table implements a capacity control and eviction mechanism. When the index array in the hot-layer accelerated index table reaches the preset capacity limit or maintenance budget, the system implements a hot spot eviction strategy to maintain the timeliness and stability of the hot-layer accelerated index table. Specifically, the system determines whether to implement the hot spot eviction strategy based on the recent access activity, recent access time and heat decay of the hot-layer index entries.

[0092] The evicted hot layer index entries are only removed from the hot layer accelerated index table, and their storage in the base layer metadata index table and shared data area remains unchanged. There is no physical data migration or base layer metadata index table rearrangement involved.

[0093] After a hot layer promotion or eviction operation is completed, the system releases the relevant concurrency control flags or lock resources. Since the base layer structure remains physically static and the hot layer exists only as a bypass index, hot layer maintenance operations do not affect the correctness of concurrent query threads.

[0094] Through the above design, the system achieves continuous adaptive acceleration of skewed access load while minimizing interference with the query path, and reduces the access cost of hot objects under high load conditions through the hot layer bypass access mechanism.

[0095] For any details not covered in this invention, please refer to the prior art.

[0096] The beneficial effects of this invention are as follows:

[0097] This invention introduces in-band hotspot representation and a budgeted hotspot update mechanism, directly encoding hotspot features into the reusable bit fields of the base layer bucket metadata without adding an external hotspot counting array. Combined with Bernoulli sampling-triggered updates and atomic bit field write-back mechanisms, the hotspot awareness process significantly reduces redundant writes and cache consistency traffic under high concurrency access, thereby reducing cache pollution, write contention, and false sharing risks introduced by hotspot statistics, and improving the concurrency scalability and access stability of dense open addressing hash tables under high load and skew access conditions.

[0098] Meanwhile, this invention constructs a closed-loop adaptive optimization process of "state awareness - gating parameter tuning - candidate identification - hierarchical acceleration": The system generates a performance degradation detection signal based on the global load factor, average detection distance, and access skewness index, and adaptively adjusts the sampling probability and the hot layer admission threshold accordingly; when the degradation of the base layer detection cost caused by skew is detected, the system uses the joint score of the base layer detection distance and the in-band heat level to identify high-value hot spot promotion candidates and promotes their index information to the hot layer acceleration index table. By using the bypass acceleration path, the system prioritizes hitting the index of the hot layer acceleration index table during the query, thereby logically skipping the redundant detection chain in the base layer metadata index table. Without changing the physical layout and conflict resolution semantics of the base layer metadata index table, the system significantly reduces the access cost of hot spot keys and suppresses high quantile latency fluctuations caused by long-tail detection; when the access distribution tends to be uniform or the load drops, the system automatically reduces the intensity of heat perception and acceleration maintenance through the gating mechanism, thereby avoiding unnecessary maintenance overhead and performance regression risks.

[0099] Furthermore, the core operations of this invention are implemented using hardware primitives such as bitmasks, bit shifts, and atomic comparisons and swaps. This allows for low-intrusion integration into existing key-value storage systems, memory caches, database indexes, and file system metadata management systems that rely on dense hash tables. This is used to improve overall access efficiency, concurrency stability, and system robustness under real complex load conditions (high load factor and access skew). Attached Figure Description

[0100] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an undue limitation of this application.

[0101] Figure 1 This is a schematic diagram of the overall architecture of the state-aware hierarchical storage index structure adaptive optimization method of the present invention;

[0102] Figure 2 This is a flowchart of the state-aware hierarchical storage index structure adaptive optimization method of the present invention;

[0103] Figure 3 This is a flowchart illustrating the performance degradation detection signal determination process in the state-aware hierarchical storage index structure adaptive optimization method of the present invention. Detailed Implementation

[0104] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. However, this is not the only description; all aspects not described in detail herein are based on conventional techniques in the art.

[0105] Terminology Explanation:

[0106] 1. Access skew: refers to the access characteristics of access requests being unevenly distributed and concentrated in hotspots among different target objects. Its degree can be characterized by the access skewness index.

[0107] 2. Hash Table: A hash structure that stores key-value pairs or key-value pair indexes in a contiguous array of buckets, using open addressing to handle collisions (including but not limited to linear probing, quadratic probing, and RobinHood probing). Its key feature is the compact layout of the bucket array and bucket data to improve space utilization and reduce memory fragmentation. Dense hash tables use a storage method that separates metadata from data: the bucket array stores metadata and indexes pointing to the data area, while the actual key-value pairs are stored in a separate data area.

[0108] 3. Bucket: A bucket is a basic unit of storage in a hash table used to hold a key-value pair or to point to the storage location of a key-value pair. A bucket typically contains metadata fields associated with the key-value pair, or an index pointing to the key-value pair entity in the data area.

[0109] 4. Base Layer Metadata Index Table: The base layer, or simply base layer, is a dense, open-addressed hash structure used to carry the full keyspace index and perform conflict resolution semantics. The base layer is implemented as a contiguous array of memory buckets, with each bucket index pointing to a key-value pair entity in the data area. The base layer maintains a predetermined probing strategy and early stopping rules, and does not perform physical bucket rearrangement or key migration operations during changes in popularity or the construction of the acceleration layer.

[0110] 5. Hot Layer Accelerated Index Table: The hot layer, or simply hot layer, refers to a capacity-limited adaptive index layer generated based on access frequency. It stores the fingerprint information of hot keys and their index positions in the data area, allowing for priority matching during queries and direct location to the data area upon a match, logically skipping the base layer's probe chain.

[0111] 6. Shared Data Area: This refers to a contiguous memory region used for physically storing the actual key-value pairs. This data area is shared by the base layer and the hot layer, meaning that both levels of indexes ultimately point to the same physical storage location within the data area, without copying the key-value pairs.

[0112] 7. Bucket Metadata Fields: These are control information fields associated with the storage bucket, used to support open address conflict resolution, location, and status determination. Metadata fields may include, but are not limited to: probe distance information, hash fingerprint (fingerprint or tag), occupied status flags (occupied, empty, tombstone), and other control bits.

[0113] Example 1

[0114] An adaptive optimization method for state-aware hierarchical storage index structure, such as Figure 1 and Figure 2 As shown, it includes:

[0115] S1, initialize the storage layout, configure three contiguous storage spaces in memory, including the base layer metadata index table for carrying the full index, the hot layer accelerated index table for carrying the hot spot bypass accelerated index, and the shared data area;

[0116] S2, during the execution of the target key query or update operation, the hot layer accelerated index table is searched first. If a match is found, the shared data area is located directly using the storage index. If a match is not found, the process falls back to the base layer metadata index table to perform standard probing and executes step S3. The target key is the identification information used for hash mapping and locating the target data object.

[0117] S3: Access and parse the base layer metadata index table. If no match is found, terminate; if a match is found, record the state vector V of the base layer metadata index table operation for the matched object, including the global load factor. Access skewness index Average detection distance and performance degradation detection signals;

[0118] S4, Adaptively adjust sampling probability based on state vector V And update the admission threshold of the thermal acceleration index table;

[0119] S5, based on sampling probability Update popularity level ;

[0120] S6, based on actual detection distance and popularity level Calculate promotion score When promotion score When the threshold is greater than the updated admission threshold, the hit object and its storage index in the shared data area are marked as hotspot promotion candidates; when When the threshold value is less than or equal to the updated admission threshold, the system does not trigger index level changes and still performs standard probe access through the base layer metadata index table, without triggering index insertion in the hot layer accelerated index table.

[0121] S7. For hot promotion candidates, perform hot layer promotion operation and insert the fingerprint of the hit object and the storage index of the hit object in the shared data area into the hot layer acceleration index table.

[0122] Example 2

[0123] An adaptive optimization method for a state-aware hierarchical storage index structure, as described in Example 1, differs in that, in step S1, the three consecutive memory storage spaces establish a mapping relationship through pointers or offsets.

[0124] The starting addresses of the base layer metadata index table and the hot layer acceleration index table are aligned with the processor cache line size (preferably 64 bytes) to improve cache utilization and reduce the risk of false sharing in a multi-core environment;

[0125] The shared data area is shared by the base layer metadata index table and the hot layer acceleration index table. The base layer metadata index table consists of multiple buckets, which are the basic storage units of the base layer metadata index table. Each bucket has a W-bit wide metadata field, which is divided into a function bit field and a popularity bit field. The popularity bit field occupies N bits, with the optimal selection... Used to encode access popularity level, the function bit field occupies WN bits, used to store hash fingerprint, occupancy status and control information used for conflict resolution and early stop criteria;

[0126] An in-band heat encoding region is constructed within the heat bit field. The heat encoding uses an N-bit saturation counter. When the count reaches 2... NThe system reaches saturation at -1 to prevent overflow from affecting the functional bit field; the system uses bit mask isolation operations to ensure that atomic updates only affect the hot bit field and keep the functional bit field unchanged; in a preferred embodiment, the hot bit field can be right-shifted and decayed or graded back-off according to the time window or operation window to reflect the timeliness of hot spots, and does not introduce an external counting array of the same size as the base layer storage bucket.

[0127] During initialization, the functional fields of the base layer metadata index table are set to empty bucket status, and the heat field is set to a preset initial value (preferably 0). The hot layer acceleration index table is initialized to empty to await subsequent hot spot promotion insertion. A shared data area is configured to store target data objects or key-value pairs, such as KV entity 1, KV entity 2, etc., and a mapping relationship with the base layer is established. The system establishes empty bucket semantic protection rules: during empty bucket determination, fingerprint comparison, and conflict resolution, only the functional fields are judged. For example, FUNC_MASK=~HOT_MASK is used to mask the heat field before performing occupancy status or fingerprint matching to ensure that any heat encoding status will not change the empty bucket determination and retrieval semantics, where =~ indicates bitwise inversion.

[0128] The base layer metadata index table maintains the original conflict resolution and early stop criteria, the hotness field is set to a preset initial value, and the hot layer acceleration index table is put into a pending admission state.

[0129] Example 3

[0130] A state-aware adaptive optimization method for hierarchical storage index structure, as described in Example 2, differs in that, in step S3, the global load factor... = , This indicates the number of buckets currently occupied by the base layer metadata index table. This indicates the total number of buckets in the base layer metadata index table;

[0131] Detection range This represents the number of probe steps from the initial hash position to the target bucket, and the average probe distance. By measuring the detection distance We get the average.

[0132] To avoid cache pollution caused by a global scan of the base layer, an access skewness metric is constructed. As an approximate representation of the degree of heat concentration. Access skewness index. The frequency of buckets that reach a high heat level during the in-band heat update process is obtained by sampling along the path, which is achieved by synchronously recording the frequency of buckets that reach a high heat level, thus approximately reflecting the concentration of hotspots.

[0133] To reduce the impact of instantaneous fluctuations within the window on the degradation determination results, the system can adjust the average detection distance. and access skewness index Smoothing can be performed using exponential moving average, sliding average, or other low-overhead time-series filtering methods.

[0134] current Performance degradation detection signal at any moment From average detection distance and access skewness index Decision, such as Figure 3 As shown, specifically:

[0135] Average detection range Preferably, the statistics are obtained by the thread-local accumulator within a preset window, when the following conditions are met. and When the query cost degradation in the base layer metadata index table is determined to be caused by access skew, and the degradation continues at K... on When a statistical window occurs, Setting it to 1 indicates that the current state is degenerate; when the query cost degenerates continuously at K... off If no statistical window occurs, Setting it to 0 indicates that the current state has exited the degenerate state and entered a non-degenerate state; other cases will... Set as ;in, This represents the threshold value for the average detection distance index. This indicates the threshold for the access skewness index; express Performance degradation detection signal at any time; K on K represents the number of degradation confirmation windows. off This indicates the number of confirmation windows that have been restored.

[0136] Preferred, Using loading factor The associated adaptive threshold mechanism can be expressed as:

[0137]

[0138] in, This represents the baseline expected detection distance under the current load factor α, which can be calibrated based on historical statistical results from the system initialization or normal operation phases. This indicates a tolerance margin, used to allow for a certain degree of normal fluctuation;

[0139] Access skewness index To characterize the concentration of access popularity in the base layer, it can be defined as:

[0140]

[0141] in, This indicates the number of times the bucket in the high-heat level state within the window has been triggered. This indicates the total number of visits within the window; A fixed percentage threshold can be selected; in this embodiment, it is set to 0.3.

[0142] Example 4

[0143] An adaptive optimization method for a state-aware hierarchical storage index structure, as described in Example 3, differs in that, in step S4, the sampling probability of updating the in-band heat value is based on the state vector V. Gating control and adaptive computation are performed to control the triggering frequency of heat update, thereby achieving a balance between heat perception accuracy and concurrent write overhead. This process can be implemented through a feedback gating module, which is an independent control module set in the system control path, independent of the base layer metadata index table, the heat layer acceleration index table, or the shared data area.

[0144] when =0, meaning that when no performance degradation occurs, the sampling probability P is set to 0 or limited to a lower range to disable or weaken the hot bit field update, thereby maintaining near-zero additional overhead for the basic layer query path;

[0145] when =1, meaning that when performance degradation occurs, the intensity of heat tracking is increased, and the adaptive calculation phase of sampling probability is initiated. First, candidate values ​​of sampling probability are calculated. :

[0146]

[0147] in, As the baseline sampling probability, This is the load suppression coefficient, which can be taken as 1 to 4. This is the skew enhancement factor, which can be taken as 1 to 3. As the global load factor, To access the skewness metric; this function satisfies the following adjustment trend: when the global load factor When it rises, Decrease to suppress atomic write conflicts and cache consistency overhead under high concurrency; when the skewness index When it rises, Increase the size to accelerate hotspot capture and heat convergence.

[0148] To ensure the usability and stability of the sampling probability, candidate values ​​for the sampling probability are... By performing interval truncation, the final sampling probability is obtained. :

[0149]

[0150] in, This represents the interval cutoff function, used to truncate candidate sampling probabilities. Limited to a preset range [ Within ], to obtain the final sampling probability. .

[0151] in, The meaning is: if < Then take = ;like ≤ ≤ ,Pick = ;like > Then take = .

[0152] , These represent the lower and upper bounds of the sampling probability, respectively; and satisfy 0 ≤ ≤1.

[0153] Example 5

[0154] An adaptive optimization method for a state-aware hierarchical storage index structure, as described in Example 4, differs in that, in step S4, a hot layer admission threshold baseline value is set during initialization. ( This is the baseline reference value for the thermal layer admission threshold. During initialization, the current thermal layer admission threshold is set. Set as ,Right now = Degradation trigger threshold and recovery trigger threshold , Less than Degradation trigger threshold Used to determine whether the current detection cost has entered the degradation range. To determine whether the current detection cost has fallen back to the recovery range, it can be set or calibrated based on the baseline statistical results of the detection cost during normal system operation, offline stress test results, or historical operating data.

[0155] The system continuously monitors the detection cost statistics within the monitoring window. , Average detection distance can be taken ;

[0156] When performance degradation detection signal =1, meaning that when performance degradation occurs, a downward update is performed on the threshold, and the amount of downward adjustment is... for:

[0157]

[0158] This is the threshold down-adjustment coefficient under the degradation state. This is the skew enhancement coefficient, used to characterize the amplification effect of the degree of skew on the threshold reduction amplitude;

[0159] Updated admission thresholds = , The previous hotspot admission threshold;

[0160] When performance degradation detection signal =0, and the current detection cost statistic < When the current base layer is determined to have exited the degradation zone and entered the recovery zone, the entry threshold is adjusted upwards to gradually tighten the hotspot entry conditions, reduce the risk of excessive expansion of the hotspot layer and frequent maintenance, and the adjustment amount is increased. for:

[0161]

[0162] in, The threshold adjustment coefficient is used in the recovery state.

[0163] Updated admission thresholds ;

[0164] When the current detection cost statistics satisfy When the system determines that the current basic layer metadata index table is in a maintenance interval, the admission threshold remains unchanged within this maintenance interval. = .

[0165] Meanwhile, to avoid performance fluctuations caused by frequent changes in the thermal layer under high load or high concurrency conditions, the system sets a thermal layer admission threshold. Implement stability protection strategies, including but not limited to: improving The lower limit, reduction The adjustment range, or the introduction of a minimum trigger interval (at least a preset number of operations or a time window between two adjacent layout maintenances), can be adjusted to avoid jitter under high load while ensuring that the detection path is substantially shortened.

[0166] Example 6

[0167] An adaptive optimization method for a state-aware hierarchical storage index structure, as described in Example 5, differs in that the implementation process in step S5 is as follows:

[0168] S51, when the target key is matched in the base layer metadata index table during a query or update operation, based on the current sampling probability... The probability of triggering this event is determined to suppress redundant hot write operations in high-frequency access scenarios, thereby reducing atomic write contention and cache consistency traffic in concurrent environments. Sampling probability. Restricted to [ , Within the range; when the system is in a non-degradable state or low-intervention mode, Can be reverted to near A smaller value is used to reduce hot write overhead.

[0169] When a hit occurs in the base layer metadata index table, a pseudo-random number R is generated. R is a random number uniformly distributed in the interval [0,1). To reduce overhead, a thread-local pseudo-random sequence or a lightweight hash function based on thread identifier and operation sequence number is used. When R < If a trending status update is triggered, step S52 is executed; otherwise, the current trending status update is ignored.

[0170] S52, after triggering the heat status update, the system uses atomic read-write operations to perform a saturated incremental update only on the heat bit field of the hit bucket, causing the heat bit field to undergo state evolution and obtain the updated heat level. And the functional bit fields remain unchanged during the update process through bit masking operations;

[0171] In step S52, to avoid disrupting the original conflict resolution semantics and early retrieval stopping criteria of the basic layer metadata index table, the system follows the following isolation principles:

[0172] The heat index bit field does not participate in empty bucket determination, fingerprint matching, or early stopping criteria of the probe chain; in logic involving conflict resolution and early stopping judgment, the system uses bitmask extraction function bit field for judgment, ensuring that changes in the heat index bit field value do not affect the correctness of the base layer retrieval; specifically:

[0173] The heat field mask is HOT_MASK, and the function field mask is FUNC_MASK, where FUNC_MASK = ~HOT_MASK, where = ~ indicates the bitwise inversion result. The displacement of the heat field is defined as SHIFT. For example, when the heat field is located in the highest N bits of the metadata field, we have: SHIFT = WN.

[0174] Perform an atomic read on the metadata field of the base layer hit bucket to obtain the value of the old metadata field. The current popularity level is extracted in the register through bit masking and bit shifting operations. :

[0175]

[0176] Here, & represents a bitwise AND operation. This indicates a bitwise right shift operation;

[0177] The updated heat level is calculated using saturation increment. :

[0178]

[0179] In step S53, the updated metadata value is constructed. :

[0180] |( )

[0181] Where | represents a bitwise OR operation. This indicates a bitwise left shift operation.

[0182] S53, in the updated popularity ranking Update the value of the metadata field based on The system uses a Compare-and-Swap (CAS) instruction to attempt to change the value of the old metadata field in memory. Updated to If the update fails due to concurrency competition, the value of the latest metadata field is obtained and steps S51-S52 are repeated. To avoid the hot bucket spinning for a long time under high concurrency, when the number of repetitions reaches the preset limit, the hot update is abandoned to ensure the throughput and tail delay stability of the query path.

[0183] The preset limit is 1 to 2 attempts. In the event of contention or conflict, the system may abandon the current hot layer promotion operation or perform only a limited number of lightweight retries to maintain the low-blocking characteristics of the query path.

[0184] To prevent the heat value from overflowing and interfering with the functional bit field under extremely high frequency access, heat encoding employs saturation protection; when the count reaches 2... N After -1, subsequent increases are suppressed, and the highest heat level is maintained.

[0185] To prevent historical hotspots from maintaining high popularity levels for extended periods and affecting the timeliness of hotspot access, the system can periodically perform attenuation processing on the popularity bit domain. Preferably, attenuation is achieved through periodic right shifts or window-triggered tiered declines, for example... := 1. Upcoming popularity level A one-bit logical right shift is performed to approximately halve the heat value, thereby achieving an exponential time decay effect. The decay can be triggered by window boundaries or gating control signals, and there is no need to introduce an external counting array of linear size to the bucket size.

[0186] Example 7

[0187] A state-aware adaptive optimization method for hierarchical storage index structure, as described in Example 6, differs in that, in step S6, the actual detection distance of the current access is used... Heat level extracted from heat domain Calculate the hot topic promotion score according to the preset scoring function. This is used to quantify the potential performance benefits that an object may bring after entering the hot layer. The potential performance benefits are mainly reflected in the fact that if the object is covered by the hot layer index, subsequent accesses can preferentially hit through the hot layer, thereby reducing the number of times the probe path is repeatedly traversed in the base layer and reducing access latency and probe overhead caused by long probes. Therefore, the scoring function preferably reflects two types of factors simultaneously: the probe cost corresponding to the current access and the activity level of the object being repeatedly accessed within the current time range. Specifically:

[0188]

[0189] in, Indicates about detection range A monotonically increasing mapping function; This indicates the popularity level. A monotonically increasing mapping function; The weighting coefficient represents the detection range. The weighting coefficient represents the popularity level.

[0190] When system degradation is mainly manifested as a significant increase in detection range, the system increases the weighting coefficient of detection range. And correspondingly reduce the weighting coefficient of the popularity level. This makes the scoring more inclined to prioritize covering keys with high access costs; when system degradation is mainly caused by increased access concentration, the system increases the weighting coefficient of the popularity level. Prioritize coverage of frequently accessed hot keys.

[0191] In step S7, the hotspot promotion operation refers to introducing the index information of the hotspot promotion candidate into the hotspot acceleration index table, thereby providing a fast access path for subsequent queries. Preferably, the promotion operation is only started when the feedback gating module allows hotspot maintenance and meets the minimum trigger interval or hotspot capacity budget constraint, so as to reduce the additional maintenance overhead during high load or jitter phases.

[0192] The process of constructing and inserting hotspot index entries is as follows:

[0193] During the hot layer promotion operation, the system constructs a corresponding hot layer index entry for the target hot object; the index entry includes at least: the fingerprint or simplified identifier of the target object (for fast matching), the storage index pointing to the actual object in the shared data area, and optional status information (such as access timestamp, heat mirror value or version number).

[0194] For objects already residing in the hot layer acceleration index table, the access timestamp, heat mirror value, or local access count of the hot layer entry is updated upon hot layer hit. If necessary, the base layer heat bit field can be asynchronously written back according to a preset sampling probability, or the detection distance of the object's most recent base layer metadata index table can be cached in the hot layer entry as a static cost item. Preferably, the hot layer index array is implemented using a capacity-constrained data structure, whose size is significantly smaller than the base layer bucket array, and can be set according to the number of concurrent threads, cache capacity, or preset maintenance budget.

[0195] Concurrency consistency protection is implemented when inserting index entries, including: using atomic compare-and-swap (CAS) to complete the occupancy of index slots; using fine-grained locks or segmented locks to protect hot-layer local areas; and introducing version number verification to avoid concurrent insertions and reads being interleaved.

[0196] In subsequent query operations, a hierarchical retrieval strategy is adopted:

[0197] The system prioritizes searching the hot-layer accelerated index table and performs fast matching on the target object. If the hot-layer accelerated index table is matched, the system first performs fast matching based on the fingerprint in the hot-layer index entry, and then performs full key verification and / or version verification on the target key in the shared data area. If the verification passes, the target object is returned; if the verification fails, it is considered a hot-layer false hit, and the system falls back to the base layer metadata index table to perform standard probing. The corresponding hot-layer entry can also be invalidated or refreshed. When the hot-layer accelerated index table is not matched, standard probing is performed according to the original conflict resolution strategy and early stopping criteria of the base layer metadata index table. The physical layout or retrieval invariants of the base layer metadata index table are not changed due to the hot-layer accelerated index table mechanism.

[0198] Preferably, the hot-layer accelerated index table implements a capacity control and eviction mechanism. When the index array in the hot-layer accelerated index table reaches the preset capacity limit or maintenance budget, the system implements a hot spot eviction strategy to maintain the timeliness and stability of the hot-layer accelerated index table. Specifically, the system determines whether to implement the hot spot eviction strategy based on the recent access activity, recent access time and heat decay of the hot-layer index entries.

[0199] The evicted hot layer index entries are only removed from the hot layer accelerated index table, and their storage in the base layer metadata index table and shared data area remains unchanged. There is no physical data migration or base layer metadata index table rearrangement involved.

[0200] After a hot layer promotion or eviction operation is completed, the system releases the relevant concurrency control flags or lock resources. Since the base layer structure remains physically static and the hot layer exists only as a bypass index, hot layer maintenance operations do not affect the correctness of concurrent query threads.

[0201] Through the above design, the system achieves continuous adaptive acceleration of skewed access load while minimizing interference with the query path, and reduces the access cost of hot objects under high load conditions through the hot layer bypass access mechanism.

[0202] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A state-aware adaptive optimization method for hierarchical storage index structure, characterized in that, include: S1, initialize the storage layout, configure three contiguous storage spaces in memory, including the base layer metadata index table for carrying the full index, the hot layer accelerated index table for carrying the hot spot bypass accelerated index, and the shared data area; S2, during the execution of the target key query or update operation, the hot layer accelerated index table is searched first. If it is found, the storage index is used to directly locate the shared data area. If it is not found, the process falls back to the base layer metadata index table to perform standard probing and execute step S3. S3, access and parse the base layer metadata index table; if not found, end. If a hit occurs, record the state vector V of the underlying metadata index table of the hit object, including the global load factor. Access skewness index Average detection distance and performance degradation detection signals; S4, Adaptively adjust sampling probability based on state vector V And update the admission threshold of the thermal acceleration index table; S5, based on sampling probability Update popularity level ; S6, based on actual detection distance and popularity level Calculate promotion score When promotion score When the threshold is greater than the updated admission threshold, the hit object and its storage index in the shared data area are marked as hotspot promotion candidates; when When the threshold value is less than or equal to the updated admission threshold, the system does not trigger index level changes and still performs standard probe access through the base layer metadata index table, without triggering index insertion in the hot layer accelerated index table. S7. For hot promotion candidates, perform hot layer promotion operation and insert the fingerprint of the hit object and the storage index of the hit object in the shared data area into the hot layer acceleration index table.

2. The adaptive optimization method for state-aware hierarchical storage index structure according to claim 1, characterized in that, In step S1, a mapping relationship is established between the three contiguous memory spaces through pointers or offsets; The starting addresses of the base layer metadata index table and the hot layer acceleration index table are aligned with the processor cache line size; The shared data area is shared by the base layer metadata index table and the hot layer acceleration index table. The base layer metadata index table consists of multiple storage buckets. Each storage bucket has a W-bit wide metadata field. The metadata field is divided into a function bit field and a heat bit field. The heat bit field occupies N bits and is used to encode the access heat level. The function bit field occupies WN bits and is used to store hash fingerprints, occupancy status, and control information used for conflict resolution and early stop criteria. An in-band heat encoding region is constructed within the heat bit field. The heat encoding uses an N-bit saturation counter. When the count reaches 2... N Saturation is reached at -1; During initialization, the function fields of the base layer metadata index table are set to empty buckets, and the heat field is set to a preset initial value; the hot layer acceleration index table is initialized to empty to await subsequent hot spot promotion insertion. Put the thermal acceleration index table into an admission-pending state.

3. The adaptive optimization method for state-aware hierarchical storage index structure according to claim 2, characterized in that, In step S3, the global load factor = , This indicates the number of buckets currently occupied by the base layer metadata index table. This indicates the total number of buckets in the base layer metadata index table; Detection range This represents the number of probe steps from the initial hash position to the target bucket, and the average probe distance. By measuring the detection distance We get the average. Access skewness index The frequency at which buckets reaching high heat levels are triggered during the in-band heat update process is obtained by sampling along the path. current Performance degradation detection signal at any moment From average detection distance and access skewness index The decision, specifically: When satisfied and When the query cost degradation in the base layer metadata index table is determined to be caused by access skew, and the degradation continues at K... on When a statistical window occurs, Setting it to 1 indicates that the current state is degenerate; When the query cost degenerates continuously at K off If no statistical window occurs, Setting it to 0 indicates that the current state has exited the degenerate state and entered a non-degenerate state; other cases will... Set as ;in, This represents the threshold value for the average detection distance index. This indicates the threshold for the access skewness index; express Performance degradation detection signal at any time; K on K represents the number of degradation confirmation windows. off This indicates the number of confirmation windows that have been restored.

4. The adaptive optimization method for state-aware hierarchical storage index structure according to claim 3, characterized in that, In step S4, the sampling probability of updating the in-band heat value is based on the state vector V. Gating control and adaptive computation are performed to control the triggering frequency of heat update, thereby achieving a balance between heat sensing accuracy and concurrent write overhead. when =0, meaning that when no performance degradation occurs, the sampling probability P is set to 0 or limited to a low range to disable or weaken the hot bit field update; when =1, meaning that when performance degradation occurs, the intensity of heat tracking is increased, and the adaptive calculation phase of sampling probability is initiated. First, candidate values ​​of sampling probability are calculated. : in, As the baseline sampling probability, This is the load suppression coefficient. This is the skew enhancement coefficient. As the global load factor, To access the skewness index; To ensure the usability and stability of the sampling probability, candidate values ​​for the sampling probability are... By performing interval truncation, the final sampling probability is obtained. : in, Represents the interval cutoff function. , These represent the lower and upper bounds of the sampling probability, respectively; and satisfy 0 ≤ ≤1.

5. The adaptive optimization method for state-aware hierarchical storage index structure according to claim 4, characterized in that, In step S4, during initialization, the thermal layer access threshold baseline value is set. Degradation trigger threshold and recovery trigger threshold , Less than Degradation trigger threshold Used to determine whether the current detection cost has entered the degradation range. Used to determine whether the current detection cost has fallen back to the recovery range; The system continuously monitors the detection cost statistics within the monitoring window. , Take average detection distance ; When performance degradation detection signal =1, meaning that when performance degradation occurs, a downward update is performed on the threshold, and the amount of downward adjustment is... for: This is the threshold down-adjustment coefficient under the degradation state. This is the skew enhancement coefficient, used to characterize the amplification effect of the degree of skew on the threshold reduction amplitude; Updated admission thresholds = , The previous hotspot admission threshold; When performance degradation detection signal =0, and the current detection cost statistic < When it is determined that the current base layer has entered the recovery interval, an upward update is performed on the threshold, and the upward adjustment amount is... for: in, The threshold adjustment coefficient is used in the recovery state. Updated admission thresholds ; When the current detection cost statistics satisfy When the system determines that the current basic layer metadata index table is in a maintenance interval, the admission threshold remains unchanged within this maintenance interval. = .

6. The adaptive optimization method for state-aware hierarchical storage index structure according to claim 5, characterized in that, The implementation process in step S5 is as follows: S51, when a query or update operation hits the base layer metadata index table, based on the current sampling probability... The probability of triggering the event is determined by sampling probability. Restricted to [ , Within the range; when a hit occurs in the basic layer metadata index table, a pseudo-random number R is generated, and when R < If a trending status update is triggered, step S52 is executed; otherwise, the current trending status update is ignored. S52, after triggering the heat status update, the system uses atomic read-write operations to perform a saturated incremental update only on the heat bit field of the hit bucket, thus obtaining the updated heat level. And the functional bit fields remain unchanged during the update process through bit masking operations; S53, in the updated popularity ranking Update the value of the metadata field based on The system uses a compare-and-swap instruction to attempt to change the value of the old metadata field in memory. Updated to If the update fails, obtain the value of the latest metadata field and repeat steps S51-S52; when the number of repetitions reaches the preset limit, abandon the current popularity update. In step S51, the pseudo-random number R is a random number uniformly distributed in the interval [0,1), which is generated by using a thread-local pseudo-random sequence or a lightweight hash function based on thread identifier and operation sequence number.

7. The adaptive optimization method for state-aware hierarchical storage index structure according to claim 6, characterized in that, In step S52, the heat bit field mask is HOT_MASK, the function bit field mask is FUNC_MASK, where FUNC_MASK = ~HOT_MASK, where = ~ indicates the bit-inverted result, and the displacement of the heat bit field is defined as SHIFT. Perform an atomic read on the metadata field of the base layer hit bucket to obtain the value of the old metadata field. The current popularity level is extracted in the register through bit masking and bit shifting operations. : Here, & represents a bitwise AND operation. This indicates a bitwise right shift operation; The updated heat level is calculated using saturation increment. : In step S53, the updated metadata value is constructed. : |( ) Where | represents a bitwise OR operation. This indicates a bitwise left shift operation; In step S53, the preset upper limit is 1 to 2 times.

8. The adaptive optimization method for state-aware hierarchical storage index structure according to claim 7, characterized in that, In step S6, based on the actual detection distance of the current visit... Heat level extracted from heat domain Calculate the hot topic promotion score according to the preset scoring function. : in, Indicates about detection range A monotonically increasing mapping function; This indicates the popularity level. A monotonically increasing mapping function; The weighting coefficient represents the detection range. The weighting coefficient represents the popularity level.

9. The adaptive optimization method for state-aware hierarchical storage index structure according to claim 8, characterized in that, In step S7, the process of constructing and inserting hot layer index entries is as follows: During the hotspot promotion operation, the system constructs a corresponding hotspot index entry for the target hotspot object; Index entries include: the fingerprint or simplified identifier of the target object, the storage index pointing to the actual object in the shared data area, and optional status information; For objects that already reside in the hot layer acceleration index table, when a hot layer hit occurs, update the access timestamp, heat mirror value, or local access count of the hot layer entry. In subsequent query operations, a hierarchical retrieval strategy is adopted: The system prioritizes searching the hot-layer accelerated index table and performs fast matching on the target object. If the hot-layer accelerated index table is found, the system first performs fast matching based on the fingerprint in the hot-layer index entry, and then performs full key verification and / or version verification on the target key in the shared data area. If the verification passes, the target object is returned; if the verification fails, it is considered a hot-layer false hit, and the system falls back to the base layer metadata index table to perform standard probing. When the hot-layer accelerated index table is not found, standard probing is performed according to the original conflict resolution strategy and early stopping criteria of the base layer metadata index table. The physical layout or retrieval invariants of the base layer metadata index table are not changed due to the hot-layer accelerated index table mechanism.

10. The adaptive optimization method for state-aware hierarchical storage index structure according to claim 9, characterized in that, The hotspot acceleration index table implements a capacity control and eviction mechanism. When the index array in the hotspot acceleration index table reaches the preset capacity limit or maintenance budget, the system executes a hotspot eviction strategy to maintain the timeliness and stability of the hotspot acceleration index table. The evicted hot layer index entries are only removed from the hot layer accelerated index table, and their storage in the base layer metadata index table and shared data area remains unchanged.

Images