Distributed big data efficient storage and index optimization method

Abstract

The invention belongs to the technical field of distributed storage optimization, and discloses a distributed big data efficient storage and index optimization method, which comprises the following steps: building a three-level storage and index system of a DPU local memory, SSD and NVMe nodes and an HDD cluster through a DPU collaborative architecture, and deeply integrating the DPU computing power into index processing; meanwhile, a corresponding rule of a query type and an index structure is established, a sparse B +-tree is started for range query, an FST reverse index is started for full-text retrieval, and synchronous migration of data and the index and metadata compression are realized by relying on IndexCompress and IndexMigranate special instructions; high-frequency access data and corresponding indexes are accurately deployed in high-performance hardware, high-frequency service access delay is reduced, and multi-service requirements in a hybrid load scene are adapted; through a dynamic weight rule of a five-dimensional evaluation matrix, the weight ratio of each dimension is flexibly adjusted in combination with a hardware health coefficient and a service priority, and high-frequency and low-frequency access fragments are intelligently divided.

Description

Technical Field

[0001] This invention belongs to the field of distributed storage optimization technology, specifically a method for efficient distributed big data storage and indexing optimization. Background Technology

[0002] Distributed storage systems have become a crucial infrastructure in fields such as cloud computing, big data, and artificial intelligence. They bear the heavy responsibility of persistent storage, efficient access, and elastic scaling of massive amounts of data, supporting the stable operation of diverse business scenarios such as online transaction processing and online analytical processing. Distributed storage systems need to adapt to heterogeneous hardware such as data processing units, non-volatile memory host controller interface standard devices, solid-state drives, hard disk drives, and mixed workload scenarios. However, existing technologies have the following technical problems:

[0003] Firstly, there is a lack of coordination: storage is tiered (hot and cold data are separated), index design is based on B+ trees and hashes, and hardware capabilities such as data processing unit computing power and media performance are independently optimized; hot data is stored on solid-state drives but the corresponding index resides on mechanical hard drives, the data processing unit is only used for data forwarding and does not participate in index calculation, and structured and unstructured data use a unified index;

[0004] Secondly, adaptive failure: relying on static strategies to cope with dynamic changes, fixed shard sizes such as 128MB lead to overload of nodes during peak load periods and low cluster utilization during idle periods. Unified consistency protocols increase the delay of non-critical data updates, and asynchronous updates cause index conflicts. Existing technologies such as HiDPU and HaSiS only solve single-module problems and have not broken through the limitations of integrated collaboration and adaptation. Summary of the Invention

[0005] The purpose of this invention is to provide a method for efficient storage and indexing of distributed big data to solve the problems mentioned in the background.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a distributed big data high-efficiency storage and indexing optimization method, including a five-dimensional intelligent sharding stage, a DPU collaborative architecture stage, an index hardware matching stage, a criticality adaptation stage, a linkage adjustment stage, a heterogeneous node balancing stage, and a query collaborative acceleration stage.

[0007] Preferably, the five-dimensional intelligent sharding stage establishes a five-dimensional evaluation matrix based on raw data, node hardware parameters, historical access logs, real-time hardware health data, and service level agreement requirements. Weights are allocated using preset dynamic weight rules. When the hardware health coefficient is <0.8, the hardware capability weight is set to 50%. When the proportion of high-priority services is >60%, the SLA weight is set to 40%. For other scenarios, the weights are allocated as follows: hardware capability 35%, load prediction 30%, SLA 15%, data characteristics 10%, and hardware health 10%. This process is used to divide high-frequency access shards into low-frequency access shards.

[0008] After the sharding migration, the DPU collects migration time and service latency data, which is then fed back to the load prediction module to correct prediction parameters and dynamically adjust the high-frequency access sharding threshold.

[0009] The final result is a list of high-frequency access shards and low-frequency access shards, an initial shard and node allocation scheme, and load prediction correction parameters.

[0010] The load prediction module employs a two-factor prediction algorithm based on the fusion of time series data and business characteristics, specifically:

[0011] Input data preprocessing: The migration time and service latency data collected by DPU are cleaned (outliers other than 3σ are removed), and features are extracted including the 5-minute moving average of migration time, the P95 quantile of service latency, and the time-series trend of shard access frequency.

[0012] Prediction parameter definition: The prediction parameters include the high-frequency access sharding threshold coefficient K (initial value 1.2) and the node load carrying capacity coefficient β (initial value 0.8), where K is used to dynamically adjust the high-frequency sharding size threshold and β is used to correct the node's assessment of the sharding carrying capacity.

[0013] Parameter correction logic: The proportional-integral (PI) adjustment algorithm is used to calculate the correction amount based on the migration time deviation rate Δt and the service delay error Δd.

[0014] Δt = (Actual migration time - Preset migration time) / Preset migration time

[0015] Δd = Actual business delay - Predicted business delay

[0016] ΔK = 0.3 × Δt + 0.1 × ∫Δtdt

[0017] Δβ = 0.2 × Δd + 0.05 × ∫Δddt

[0018] When Δt > 10% or Δd > 5ms, immediate correction is triggered; otherwise, batch correction is performed at 5-minute intervals.

[0019] Linkage with the five-dimensional evaluation matrix: The corrected K and β parameters are fed back to the dynamic weight allocation stage. K is used to adjust the division threshold of high-frequency segments, and the high-frequency segment size = base size × K. β is used to correct the weight ratio of the hardware capability dimension, and the corrected hardware capability weight = original hardware capability weight × β.

[0020] The formulas for calculating the PI adjustment parameters and threshold calibration of the load prediction module are as follows:

[0021]

[0022] Actual business delay - Predicted business delay

[0023]

[0024]

[0025] High-frequency slice size = baseline size

[0026] Corrected hardware capability weights = Original hardware capability weights

[0027] The integral term , The integration interval is the current correction period. For immediate correction, the integration interval is 1 second; for batch correction, the integration interval is 5 minutes. The sampling interval for immediate correction is 1 second, and the condition is met three times consecutively. or Correction is triggered at 5ms; the initial value of the high-frequency shard base size is 16MB by default, and can be adjusted according to the total capacity of the storage cluster. Total capacity of storage cluster Dynamic calibration (MB) and 8MB ensures the reference size is within a reasonable range.

[0028] Preferably, the DPU collaborative architecture stage, based on the sharding list and initial allocation scheme of the five-dimensional intelligent sharding stage, and combined with the parameters of each storage level including DPU local memory capacity, SSD performance, NVMe node performance, and HDD cluster capacity, establishes a three-level storage and indexing architecture of DPU local memory, SSD and NVMe nodes, and HDD cluster, and simultaneously deploys the DPU collaborative control plane; the control plane has built-in hardware adaptation rules, including automatically allocating 30% of the DPU computing cores to process prefix matching when NVMe nodes adapt to the digital prefix tree index, synchronously triggering index metadata updates when migrating across levels, and developing dedicated commands for IndexCompress and IndexMigrate. IndexCompress compresses index metadata in segments according to value ranges using the RoaringBitmap algorithm, and IndexMigrate realizes synchronous migration of data and index, controlling the processing time of index metadata;

[0029] The DPU monitors the storage utilization of each level in real time and triggers low-frequency access data archiving and index dimensionality reduction operations when the utilization exceeds the preset threshold, so as to balance the load of each level.

[0030] The final output includes a three-level storage and index architecture configuration table, DPU control plane scheduling rules, and index metadata processing standards.

[0031] Preferably, the index hardware matching stage is based on the configuration table and DPU control plane scheduling rules of the DPU collaborative architecture stage, combined with the query request samples and the index support capabilities of each hardware node, to establish a rule for the correspondence between query type and index structure; range queries use sparse B+- trees, prefix matching queries use numeric or character prefix trees, and full-text search uses FST inverted indexes; after receiving a query request, the DPU parses the request header and matches the corresponding index structure according to the hardware adaptation rules;

[0032] For SSD nodes, when IO utilization is less than 60% and query frequency exceeds 50 times / minute, record-level indexes are split into field-level indexes. For HDD nodes, when access frequency is less than 5 times / minute, adjacent 4MB block-level indexes are merged into a 16MB superblock index to reduce the number of IO operations. At the same time, DPUs share key values ​​and physical address caches with the host to reduce interaction resource consumption.

[0033] This ultimately results in a table mapping queries, indexes, and hardware, as well as instructions for splitting and merging indexes.

[0034] Index splitting / merging algorithm steps:

[0035] Step 1: Input hardware node IO utilization (U), query frequency (F), and current index granularity (G);

[0036] Step 2: For SSD nodes, the trigger condition for splitting is U < 60% and F ≥ 50 times / minute. The splitting algorithm is new index granularity G' = G / number of fields. The number of fields is determined by the number of frequently filtered fields in the query request, and the minimum granularity is not less than 1KB.

[0037] Step 3: For HDD nodes, the trigger condition for merging is F < 5 times / minute, and the merging algorithm is new index granularity G'' = min (G × 4, 16MB). During merging, adjacent index blocks are merged according to the principle of physical address continuation.

[0038] Step 4: Output index split / merge instructions, including target granularity, involved index IDs, and execution time window, avoiding peak business periods, i.e., periods when the query frequency is less than 10% of the daily average.

[0039] Shared cache interaction logic: The DPU and the host share a cache using a key-value pair storage structure, where the key is the index ID + data fragment ID, and the value is the physical address and data digest;

[0040] The cache hit determination algorithm is "first match the key hash value, then check the data digest consistency". The cache expiration time is dynamically set according to the query frequency × 0.1 seconds, with a minimum of 10 seconds and a maximum of 360 seconds.

[0041] The daily average query frequency statistics period is natural days, and the execution time window duration is set to 30 minutes. The query frequency within the window must be ≤ daily average query frequency × 10%. If there is no window that meets the conditions for three consecutive natural days, the execution will be carried out by default from 2:00 to 2:30 am to avoid splitting or merging operations from affecting query performance during peak business hours.

[0042] The key hash value is calculated using the SHA-256 algorithm, and the first 64 bits of the hash value are used as the cache key index. Hash collision handling adopts the chaining method, and the threshold for the length of the collision chain is set to 8. If the threshold is exceeded, the cache item is triggered for eviction. The cache item with the earliest access time in the collision chain is evicted first to ensure the access efficiency of the shared cache.

[0043] Preferably, the criticality adaptation stage is based on the criticality classification of the five-dimensional intelligent sharding stage and the index splitting and merging execution instructions of the index hardware matching stage. Combined with the real-time utilization rate of HDD cluster IO and the communication parameters between DPU and host, an asynchronous update and batch version vector mechanism is adopted for non-critical data. Each data shard is associated with a 32-bit version number. When the HDD cluster IO utilization rate exceeds 70%, the synchronization period is extended and adjusted so that every 32 data items share one 32-bit version number.

[0044] For critical data, the DPU and host work together using the Raft protocol. The DPU completes the voting phase communication via PCIe direct connection, while the host is used for log persistence. When archived data updates the log, the DPU marks high-frequency modification items according to data type. The rollback operation only traverses the index entries corresponding to the field and controls the rollback time, while retaining the update log.

[0045] The final result is a data criticality and consistency policy mapping table, index rollback rules, and DPU and host communication configuration.

[0046] Version number update algorithm:

[0047] The 32-bit version number consists of the following: the high 16 bits of the master node update count = the previous count + 1, which increments every time data is written to the master node; and the low 16 bits of the replica node synchronization count = the number of replicas that have completed synchronization. Version number synchronization uses an "incremental synchronization algorithm," which transmits only the changed bits of the version number instead of the complete 32-bit data. The synchronization trigger condition is that the replica node synchronization count < the master node update count - 1.

[0048] Synchronization period adjustment algorithm:

[0049] Assuming the HDD cluster IO utilization is U, and the initial synchronization period T0 = 10 seconds, the adjustment formula is:

[0050] T=min (T0×(1+(U-70%)×2), 30 seconds)

[0051] When U≤70%, restore T0=10 seconds;

[0052] The data sharing version number threshold is dynamically calculated using floor((U-50%)×16+16), where U is the IO utilization rate. The result is between 16 and 32 records, meaning that the higher the IO utilization rate, the more data records the shared version number will be.

[0053] The 32-bit version number update and related calculation rules are as follows:

[0054] Version number synchronization trigger conditions:

[0055]

[0056] The master node update count is the high 16 bits of the 32-bit version number, and the replica node synchronization count is the low 16 bits of the 32-bit version number.

[0057] Adjustment of non-critical data synchronization cycle:

[0058]

[0059] in =Initial synchronization period=10 seconds, U=HDD cluster IO utilization;

[0060] Data sharing version number threshold:

[0061]

[0062] Where U = HDD cluster IO utilization, floor() is a floor function, the result is between 16 and 32 records. The higher the IO utilization, the more data records with shared version numbers, reducing synchronization overhead.

[0063] 32-bit version number overflow handling: When the master node update count reaches 65535 (maximum value of the high 16 bits), the version number is reset, and all replica nodes are synchronized to clear the version number to zero and record the reset log; DPU and master collaborative Raft protocol core parameters: candidate node voting timeout time = 100ms, master node heartbeat interval = 20ms, log synchronization batch size = 128KB, voting communication timeout retry count = 2 times, to ensure the efficient operation of the critical data consistency protocol.

[0064] Preferably, the linkage adjustment phase is based on the query frequency data of the index hardware matching phase, the data criticality and consistency strategy correspondence table of the criticality adaptation phase, and the sharding access frequency of the five-dimensional intelligent sharding phase, combined with the real-time utilization rate of SSD and NVMe node IO, to set access frequency level standards; any of the following conditions are used to determine strong correlation high-frequency access: the ratio of index query frequency to data access frequency > 2, the index query frequency exceeds 50 times / minute for 3 consecutive periods, and the data access frequency exceeds 30 times / minute for 2 consecutive periods;

[0065] For strongly correlated, frequently accessed data, the consistency strategy of the criticality adaptation phase is migrated to the NVMe node, and the corresponding index metadata is written to the DPU local memory; the SSD IO utilization is monitored in real time, and when it exceeds 80%, the record-level index is merged and the node shard allocation weight is reduced, and when the IO utilization is <60%, the shard receiving priority is increased.

[0066] The final solution includes a data and index access frequency level migration plan and hardware load adjustment instructions.

[0067] Preferably, the heterogeneous node balancing phase is based on the data and index access frequency level migration scheme of the linkage adjustment phase and the business SLA requirements of the five-dimensional intelligent sharding phase. It combines the real-time load of each node and the rack deployment topology to establish a load assessment matrix that includes the SLA compliance rate. The load threshold is set according to the hardware type: 90% for NVMe nodes, 80% for SSD nodes, and 70% for HDD nodes. When the SLA compliance rate of high-priority business of a node is <99%, sharding migration is triggered based on the access frequency level migration scheme of the linkage adjustment phase.

[0068] The primary and backup replicas are deployed in three different racks to ensure that the hardware configuration of the backup replica node is consistent with that of the primary replica node and the difference in PCIe link bandwidth is ≤10%. When the load of high-frequency access shards accounts for more than 40% of the total load of the node, it is split into 2-4 sub-shards, and the index metadata is synchronously copied and then migrated to idle nodes of the same type.

[0069] The final sharding and node load balancing scheme and replica deployment topology are thus formed.

[0070] Load evaluation matrix algorithm:

[0071] Load assessment value Load = 0.2 × SLA compliance rate + 0.3 × CPU utilization + 0.3 × IO utilization + 0.1 × memory utilization + 0.1 × network bandwidth utilization. All indicators are normalized to the range [0, 1].

[0072] The conditions for triggering shard migration are Load > hardware load threshold and SLA compliance rate < 99%. Migration priority is sorted by (Load - hardware load threshold) × 100, with higher values ​​indicating higher priority.

[0073] High-frequency sharding algorithm:

[0074] Let R be the proportion of high-frequency access shard load to the total node load, and K be the number of shards calculated as K=ceil(R / 40%), where K∈[2,4]. When splitting, the data hash sharding algorithm is used, and the data primary key hash value is allocated to K sub-shards modulo K. Index metadata synchronization adopts a combination of full replication and incremental synchronization, and the replication time is controlled within shard size × 0.1ms / GB.

[0075] Preferably, the query collaborative acceleration stage is based on the sharding and node final balancing scheme and replica deployment topology map of the heterogeneous node balancing stage, combined with the corresponding lookup table and cache parameters of each node in the index hardware matching stage. After receiving the query request, the DPU filters HDD nodes with IO utilization exceeding 80% through the index metadata cache and hardware load table, selects 3 candidate nodes for ping testing, and selects the node with lower latency as the query node; when querying across shards, the corresponding thread is started according to the number of racks distributed in the shards to schedule the query of each shard in parallel.

[0076] Establish a three-level caching system, including DPU cache, node cache, and cluster shared cache; when shared cache data expires, the DPU triggers the node cache to synchronize with the shared cache, and adjusts the expiration time according to the size of the query result;

[0077] After the query results are merged on the DPU side according to the query conditions, they are returned to the application in pages. At the same time, a cache update log is generated and fed back to the load prediction module in the five-dimensional intelligent sharding stage.

[0078] Low-latency node selection algorithm:

[0079] A weighted scoring model was used, with the following metrics selected: IO utilization (weight 0.4), ping latency (weight 0.3), CPU load (weight 0.2), and cache hit rate (weight 0.1). The scoring formula is as follows:

[0080] S=0.4×(1-U / 100)+0.3×(1-L / 100ms)+0.2×(1-CPU / 100)+0.1×H

[0081] Where U is I / O utilization, L is ping latency, CPU is CPU utilization, and H is cache hit rate.

[0082] The filtering process is as follows: First, filter nodes with U > 80%, then calculate the S value for the remaining nodes, take the top 3 as candidate nodes, and finally select the node with the highest S value and L ≤ 5ms as the query point.

[0083] Cross-shard parallel scheduling algorithm:

[0084] Suppose there are N racks distributed in a sharded manner, and start N independent threads. The thread ID corresponds one-to-one with the rack ID. The thread scheduling adopts a load balancing algorithm, and CPU time slices are allocated according to the number of shards of each rack × the average query time. Data communication between threads is implemented through DPU shared memory, and the communication latency is controlled within N×5μs.

[0085] The beneficial effects of this invention are as follows:

[0086] 1. This invention establishes a three-tiered storage and indexing system through a DPU collaborative architecture, comprising DPU local memory, SSD and NVMe nodes, and HDD clusters. It deeply integrates DPU computing power into index processing. For example, when an NVMe node is adapted to a numeric prefix tree index, 30% of the DPU computing cores are automatically allocated to handle prefix matching. Simultaneously, it establishes a correspondence rule between query types and index structures, enabling sparse B+-trees for range queries and FST inverted indexes for full-text search. Furthermore, it utilizes dedicated IndexCompress and IndexMigrate commands to achieve synchronous migration of data and indexes and metadata compression. This allows frequently accessed data and corresponding indexes to be precisely deployed on high-performance hardware, reducing latency for high-frequency business access and adapting to diverse business needs in mixed load scenarios.

[0087] 2. This invention uses a dynamic weighting rule of a five-dimensional evaluation matrix, combined with hardware health coefficients and business priorities, to flexibly adjust the weight ratio of each dimension and intelligently divide high-frequency and low-frequency access shards. At the same time, it designs a differentiated consistency mechanism for data of different criticality. Critical data adopts the Raft protocol in collaboration between DPU and host, while non-critical data adopts asynchronous update and batch version vector mechanism. Furthermore, through linkage adjustment and heterogeneous node balancing, the load is dynamically adjusted to avoid the limitations of static strategies, continuously correct load prediction parameters, improve SLA compliance rate, and fully leverage the performance potential of heterogeneous hardware.

[0088] 3. In the query collaboration acceleration stage, this invention relies on a three-level caching system, low-latency node selection, and cross-shard parallel scheduling to optimize query response. For example, during cross-shard queries, corresponding threads are started for parallel processing based on the number of racks. Simultaneously, the index structure is dynamically adjusted according to hardware characteristics: SSDs are split into field-level indexes during high concurrency, while HDDs are merged into superblock indexes during low access, reducing the number of IO operations. Regarding system stability, multi-rack replica deployment is used during the heterogeneous node balancing phase to ensure consistent hardware configuration between primary and backup nodes and a PCIe link bandwidth difference of ≤10%. When high-frequency shard load is too high, it is split into 2-4 sub-shards for migration. Critical data rollback only traverses the index of frequently modified items, controlling rollback time. This effectively addresses high-concurrency queries and hardware failures, ensuring stable operation of OLTP and OLAP services under mixed loads and improving overall system availability. Attached Figure Description

[0089] Figure 1 This is a flowchart illustrating the overall optimization process of distributed big data storage and indexing in this invention.

[0090] Figure 2 This is the core flowchart of the five-dimensional intelligent fragmentation of the present invention;

[0091] Figure 3 This is a flowchart of the query collaboration acceleration process of the present invention. Detailed Implementation

[0092] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0093] like Figures 1 to 3As shown, this embodiment of the invention provides a method for efficient storage and indexing optimization of distributed big data, including a five-dimensional intelligent sharding stage, a DPU collaborative architecture stage, an index hardware matching stage, a keyness adaptation stage, a linkage adjustment stage, a heterogeneous node balancing stage, and a query collaborative acceleration stage. The specific implementation of each stage is as follows:

[0094] The five-dimensional intelligent sharding stage establishes a five-dimensional evaluation matrix based on raw data, node hardware parameters, 14-day historical access logs, real-time hardware health data, and service level agreement (SLA) requirements.

[0095] The raw data includes the primary key distribution and field length of structured data, as well as the semantic density and file format of unstructured data; the node hardware parameters include the number of input / output operations per second (IOPS) and bandwidth of the storage medium, and the memory bandwidth and number of computing cores of the data processing unit (DPU); the real-time hardware health data includes CPU temperature, disk bad sector rate, and PCIe link stability; the specific service level agreement (SLA) requirements are ≤1ms for financial transactions, ≤10ms for general queries, and ≤100ms for log storage;

[0096] The weights are allocated using a preset dynamic weighting rule. When the hardware health coefficient is <0.8, the hardware capability weight is set to 50%. When the proportion of high-priority business is >60%, the SLA weight is set to 40%. For other scenarios, the weights are allocated as follows: hardware capability 35%, load prediction 30%, SLA 15%, data characteristics 10%, and hardware health 10%. This results in the division into high-frequency access fragments of 8-16MB and low-frequency access fragments of 4-8GB.

[0097] The specific quantitative indicators for each dimension of the five-dimensional evaluation matrix are as follows:

[0098] 1. Original data dimension: Quantitative value = data structuring coefficient × 0.6 + data access popularity coefficient × 0.4; where the data structuring coefficient is 1.0 for structured data, 0.7 for semi-structured data, and 0.3 for unstructured data; the data access popularity coefficient is the average access frequency in the past 7 days / the average access frequency in the past 30 days, with a value range of [0.1, 2.0].

[0099] 2. Node hardware parameter dimension: Quantified value = (Number of input / output operations per second / Average number of input / output operations per second of the cluster) × 0.3 + (Bandwidth / Average bandwidth of the cluster) × 0.3 + (Number of data processing unit computing cores / Maximum number of data processing unit computing cores of the cluster) × 0.2 + (Memory bandwidth / Average memory bandwidth of the cluster) × 0.2, with a value range of [0.1, 1.0];

[0100] 3. Historical access log dimension: Quantitative value = 90th percentile of access frequency in the past 14 days / maximum value of 90th percentile of cluster access frequency × 0.7 + variance of access duration, normalized to [0, 0.3], with a value range of [0.1, 1.0].

[0101] 4. Hardware real-time health data dimension: namely, hardware health coefficient, with a value range of [0,1], and the calculation method is clearly defined;

[0102] 5. Service Level Agreement (SLA) Requirements Dimension: Quantitative value = reciprocal of the SLA latency threshold × 1000. For example: Financial transaction SLA ≤ 1 millisecond, quantitative value = 1000; normal query ≤ 10 milliseconds, quantitative value = 100; log storage ≤ 100 milliseconds, quantitative value = 10. The value range is [10, 1000].

[0103] After quantization of each dimension, the data is weighted and summed according to dynamic weighting rules. Data with a threshold of ≥0.6 is determined to be high-frequency access fragments, and data with a threshold of <0.6 is determined to be low-frequency access fragments.

[0104] High-frequency access sharding adapts to Online Transaction Processing (OLTP) for high-frequency small queries; low-frequency access sharding adapts to Online Analytical Processing (OLAP) for low-frequency large queries.

[0105] After the sharding migration, the DPU collects migration time and service latency data, which is then fed back to the load prediction module to correct the 12-hour prediction parameters and dynamically adjust the high-frequency access sharding threshold to avoid single-point overload.

[0106] The final result is a list of high-frequency access shards and low-frequency access shards, an initial shard and node allocation scheme, and load prediction correction parameters.

[0107] The hardware health coefficient is calculated using a weighted summation method, as shown in the following formula:

[0108] CPU health score

[0109] Disk health score = 1 - disk bad sector rate

[0110] PCIe Link Health Score = 1 - (Link Error Rate)

[0111] Memory health score

[0112] Hardware health coefficient CPU health score Disk health score PCIe link health score Memory health score;

[0113] All sub-scores are normalized to [0, 1]. In the CPU health score calculation, the normal temperature threshold is 60°C by default, and the warning temperature threshold is 90°C by default. In the PCIe link health score calculation, the link error rate must be ≤0.1 to avoid negative scores.

[0114] The DPU collaborative architecture phase, based on the sharding list and initial allocation scheme from the five-dimensional intelligent sharding phase, and combined with parameters of each storage level including DPU local memory capacity, SSD performance, NVMe node performance, and HDD cluster capacity, establishes a three-level storage and indexing architecture consisting of DPU local memory, SSD and NVMe nodes, and HDD cluster, and simultaneously deploys the DPU collaborative control plane. The control plane has built-in hardware adaptation rules, such as automatically allocating 30% of the DPU computing cores to handle prefix matching when NVMe nodes are adapted to the digital prefix tree index, and synchronously triggering index metadata updates during cross-level migration, such as automatically splitting the block-level index into record-level indexes when migrating from HDD to SSD.

[0115] Tier 3 storage capacity allocation reference:

[0116] DPU local memory capacity ≥ 0.5% of total cluster storage capacity, minimum 8GB, maximum 64GB; total capacity of SSD and NVMe nodes ≥ 2 times the total capacity of high-frequency access shards; total capacity of HDD cluster ≥ 1.5 times the total capacity of low-frequency access shards + archived data capacity.

[0117] Recommended I / O bandwidth allocation for each storage tier:

[0118] NVMe node IO bandwidth ≥1GB / s, SSD node IO bandwidth ≥500MB / s, HDD cluster IO bandwidth ≥200MB / s, ensuring that the performance of each storage tier matches the business access requirements.

[0119] Simultaneously, dedicated commands IndexCompress and IndexMigrate were developed. IndexCompress compresses index metadata in segments according to value range using the RoaringBitmap algorithm, while IndexMigrate enables synchronous migration of data and index, keeping the processing time of index metadata within 0.8ms.

[0120] IndexCompress segmentation rules based on value range:

[0121] Integer indices are segmented into intervals of 65536, such as [0-65535), [65535-131071), etc.; string indices are segmented into intervals of ASCII code of the first character, with each interval consisting of 32 ASCII codes.

[0122] IndexMigrate time control logic: A parallel migration strategy is adopted, and index metadata is split into 1MB blocks for parallel transmission. The timeout threshold for a single block transmission is set to 0.1ms. If the timeout occurs, the DPU local cache is temporarily stored and retried after the link is idle. The maximum number of retries is 3. The total timeout = parallel transmission timeout + retry timeout ≤ 0.8ms, ensuring the timeliness of index and data synchronous migration.

[0123] Exception handling for index migration specific instructions:

[0124] If the index metadata migration times out after three retries, a degradation strategy is triggered: the index metadata is temporarily stored in the temporary cache of the solid-state drive node, and the migration is re-initiated after the data processing unit detects that the link is idle. At the same time, an exception log is recorded, including the migration failure time, index identifier, link status, and an alarm is triggered to notify the operation and maintenance personnel.

[0125] 2. Handling faults in shard migration nodes:

[0126] If the target node fails during the shard migration process, the migration is immediately terminated, and a backup node of the same type is selected to restart the migration. The migration progress resumes from the completed part. The breakpoint resume mechanism is based on the index migration special instruction: the transmission status is recorded in 1-megabyte blocks, and the untransmitted blocks are retransmitted.

[0127] 3. Handling of anomalies in critical data consistency protocols:

[0128] If a candidate node fails to receive enough votes within the voting timeout period, the data processing unit automatically extends the voting timeout to 200 milliseconds and retryes once. If it still fails, a new master node election is triggered. During the election, key data is written to the local memory of the data processing unit and synchronized to the new master node log after the election is completed.

[0129] The DPU monitors the storage utilization of each level in real time. When it exceeds 85%, it triggers low-frequency access data archiving and index dimensionality reduction operations. The HDD low-frequency access data index is simplified from a 256-bit secure hash algorithm (SHA-256) digest to a 32-bit cyclic redundancy check (CRC32) to balance the load of each level.

[0130] The final output includes a three-level storage and index architecture configuration table, DPU control plane scheduling rules, and index metadata processing standards.

[0131] The index hardware matching stage is based on the configuration table and DPU control plane scheduling rules of the DPU collaborative architecture stage. It combines query request samples (including range queries, prefix matching, and full-text search types) and the index support capabilities of each hardware node, such as SSD supporting field-level indexes and HDD supporting block-level indexes, to establish a rule for the correspondence between query types and index structures. Range queries use sparse B+-trees, prefix matching queries use numeric or character prefix trees, and full-text search uses FST inverted indexes. After receiving a query request, the DPU parses the request header within 100μs, extracts the query type and field range, and matches the corresponding index structure according to the hardware adaptation rules.

[0132] For SSD nodes, when IO utilization is less than 60% and query frequency exceeds 50 times / minute, record-level indexes are split into field-level indexes. For example, order data is split by user ID and timestamp. For HDD nodes, when access frequency is less than 5 times / minute, adjacent 4MB block-level indexes are merged into a 16MB superblock index to reduce the number of IO operations. At the same time, DPUs share key values ​​and physical address caches with the host to reduce interaction resource consumption.

[0133] This ultimately results in a table mapping queries, indexes, and hardware, as well as instructions for splitting and merging indexes.

[0134] The criticality adaptation stage is based on the criticality classification of the five-dimensional intelligent sharding stage, the index splitting and merging execution instructions of the index hardware matching stage, and combined with the real-time utilization of HDD cluster IO and the communication parameters between DPU and host (PCIe link latency).

[0135] Key data includes financial transactions and user balances, while non-critical data includes system logs and monitoring metrics. For non-critical data, an asynchronous update and batch version vector mechanism is used, with each data shard associated with a 32-bit version number. The high 16 bits are used for the master node update count, and the low 16 bits are used for the replica node synchronization count. When the HDD cluster IO utilization exceeds 70%, the synchronization period is extended to 30 seconds, and the 32-bit version number is shared by every 32 data items.

[0136] For critical data, the DPU and host work together using the Raft protocol. The DPU completes the voting phase communication via PCIe direct connection with latency controlled within 1ms, while the host is used for log persistence. When archived data updates the log, the DPU marks high-frequency modification items according to data type, such as the user balance field. The rollback operation only traverses the index entries corresponding to such fields and controls the rollback time to within 2 minutes, retaining the update log for 7 days, thus supporting complete rollback in failure scenarios.

[0137] The final result is a data criticality and consistency policy mapping table, index rollback rules, and DPU and host communication configuration.

[0138] High-frequency modification item marking rules:

[0139] The data field is judged based on the frequency of modification per unit time, with the unit time set to 5 minutes. Fields with a modification frequency of ≥10 times / 5 minutes are marked as high-frequency modification items. If the modification frequency of a field is less than 5 times / 5 minutes for 3 consecutive unit times, the high-frequency mark is removed to ensure that the rollback operation only targets fields that are truly frequently modified, thus controlling the rollback time.

[0140] The linkage adjustment phase is based on the query frequency data of the index hardware matching phase, the data criticality and consistency strategy correspondence table of the criticality adaptation phase, and the sharding access frequency of the five-dimensional intelligent sharding phase, combined with the real-time utilization rate of SSD / NVMe node IO, to set access frequency level standards; any of the following conditions are used to determine strong correlation high-frequency access: the ratio of index query frequency to data access frequency > 2, the index query frequency exceeds 50 times / minute for 3 consecutive periods, and the data access frequency exceeds 30 times / minute for 2 consecutive periods.

[0141] The default period for access frequency levels is 5 minutes, which can be configured according to the business type. The minimum period for OLTP business is 1 minute, and the maximum period for OLAP business is 10 minutes. The step size for adjusting the weight of SSD node shard allocation is 0.1, with an interval of 30 seconds between each adjustment. The gradient is adjusted to the target weight. For example, it takes 4 adjustments to reduce the weight from 1.0 to 0.6. This avoids sudden weight changes that may cause load fluctuations and ensures stable load on storage nodes.

[0142] For strongly correlated, frequently accessed data, a consistency strategy based on criticality adaptation is used to migrate the data to NVMe nodes, and the corresponding index metadata is written to the DPU's local memory, with 30% of memory space reserved. The SSD's IO utilization is monitored in real time. When it exceeds 80%, record-level indexes are merged and the node shard allocation weight is reduced from 1.0 to 0.6 to reduce the access of new shards. When the IO utilization is <60%, the shard receiving priority is increased, thereby optimizing resource utilization.

[0143] The final solution includes a data and index access frequency level migration plan and hardware load adjustment instructions.

[0144] The heterogeneous node balancing phase is based on the data and index access frequency level migration scheme of the linkage adjustment phase and the business SLA requirements of the five-dimensional intelligent sharding phase. It combines the real-time load of each node and the rack deployment topology to establish a load assessment matrix that includes the SLA compliance rate (weight 20%). Load thresholds are set according to hardware type: 90% for NVMe nodes, 80% for SSD nodes, and 70% for HDD nodes. When the SLA compliance rate of high-priority business of a node is <99%, sharding migration is triggered based on the access frequency level migration scheme of the linkage adjustment phase.

[0145] Real-time node load includes CPU utilization, memory utilization, storage I / O utilization, and network bandwidth utilization.

[0146] The primary and backup replicas are deployed on three different racks to ensure that the hardware configuration (number of CPU cores, memory capacity) of the backup replica node is consistent with that of the primary replica node, the difference in PCIe link bandwidth is ≤10%, and the synchronization latency is controlled within 1.5ms. When the high-frequency access shard load accounts for more than 40% of the total node load, it is split into 2-4 sub-shards, and after synchronously replicating the index metadata, it is migrated to idle nodes of the same type.

[0147] The final sharding and node load balancing scheme and replica deployment topology are thus formed.

[0148] The query collaborative acceleration phase is based on the sharding and node final balancing scheme and replica deployment topology of the heterogeneous node balancing phase, combined with the corresponding lookup table of the index hardware matching phase and the cache parameters of each node (DPU memory capacity, 20% cache space of node memory, cluster shared memory size). After receiving the query request, the DPU filters HDD nodes with IO utilization exceeding 80% through the index metadata cache and hardware load table, selects 3 candidate nodes for ping testing, and selects the node with lower latency as the query node. When querying across shards, the corresponding thread is started according to the number of racks distributed in the shard distribution (1 thread per rack) to schedule the query of each shard in parallel.

[0149] A three-tiered caching system is established, with each cache configured as follows: The DPU cache stores index metadata, with its validity period set according to update frequency: 5 minutes for updates >1 time / hour and 12 hours for updates <1 time / day; the node cache stores frequently accessed data, employing a Least Recently Used (LRU) eviction policy; the cluster shared cache stores frequently accessed query results, deployed in the DPU cluster's shared memory, with its validity period set according to query result size: 1 hour for results >100MB and 12 hours for results <10MB; when shared cache data expires, the DPU triggers synchronization between the node cache and the shared cache, with the time taken controlled within 50ms.

[0150] The query results are merged on the DPU side according to the query conditions (such as sorting and filtering) and returned to the application in pages. At the same time, a cache update log is generated and fed back to the load prediction module in the five-dimensional intelligent sharding stage for iterative optimization of the sharding strategy.

[0151] The specific implementation steps of the load prediction module are as follows:

[0152] Step 1: Data Acquisition and Preprocessing: The DPU collects migration time (in milliseconds) and business latency (in milliseconds) at 1 second / time. The average migration time t is calculated using a sliding window (window size = 30 seconds). avg d, the P95 quantile of business delayp95 Remove t avg >50ms or d_ p95 Abnormal data window >20ms;

[0153] Step 2: Extract 3 types of features:

[0154] ①Time series characteristics: t avg The 10-minute trend slope and the 5-minute rolling variance of d_p95;

[0155] ② Business association characteristics: SLA level of corresponding shards (high=3, medium=2, low=1), data characteristic coefficient, structured data=1.5, unstructured data=0.8;

[0156] ③ Hardware status characteristics: node hardware health coefficient, IO utilization;

[0157] Specific calculation method for features:

[0158] 1. Temporal characteristics:

[0159] The 10-minute trend slope of the average migration time = (average migration time in the 10th minute - average migration time in the 1st minute) / 9, in ms / min;

[0160] The 5-minute rolling variance of the 95th percentile of business latency = Σ(95th percentile of business latency per minute - mean of the 95th percentile of business latency over 5 minutes)² / 5 (calculated over a continuous 5 minutes, unit: ms²).

[0161] 2. Business-related characteristics:

[0162] Data characteristic coefficient = Structured data (including JSON, CSV and other data with directly parsable fields) = 1.5, Unstructured data (including video, audio, and binary files without fixed fields) = 0.8;

[0163] 3. Hardware status characteristics:

[0164] The calculation method for node hardware health coefficient is the same as the hardware real-time health data dimension in the five-dimensional evaluation matrix.

[0165] Input / output utilization = current number of input / output read / write operations / maximum number of input / output read / write operations of the node, with a value range of [0,1].

[0166] Step 3: Predictive model training and deployment: The XGBoost gradient boosting tree model is used. The training set is preprocessed data from the past 30 days, and the label is the peak access frequency of each shard in the next hour.

[0167] Model hyperparameter settings: learning rate = 0.1, tree depth = 5, number of iterations = 100; update model parameters every 24 hours during deployment to ensure prediction accuracy;

[0168] Step 4: Parameter Correction and Threshold Adjustment: When the corrected high-frequency access fragmentation threshold coefficient K≥1.5, the high-frequency fragmentation base size is reduced from 16MB to 8MB; when K≤0.8, the high-frequency fragmentation base size is increased from 8MB to 16MB; when the node load carrying coefficient β≤0.6, the hardware capability weight of the node is reduced to 60% of the original weight to avoid overload.

[0169] The explanations of letters or abbreviations in the document are as follows:

[0170] DPU: Data Processing Unit;

[0171] SSD: short for Solid State Drive, refers to a solid-state drive;

[0172] NVMe: short for Non-Volatile Memory Express, refers to the non-volatile memory host controller interface specification;

[0173] HDD: Full name Hard Disk Drive, refers to mechanical hard drive;

[0174] B+-tree: short for B-plus tree, referring to a B+ tree;

[0175] FST: short for Finite State Transducer;

[0176] SHA-256: short for Secure Hash Algorithm 256-bit, refers to a 256-bit secure hash algorithm;

[0177] CRC32: Short for Cyclic Redundancy Check 32-bit, which refers to a 32-bit cyclic redundancy check.

[0178] RoaringBitmap: Full name Roaring Bitmap, refers to the Roar bitmap algorithm;

[0179] Raft: short for Raft Consensus Algorithm, refers to the Raft consensus protocol.

[0180] PCIe: short for Peripheral Component Interconnect Express, refers to high-speed peripheral component interconnection;

[0181] RDMA: short for Remote Direct Memory Access;

[0182] SLA: Service Level Agreement;

[0183] IOPS: Input / Output Operations Per Second, refers to the number of input / output operations per second;

[0184] OLTP: Online Transaction Processing;

[0185] OLAP: Online Analytical Processing;

[0186] XGBoost: short for eXtreme Gradient Boosting, refers to extreme gradient boosting trees;

[0187] LRU stands for Least Recently Used, referring to the least recently used data.

[0188] PI: full name Proportional-Integral;

[0189] 3σ: 3 Standard Deviations, referring to 3 times the standard deviation;

[0190] P95: 95th Percentile.

[0191] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0192] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for optimizing efficient storage and indexing of distributed big data, characterized in that, This includes the five-dimensional intelligent sharding stage, the DPU collaborative architecture stage, the index hardware matching stage, the criticality adaptation stage, the linkage adjustment stage, the heterogeneous node balancing stage, and the query collaborative acceleration stage. The five-dimensional intelligent sharding stage establishes a five-dimensional evaluation matrix based on raw data, node hardware parameters, historical access logs, real-time hardware health data, and business service level agreement requirements. It allocates weights according to preset dynamic weight rules, divides high-frequency access shards into low-frequency access shards, collects migration feedback data through DPU to correct load prediction parameters, and outputs a sharding list and initial allocation scheme. The DPU collaborative architecture phase combines the sharding list and parameters of each storage level to build a three-level storage and indexing architecture of DPU local memory, SSD and NVMe nodes, and HDD cluster, as well as a DPU collaborative control plane. It configures instructions to optimize index processing, monitors storage load to trigger data archiving and index dimensionality reduction, and outputs architecture configuration and scheduling rules. During the index hardware matching phase, rules are established to correspond query types to index structures. Query requests are parsed to match indexes and dynamically split or merged according to the status of hardware nodes. Interactive resource consumption is reduced through shared cache. During the criticality adaptation phase, different consistency mechanisms are designed based on the differences in criticality of sharded data, the log archiving and index rollback process is optimized, and a data criticality and consistency mapping table and communication configuration are output. During the linkage adjustment phase, access frequency level judgment criteria are set, strongly correlated high-frequency access data is migrated to the corresponding nodes, and hardware load weight is dynamically adjusted. During the heterogeneous node balancing phase, load thresholds are set according to hardware type, and shard migration is triggered based on the business SLA compliance rate to optimize replica deployment topology. During the query collaboration acceleration phase, after receiving a query request, low-latency nodes are selected as query points, a three-level caching system is established, query results are merged and returned to the application, and cache update logs are fed back.

2. The distributed big data high-efficiency storage and indexing optimization method according to claim 1, characterized in that, The dynamic weighting rules in the five-dimensional intelligent sharding stage adjust the weight ratio of each dimension according to different hardware health status and business priority scenarios. After dividing high-frequency access shards into low-frequency access shards, migration feedback data, including migration time and business latency, is collected by the DPU and fed back to the load prediction module to correct the load prediction parameters. Based on the corrected load prediction parameters, the high-frequency access sharding threshold is dynamically adjusted, and finally a sharding list, initial shard and node allocation scheme, and load prediction correction parameters are formed.

3. The distributed big data high-efficiency storage and indexing optimization method according to claim 2, characterized in that, The three-level storage and index architecture in the DPU collaborative architecture stage is built based on the hardware parameters of each storage level. The DPU collaborative control plane has built-in hardware adaptation rules, allocates DPU computing resources for different index types, and develops dedicated instructions for IndexCompress and IndexMigrate. The IndexCompress instruction compresses index metadata, while the IndexMigrate instruction enables synchronous migration of data and indexes and controls the processing time of index metadata. By monitoring the utilization of each storage level in real time, low-frequency data archiving and index dimensionality reduction operations are triggered, generating a three-level storage and index architecture configuration table, DPU control plane scheduling rules, and index metadata processing standards.

4. The distributed big data high-efficiency storage and indexing optimization method according to claim 3, characterized in that, The corresponding rules for the index hardware matching stage include enabling sparse B+- trees for range queries, enabling numeric or character prefix trees for prefix matching queries, and enabling FST inverted indexes for full-text search. After parsing the query request, the DPU matches the index structure according to the hardware adaptation rules. Based on IO utilization and query frequency, it dynamically splits or merges the index for each hardware node. By sharing cached data with the host, the DPU reduces the consumption of interactive resources and forms a lookup table of query, index and hardware correspondence, as well as instructions for index splitting and merging.

5. The distributed big data high-efficiency storage and indexing optimization method according to claim 4, characterized in that, The criticality adaptation phase employs asynchronous updates and batch version vector mechanisms for non-critical data, adjusting the synchronization cycle and version number sharing strategy based on HDD cluster IO utilization; for critical data, a Raft protocol is used in collaboration between the DPU and the host, with the DPU responsible for communication during the voting phase and the host responsible for log persistence. When archiving data updates the log, the DPU marks frequently modified items according to data type; the rollback operation only traverses the index entries corresponding to the frequently modified items and controls the rollback time, ultimately forming a data criticality and consistency policy correspondence table, index rollback rules, and communication configuration between the DPU and the host.

6. The distributed big data high-efficiency storage and indexing optimization method according to claim 5, characterized in that, The access frequency level standard in the linkage adjustment phase is based on query frequency data, data criticality classification, and hardware IO real-time utilization setting. Strongly correlated high-frequency access data is migrated to NVMe nodes, and the corresponding index metadata is written to the DPU local memory. The index structure and node shard allocation weight are dynamically adjusted according to the SSD IO utilization to form a data and index access frequency level migration scheme and hardware load adjustment instructions.

7. The distributed big data high-efficiency storage and indexing optimization method according to claim 6, characterized in that, The heterogeneous node balancing phase sets load thresholds according to hardware type, triggers shard migration based on the SLA compliance rate of high-priority services, deploys primary and backup replicas on different racks, and ensures that the hardware configuration of the backup replica nodes is consistent with that of the primary replica nodes and that the difference in link bandwidth is within a preset threshold. When the load on a high-frequency access shard accounts for too high a proportion of the total load on a node, it is split into 2-4 sub-shards. After synchronously replicating the index metadata, the shards are migrated to idle nodes of the same type, forming the final shard and node load balancing scheme and replica deployment topology.

8. The distributed big data high-efficiency storage and indexing optimization method according to claim 7, characterized in that, During the query collaboration acceleration phase, after receiving a query request, the DPU filters high-load nodes with IO utilization exceeding a preset threshold through the index metadata cache and hardware load table, and selects low-latency nodes as query points. Parallel scheduling is initiated during cross-shard queries, and a three-level caching system is established. After merging the query results according to the query conditions, the results are returned to the application in pages, a cache update log is generated, and the load prediction module in the five-dimensional intelligent sharding stage is fed back.

Images