A dynamic expansion and contraction method for replica storage based on access heat and storage bandwidth

Abstract

This invention discloses a dynamic scaling method for replica storage based on access popularity and storage bandwidth, belonging to the field of computer data storage. It aims to solve the problems of imbalance between storage system resource utilization and service reliability, and insufficient bandwidth adaptation. The method includes: First, collecting hardware parameters of storage nodes to calculate the actual bandwidth and determine the bandwidth thresholds corresponding to hot, warm, and cold blocks; second, collecting and preprocessing data block access logs, and statistically classifying them based on popularity; third, combining system availability targets, screening candidate storage nodes and calculating the minimum number of replicas for hot and warm blocks and the erasure coding storage requirements for cold blocks; then, performing replica scaling and cold block mode conversion to ensure that the total bandwidth of storage nodes meets the target; finally, verifying bandwidth and popularity after the operation to avoid adaptation failures. This invention is applicable to scenarios of storage resource and access performance optimization, providing a bandwidth adaptation solution, balancing storage costs and reliability, and improving system performance.

Description

Technical Field

[0001] This invention belongs to the field of computer data storage, and mainly focuses on the efficient utilization of storage resources and the coordinated guarantee of data access availability in storage systems. Background Technology

[0002] Current storage systems generally employ a fixed replica strategy, failing to adequately consider the dynamic differences in data access frequency and the heterogeneity of storage node bandwidth. This leads to high access latency and excessive storage node load for hot data due to insufficient replicas, while storing cold data with multiple replicas results in resource waste. Furthermore, replica deployment methods that ignore the differences in storage node bandwidth further weaken data availability. Existing solutions lack quantitative assessment of storage node bandwidth and a dynamic mapping mechanism of "access frequency - storage bandwidth - number of replicas," making it difficult to achieve fine-grained replica scheduling. Summary of the Invention

[0003] The purpose of this invention is to address the problems of poor adaptability, severe resource waste, and insufficient data availability associated with fixed replica strategies in existing storage systems. It provides a dynamic replica scaling method based on data access frequency and storage node bandwidth, enabling on-demand allocation and intelligent scheduling of storage resources. This invention employs a collaborative architecture of "storage node bandwidth measurement + data access frequency recording + dynamic replica scaling." By tiering storage node bandwidth and data access frequency, it dynamically matches the optimal number of replicas and deployment strategy, effectively balancing storage resource overhead, access performance, and data availability.

[0004] Technical solution of the present invention

[0005] A method for dynamically scaling up and down replica storage based on access popularity and storage bandwidth, the method comprising the following specific steps:

[0006] Step 1: The "Bandwidth Assessment Module" completes the storage bandwidth measurement of the storage node;

[0007] Step 2: The "Node Acquisition Module" works in conjunction with the "Popularity Analysis Module" to record the data access popularity.

[0008] Step 3: The "Replica Strategy Calculation Module" performs dynamic calculation of the number of replicas;

[0009] Step 4: The "Replica Expansion Execution Module" completes the dynamic expansion operation of the replicas;

[0010] Step 5: The "Replica Shrinkage and Mode Conversion Module" completes the dynamic shrinkage of replicas and the conversion of cold block storage;

[0011] Step 6: The "Bandwidth and Popularity Verification Module" completes dynamic verification of bandwidth and popularity.

[0012] Step 7: Program execution ends;

[0013] in,

[0014] Step 1, the method for the "bandwidth assessment module" to complete the storage node storage bandwidth measurement is as follows: The "bandwidth assessment module" collects the hardware performance parameters of each storage node, calculates and determines the actual storage bandwidth of each storage node; at the same time, the "bandwidth assessment module" determines the required storage bandwidth threshold for each level of data block according to the system's preset hot block, warm block, and cold block classification standards, ensuring that the total bandwidth of the storage nodes for subsequent replica deployment meets the corresponding threshold requirements, and providing basic parameters for replica deployment and reliability calculation;

[0015] Step 2, "The 'node acquisition module' works in conjunction with the 'heat analysis module' to record data access heat," is as follows: The 'node acquisition module' built into each storage node collects local data block access logs in real time and reports them to the system's 'heat analysis module' according to a preset time period. The 'heat analysis module' aggregates and deduplicates the reported logs, and counts the total number of successful reads for each data block within the current system epoch, which serves as the dynamic access heat of that data block. At the same time, the 'heat analysis module' classifies each data block into hot, warm, and cold blocks based on a preset heat threshold, providing a basis for subsequent bandwidth threshold matching and replica calculation.

[0016] Step 3, the method for the "Replica Strategy Calculation Module" to dynamically calculate the number of replicas is as follows: The "Replica Strategy Calculation Module" retrieves and defines the system's preset availability target. For hot blocks and warm blocks after classification, it first matches the storage bandwidth threshold of their corresponding level, selects candidate storage nodes from the storage node pool whose actual total bandwidth is higher than the corresponding threshold and meets the performance requirements, and then it iteratively calculates through the built-in reliability model to obtain the minimum number of replicas that meets the system availability target. For cold blocks, it first matches the storage bandwidth threshold corresponding to the cold block, selects storage nodes that meet the bandwidth requirements, determines the erasure coding storage scheme, and calculates the required number of coded blocks and the corresponding storage node requirements.

[0017] Step 4, the method by which the "Replica Expansion Execution Module" completes the dynamic expansion operation of replicas is as follows: The "Replica Expansion Execution Module" first checks the current number of replicas of hot blocks and warm blocks, the deployed storage nodes, and the total bandwidth of each deployed storage node. It compares this with the minimum number of replicas and the corresponding bandwidth threshold obtained by the "Replica Policy Calculation Module" in Step 3. If it is determined that the current number of replicas is insufficient, or the total bandwidth of some deployed storage nodes has not reached the corresponding threshold, the "Replica Expansion Execution Module" selects the optimal storage node from the candidate storage nodes to deploy the new replica. The "Replica Expansion Execution Module" ensures that the total bandwidth of the deployed storage nodes of the replicas remains higher than the corresponding threshold after expansion, thus completing the expansion operation and ensuring data access performance.

[0018] Step 5, the method for the "Replica Scaling and Mode Conversion Module" to complete replica dynamic scaling and cold block storage conversion is as follows: The "Replica Scaling and Mode Conversion Module" verifies the current replica numbers of hot blocks and warm blocks, the deployed storage nodes, and the total bandwidth of each deployed storage node, and compares with the minimum replica number and the corresponding bandwidth threshold obtained by the "Replica Policy Calculation Module" in Step 3. If it is determined that the current replica number exceeds the limit and the total bandwidth of the remaining deployed storage nodes after deleting redundant replicas can still meet the corresponding threshold requirements, then redundant replicas are deleted to complete the scaling operation and save storage resources. At the same time, the "Replica Scaling and Mode Conversion Module" verifies the current situation of cold blocks and the total bandwidth of the corresponding storage nodes to ensure that the total bandwidth of the storage nodes meets the cold block threshold requirements, and then performs the conversion operation of cold blocks to the erasure code encoding storage mode according to the number of coded blocks and the storage node requirements obtained by the "Replica Policy Calculation Module" to ensure the security and bandwidth adaptability of cold block storage.

[0019] Step 6, the method for the "Bandwidth and Heat Check Module" to complete bandwidth and heat dynamic check is as follows: After the replica expansion, scaling, and cold block conversion operations are completed, the "Bandwidth and Heat Check Module" performs a secondary check on the total bandwidth of the storage nodes corresponding to each data block to confirm that the total bandwidth of the storage nodes is continuously higher than the bandwidth threshold of the corresponding level.

[0020] Furthermore, in Step 2, the method for the preset heat threshold system to adopt a three-layer dynamic partitioning mechanism is as follows: Let the maximum access heat of all data blocks in the current system epoch be H max , then the determination condition for hot blocks is the access heat x•H max ≤H i , the determination condition for warm blocks is y•H max ≤H i <x•H max , the determination condition for cold blocks is 0≤H i <y•H max , where 0<y<x<1 is a configurable normalized heat coefficient, x is the normalized heat coefficient for hot blocks, y is the normalized heat coefficient for warm blocks, both are constants configured manually according to the business scenario requirements, and satisfy the value range of 0<y<x<; H i is the dynamic access heat of the i-th data block in the current system epoch, i is the unique identification serial number of the data block, and its value is a positive integer, that is, i = 1, 2, 3,..., n, and n is the total number of data blocks in the current system.

[0021] Furthermore, the measurement steps for storage node bandwidth specifically include: a "bandwidth evaluation module" collecting hardware performance-related indicators for each storage node, including but not limited to disk throughput, network interface speed, CPU processing load, and cache hit efficiency; assigning weights to the above indicators based on the analytic hierarchy process to form a multidimensional evaluation system; then using a multiplicative synthesis mechanism to normalize and fuse the indicators to generate a comprehensive storage bandwidth score for each storage node; finally, based on a preset scoring threshold, dividing all network storage nodes into three levels: high bandwidth, medium bandwidth, and low bandwidth, and constructing corresponding storage node pools for subsequent resource deployment matching according to data popularity levels.

[0022] Furthermore, the data access heat recording step includes: each storage node's "node acquisition module" continuously monitors the read operations of the data blocks it carries locally, records successful access events in real time, and generates access logs; the access logs are reported in batches to the "heat analysis module" according to a preset time period; after receiving logs from multiple storage nodes, the "heat analysis module" performs aggregation and deduplication processing to eliminate duplicate or redundant records; based on this, it counts the cumulative number of successful reads for each data block in the current system epoch and uses the number of reads as the dynamic access heat of the data block; subsequently, according to a preset heat classification threshold, all data blocks are divided into three levels: hot blocks, warm blocks, and cold blocks, providing input for subsequent replica strategy decisions.

[0023] Furthermore, the aforementioned dynamic calculation step for the number of replicas specifically includes: first, clarifying the overall availability target preset by the system; for data blocks divided into hot blocks or warm blocks, selecting candidate storage nodes that meet the bandwidth and performance requirements from the storage node pool that matches the heat level of the data blocks, and iteratively evaluating them based on the reliability model until the minimum number of replicas that can meet the availability target is determined; for data blocks divided into cold blocks, instead of using a multi-replica redundancy method, switching to erasure coding storage mode, calculating the required number of original data fragments and parity fragments based on the reliability requirements of cold blocks and available storage node resources, and determining the total number of coded blocks to be deployed and the corresponding number of storage nodes accordingly.

[0024] Furthermore, the aforementioned dynamic expansion, contraction, and storage mode conversion operations for replicas further include the following optimization strategies: When performing expansion, priority is given to selecting storage nodes with higher storage bandwidth, lower network latency, and geographically dispersed from other replica storage nodes from idle storage nodes to improve concurrent access capabilities and disaster recovery capabilities; when performing contraction, priority is given to deleting redundant replicas deployed on storage nodes with lower storage bandwidth, higher current load, or higher failure risk to release inefficient resources and maintain service quality; when performing the conversion operation of cold blocks from multi-replica to erasure coding mode, the following sub-steps are executed in sequence: first, all existing replicas of the cold block are aggregated to ensure data integrity; then, the data is encoded based on the selected erasure coding scheme to generate multiple coded blocks; next, the coded blocks are distributed to multiple target storage nodes that meet the bandwidth requirements of the cold block; finally, after confirming that all coded blocks are successfully persisted, the original multi-replica data is safely deleted, completing the smooth migration of storage modes.

[0025] Furthermore, during the process of dynamically expanding, shrinking, or switching to cold block storage mode, the system applies a lightweight metadata lock to the target data block and employs a two-phase commit protocol to ensure the atomicity and consistency of the operation. During the expansion phase, the replica location metadata of the data block is only updated after the new replica has been written to the target storage node and a persistent confirmation has been returned. During the shrinking phase, redundant replicas are only allowed to be deleted if the remaining number and distribution still meet availability and bandwidth requirements. During the cold block mode switching phase, the deletion operation of the original multiple replicas is delayed until all erasure coding blocks have been successfully distributed and verified, thereby avoiding data loss or service unavailability due to operation interruption or storage node failure.

[0026] The specific content involved in this invention and the meaning of the terms used are as follows:

[0027] (1) Node storage bandwidth measurement: refers to the process of comprehensively evaluating and quantifying the storage bandwidth capability and related performance of storage nodes. The core purpose is to provide a basis for node selection for replica deployment. The evaluation scope covers the node's real-time storage bandwidth, historical bandwidth stability and related hardware support capabilities.

[0028] (2) Dynamic access popularity: refers to the frequency of successful access to a data block within a specific time period, i.e., within a system epoch. It is quantified by the total number of successful reads and used to distinguish the popularity of data blocks, providing data basis for the dynamic adjustment of the number of replicas.

[0029] (3) System Era: refers to the system's preset periodic time window, which serves as the trigger period for node storage bandwidth measurement, data access heat statistics, and replica number optimization, ensuring that various evaluation and adjustment operations are carried out in an orderly and regular manner.

[0030] (4) Hot block / warm block / cold block: Data block levels are divided according to dynamic access hotness thresholds. Hot block refers to high-frequency access data block with access frequency higher than the high threshold, warm block refers to medium-frequency access data block with access frequency between the high and low thresholds, and cold block refers to low-frequency access data block with access frequency lower than the low threshold. Different storage and replication strategies are adopted for different levels of data blocks.

[0031] (5) Dynamic expansion and contraction of replicas: refers to the operation of dynamically adjusting the number of data block replicas or storage mode based on the data block heat registration and node storage bandwidth assessment results, including expansion, contraction and mode conversion of cold blocks from multi-replica to erasure coding storage.

[0032] Advantages and positive effects of the present invention:

[0033] To achieve on-demand allocation of storage resources and reduce resource waste: This invention uses data access popularity stratification and adopts erasure coding storage scheme for cold data instead of traditional multi-copy storage, which can save storage space compared to multi-copy storage; at the same time, it dynamically matches the minimum number of copies for hot blocks and warm blocks, avoiding the double waste of redundant storage of cold data and excessive redundancy of hot data, and significantly reducing the total cost of ownership of the system.

[0034] Improve data access performance and optimize user experience: Implement tiered deployment based on node storage bandwidth measurement, prioritize the deployment of hot data on nodes with high storage bandwidth and low latency, and increase the number of replicas of hot data through dynamic expansion to effectively distribute access pressure and improve the response speed of frequently accessed data.

[0035] To ensure high data availability and enhance system reliability: Through a collaborative mechanism of "storage bandwidth tiering + dynamic calculation of replica count", the system accurately matches replica configurations that meet availability targets based on a reliability model. At the same time, it prioritizes the deployment of replicas on nodes with stable storage bandwidth and geographically dispersed locations to avoid the risk of single point of failure. The cold data erasure coding scheme only requires a portion of the encoded blocks to restore the original data, further improving the data loss resistance and ensuring that the overall system availability meets the standards. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the system flow in this invention.

[0037] Figure 2 This is a system module diagram of the present invention. Detailed Implementation

[0038] This invention provides a method for dynamically scaling up and down replica storage based on access popularity and storage bandwidth.

[0039] like Figure 1 The method for dynamic scaling of replica storage based on access popularity and storage bandwidth provided by the present invention includes the following specific steps:

[0040] Step 1: The "Bandwidth Assessment Module" completes the storage bandwidth measurement of the storage node;

[0041] Step 2: The "Node Acquisition Module" works in conjunction with the "Popularity Analysis Module" to record the data access popularity.

[0042] Step 3: The "Replica Strategy Calculation Module" performs dynamic calculation of the number of replicas;

[0043] Step 4: The "Replica Expansion Execution Module" completes the dynamic expansion operation of the replicas;

[0044] Step 5: The "Replica Shrinkage and Mode Conversion Module" completes the dynamic shrinkage of replicas and the conversion of cold block storage;

[0045] Step 6: The "Bandwidth and Popularity Verification Module" completes dynamic verification of bandwidth and popularity.

[0046] Step 7: Program execution ends;

[0047] in,

[0048] Step 1, the method for the "bandwidth assessment module" to complete the storage node storage bandwidth measurement is as follows: The "bandwidth assessment module" collects the hardware performance parameters of each storage node, calculates and determines the actual storage bandwidth of each storage node; at the same time, the "bandwidth assessment module" determines the required storage bandwidth threshold for each level of data block according to the system's preset hot block, warm block, and cold block classification standards, ensuring that the total bandwidth of the storage nodes for subsequent replica deployment meets the corresponding threshold requirements, and providing basic parameters for replica deployment and reliability calculation;

[0049] Step 2, "The 'node acquisition module' works in conjunction with the 'heat analysis module' to record data access heat," is as follows: The 'node acquisition module' built into each storage node collects local data block access logs in real time and reports them to the system's 'heat analysis module' according to a preset time period. The 'heat analysis module' aggregates and deduplicates the reported logs, and counts the total number of successful reads for each data block within the current system epoch, which serves as the dynamic access heat of that data block. At the same time, the 'heat analysis module' classifies each data block into hot, warm, and cold blocks based on a preset heat threshold, providing a basis for subsequent bandwidth threshold matching and replica calculation.

[0050] Step 3, the method for the "Replica Strategy Calculation Module" to dynamically calculate the number of replicas is as follows: The "Replica Strategy Calculation Module" retrieves and defines the system's preset availability target. For hot blocks and warm blocks after classification, it first matches the storage bandwidth threshold of their corresponding level, selects candidate storage nodes from the storage node pool whose actual total bandwidth is higher than the corresponding threshold and meets the performance requirements, and then it iteratively calculates through the built-in reliability model to obtain the minimum number of replicas that meets the system availability target. For cold blocks, it first matches the storage bandwidth threshold corresponding to the cold block, selects storage nodes that meet the bandwidth requirements, determines the erasure coding storage scheme, and calculates the required number of coded blocks and the corresponding storage node requirements.

[0051] Step 4, the method by which the "Replica Expansion Execution Module" completes the dynamic expansion operation of replicas is as follows: The "Replica Expansion Execution Module" first checks the current number of replicas of hot blocks and warm blocks, the deployed storage nodes, and the total bandwidth of each deployed storage node. It compares this with the minimum number of replicas and the corresponding bandwidth threshold obtained by the "Replica Policy Calculation Module" in Step 3. If it is determined that the current number of replicas is insufficient, or the total bandwidth of some deployed storage nodes has not reached the corresponding threshold, the "Replica Expansion Execution Module" selects the optimal storage node from the candidate storage nodes to deploy the new replica. The "Replica Expansion Execution Module" ensures that the total bandwidth of the deployed storage nodes of the replicas remains higher than the corresponding threshold after expansion, thus completing the expansion operation and ensuring data access performance.

[0052] Step 5, the method for the "Replica Scaling Down and Mode Conversion Module" to complete dynamic replica scaling down and cold block storage conversion is as follows: The "Replica Scaling Down and Mode Conversion Module" checks the current number of replicas of hot blocks and warm blocks, the deployed storage nodes, and the total bandwidth of each deployed storage node. It compares this with the minimum number of replicas and the corresponding bandwidth threshold obtained by the "Replica Policy Calculation Module" in Step 3. If it is determined that the current number of replicas exceeds the limit, and the total bandwidth of the remaining deployed storage nodes after deleting redundant replicas can still meet the corresponding threshold requirements, then the redundant replicas are deleted, the scaling down operation is completed, and storage resources are saved. At the same time, the "Replica Scaling Down and Mode Conversion Module" checks the current status of cold blocks and the total bandwidth of the corresponding storage nodes to ensure that the total bandwidth of the storage nodes meets the cold block threshold requirements. Then, according to the number of coded blocks obtained by the "Replica Policy Calculation Module" and the storage node requirements, it performs the conversion operation from cold blocks to erasure coding storage mode to ensure the security and bandwidth adaptability of cold block storage.

[0053] Step 6, the method for the "bandwidth and heat verification module" to complete the dynamic verification of bandwidth and heat is as follows: After the replica expansion, shrinkage and cold block conversion operations are completed, the "bandwidth and heat verification module" performs a secondary verification of the total bandwidth of the storage node corresponding to each data block to confirm that the total bandwidth of the storage node is continuously higher than the bandwidth threshold of the corresponding level.

[0054] like Figure 2The replica storage dynamic scaling method 100 based on access popularity and storage bandwidth provided by the present invention generally includes the following modules: "Bandwidth assessment module" 110 is responsible for outputting basic bandwidth data of the underlying nodes; "Node acquisition module" 120 and "Popularity analysis module" 130 complete the data popularity classification; "Replica strategy calculation module" 140 outputs scaling and encoding block configuration schemes for the core decision-making unit; "Replica scaling execution module" 150 and "Replica scaling down and mode conversion module" 160 are the execution unit to implement specific operations; "Bandwidth popularity verification module" 170 serves as the last verification checkpoint to ensure the effectiveness of the operation. There is no functional overlap and the process is smoothly connected.

[0055] To further understand the technical content, features, and effects of this invention, the following application examples are provided, along with detailed descriptions in conjunction with the accompanying drawings:

[0056] Application Example 1

[0057] A major public cloud service provider has fully deployed the dynamic scaling method for replica storage based on access popularity and storage bandwidth described in this invention in its distributed object storage system. The system consists of 300 heterogeneous storage nodes distributed across three availability zones, including high-performance nodes (equipped with NVMe SSDs and 25GbE networks), standard performance nodes (SATA SSDs and 10GbE networks), and archive nodes (large-capacity HDDs and 1GbE networks). The system operates on a 10-minute epoch cycle and is configured with the normalized popularity coefficients x=0.2 and y=0.02 as described in claim 2, continuously executing the dynamic scaling strategy.

[0058] In step 1, "Storage Node Bandwidth Measurement," each storage node uses a built-in monitoring agent to collect real-time hardware performance parameters such as disk sequential read / write throughput, actual network interface bandwidth utilization, average CPU load, and local cache hit rate, reporting these parameters to the "Bandwidth Assessment Module" every 2 minutes. The "Bandwidth Assessment Module," based on a preset assessment model, calculates the actual effective storage bandwidth for each node by combining these indicators. For example, a high-performance node with a measured disk throughput of 2.1GB / s, network egress bandwidth of 2.3GB / s, CPU load of 30%, and cache hit rate of 92% has an effective storage bandwidth of 1.95GB / s after weighted fusion. Simultaneously, tiered bandwidth thresholds are set according to business requirements: hot blocks require a total bandwidth of no less than 800MB / s for deployed nodes, warm blocks no less than 400MB / s, and cold blocks no less than 100MB / s. All nodes are thus categorized into high, medium, and low bandwidth pools, providing a resource foundation for subsequent data placement.

[0059] In step 2, "Data Access Heat Recording," each storage node's "Node Acquisition Module" continuously monitors read requests for the objects it hosts, recording only successfully completed GET operations and generating access logs containing the object ID, timestamp, and client IP. Every 5 minutes, the node batch compresses the logs and uploads them to the "Heat Analysis Module," which performs deduplication (removing duplicate records caused by retries or proxy forwarding) and aggregates them by object ID to calculate the maximum access heat H of all data blocks across the entire system within the current epoch. max =912250 times (the number of successful reads of a popular short video object), and then the number of successful reads of each data block in the current epoch is counted as the access popularity H of that data block. i Among them, the popularity of the short video "video_88765" is H. i =182450 times, satisfying x・H max =0.2×912250=182450 times ≤ H i The document template object "doc_template_v2" meets the criteria for being classified as a hot block and is therefore designated as such; the access popularity H of this document template object is... i =58600 times, satisfying y・H max =0.02×912250=18245 times ≤ H i <182450 visits, meeting the criteria for a warm block, and thus classified as a warm block; the access popularity H of the historical log object "log_20250101" i =8600 times, satisfying 0≤H i The number of occurrences was less than 18245, meeting the criteria for identifying a cold block, and the block was classified as a cold block, providing precise input for the next strategic decision.

[0060] In step 3, "Dynamic Calculation of Replica Count," the system presets an overall availability target of 99.99%. For "video_88765," which is classified as a hot block, the policy engine filters all candidate nodes with current effective bandwidth ≥800MB / s and load below 70% from the high-bandwidth node pool, resulting in 42 available nodes. Subsequently, the system invokes a reliability model, combining the historical monthly failure rate of each candidate node (from the operations and maintenance database), current temperature alarm status, and network packet loss statistics to iteratively evaluate the probability of system unavailability under different replica counts. Calculations show that when 4 replicas are deployed, the overall availability reaches 99.992%, meeting the target; with 3 replicas, it is only 99.97%, failing to meet the target. Therefore, the minimum number of replicas is determined to be 4. Meanwhile, for the historical log object "log_20250101" that has been classified as a cold block, the system no longer uses multiple replicas, but instead selects the RS(8,4) erasure coding scheme, which requires the generation of 12 coding blocks and deployment on at least 12 nodes that meet the cold block bandwidth threshold (≥100MB / s).

[0061] In step 4, "Dynamic Replica Expansion Execution," the system checks that "video_88765" currently only has two replicas deployed, with one located on a standard performance node (measured total bandwidth of only 520MB / s, below the 800MB / s threshold for hot blocks). Therefore, the expansion process is triggered. The policy engine selects two idle nodes from the high-bandwidth pool: one located in East China Availability Zone A (bandwidth 2.0GB / s, low latency), and the other located in South China Availability Zone B (bandwidth 1.9GB / s, geographically dispersed from existing replicas). The system issues replication commands to these two nodes, asynchronously transmitting the complete object data. After both new replicas return "persistence complete" confirmations, the metadata is updated to a 4-replica deployment, with a total effective bandwidth of 7.8GB / s, far exceeding the 800MB / s threshold.

[0062] In step 5, "Dynamic Replica Scaling and Cold Block Storage Conversion Execution," the system detected redundancy in the warm block object "doc_template_v2." This object currently has 3 replicas, but the calculated minimum number of replicas is 2. Furthermore, after deleting any one replica, the total bandwidth of the remaining two nodes is still 950MB / s, exceeding the warm block's 400MB / s threshold. The system prioritizes deleting the replica deployed on the node with a CPU load of 85% to relieve its I / O pressure. Additionally, for the cold block "log_20250101," the system initiates a mode conversion: first, it selects one complete replica from the existing 3 replicas as the source data and performs RS(8,4) encoding to generate 12 fragments; then, it distributes these 12 encoded blocks to 12 archive nodes (each node with a bandwidth ≥110MB / s); after all encoded blocks are successfully written and verified by checksum, the original 3 multi-replicas are securely deleted, completing the smooth migration from multi-replica to erasure coding.

[0063] In step 6, "Dynamic Bandwidth and Popularity Verification," after all scaling and mode conversion operations are completed, the "Bandwidth and Popularity Verification Module" immediately performs a secondary verification of the deployment topology of the relevant data blocks. For "video_88765," the total effective bandwidth of the four deployment nodes is recalculated to be 7.85GB / s, consistently exceeding the 800MB / s threshold. For the nodes containing the 12 encoded blocks of "log_20250101," although the bandwidth of a single node is low, it can aggregate to provide 1.32GB / s bandwidth during concurrent reads, meeting the requirements for batch reading of cold data. After the verification passes, the operations for this epoch cycle are officially committed. If any data block is found to be non-compliant with the threshold, replica migration or an alarm is automatically triggered to ensure that service quality remains under control.

[0064] Application Example 2

[0065] The preset heat threshold system adopts a three - layer dynamic partitioning mechanism: Let the maximum access heat of all data blocks in the current epoch of the system be H max Then, the determination condition for a hot block is that the access heat x•H max ≤H i The determination condition for a warm block is y•H max ≤H i <x•H max The determination condition for a cold block is 0≤H i <y•H max where 0 < y < x < 1 are configurable normalized heat coefficients. x is the normalized heat coefficient for hot blocks, and y is the normalized heat coefficient for warm blocks. Both are constants configured manually according to the requirements of the business scenario and satisfy the value range of 0 < y < x < 1; H i is the dynamic access heat of the i - th data block in the current system epoch. i is the unique identification serial number of the data block, and its value is a positive integer, that is, i = 1, 2, 3, …, n, where n is the total number of data blocks in the current system.

[0066] The back - end storage system of a certain news aggregation App processes hundreds of millions of article reads every day. The system sets the heat normalization coefficients x = 0.6 and y = 0.15.

[0067] In one epoch, the hottest article A in the whole system is read 500,000 times (H max = 500000).

[0068] Article A (H i = 500000≥0.6×500000) is classified as a hot block, triggering multi - replica high - bandwidth deployment;

[0069] Article B (H i = 80000, satisfying 0.15×500000 = 75000≤80000<300000) is classified as a warm block and retained with 2 replicas on standard nodes;

[0070] Article C (H i = 50000<75000) is classified as a cold block and subsequently converted to erasure - code storage.

[0071] Application Example 3

[0072] The specific steps for measuring the storage bandwidth of the storage nodes include: a "bandwidth evaluation module" collecting hardware performance-related indicators for each storage node, including but not limited to disk throughput, network interface speed, CPU processing load, and cache hit efficiency; assigning weights to the above indicators based on the analytic hierarchy process (AHP) to form a multidimensional evaluation system; then using a multiplicative synthesis mechanism to normalize and fuse the indicators to generate a comprehensive storage bandwidth score for each storage node; finally, based on a preset scoring threshold, dividing all storage nodes in the network into three levels: high bandwidth, medium bandwidth, and low bandwidth, and constructing corresponding storage node pools for subsequent resource deployment matching according to data popularity levels.

[0073] A smart driving company is building a vehicle-road cooperative data lake with complex node types (edge ​​servers, central cloud nodes, GPU-accelerated nodes). The "bandwidth assessment module" uses the Analytic Hierarchy Process (AHP) to determine the weights of the following metrics: disk throughput (0.35), network speed (0.30), CPU load (0.20), and cache hit rate (0.15).

[0074] Node N1 (NVMe SSD, 25GbE, low load, high cache hit) scored 0.92 → placed in the high bandwidth pool;

[0075] Node N2 (SATA SSD, 1GbE, high load) scored 0.45 → moved to the medium bandwidth pool;

[0076] Node N3 (mechanical disk, old network card) scored 0.20 → moved to the low bandwidth pool.

[0077] Application Example 4

[0078] The data access heat recording steps include: each storage node's "node acquisition module" continuously monitors the read operations of the data blocks it carries locally, records successful access events in real time, and generates access logs; the access logs are reported in batches to the "heat analysis module" according to a preset time period; after receiving logs from multiple storage nodes, the "heat analysis module" performs aggregation and deduplication processing to eliminate duplicate or redundant records; based on this, it counts the cumulative number of successful reads for each data block in the current system epoch and uses the number of reads as the dynamic access heat of the data block; subsequently, according to a preset heat classification threshold, all data blocks are divided into three levels: hot blocks, warm blocks, and cold blocks, providing input for subsequent replica strategy decisions.

[0079] An online education platform records the playback behavior of course videos. Each storage node's "node acquisition module" compiles access logs and sends them to the "popularity analysis module" every 2 minutes. After aggregation and deduplication, it is confirmed that the video was successfully read 112 times in the current epoch, which is used as its dynamic access popularity.

[0080] Based on preset thresholds (hot blocks ≥ 100 times, warm blocks ≥ 20 times), the video was classified as a hot block, and the system added a copy of it and deployed it to a high-bandwidth node.

[0081] Application Example 5

[0082] The specific steps for dynamically calculating the number of replicas include: first, clarifying the overall availability target preset by the system; for data blocks divided into hot or warm blocks, selecting candidate storage nodes that meet the bandwidth and performance requirements from the storage node pool that matches the heat level of the data blocks, and iteratively evaluating them based on the reliability model until the minimum number of replicas that can meet the availability target is determined; for data blocks divided into cold blocks, instead of using a multi-replica redundancy method, switching to erasure coding storage mode, calculating the required number of original data fragments and parity fragments based on the reliability requirements of cold blocks and available storage node resources, and determining the total number of coded blocks to be deployed and the corresponding number of storage nodes accordingly.

[0083] An e-commerce platform's order system has a set availability target of 99.99%.

[0084] The inventory data block for popular products is classified as a hot block. The strategy engine selects 8 candidate nodes from the high-bandwidth pool and calculates that 3 replicas are needed to meet the availability target based on historical failure rate modeling.

[0085] A snapshot of an order from a week ago was classified as a cold block. The system selected the RS(6,3) erasure coding scheme to generate 9 coding blocks, which need to be deployed on at least 9 nodes that meet the cold block bandwidth threshold (≥80MB / s).

[0086] The system automatically allocates 6 standard nodes + 3 archive nodes to complete the deployment, reducing storage overhead from 3 replicas (300%) to 150%, while maintaining reliability that can tolerate the failure of 3 nodes.

[0087] Application Example 6

[0088] The dynamic expansion, contraction, and storage mode conversion operations of the replicas further include the following optimization strategies: When performing expansion, priority is given to selecting storage nodes with higher storage bandwidth, lower network latency, and geographically dispersed from other replica storage nodes from idle storage nodes to improve concurrent access capabilities and disaster recovery capabilities; when performing contraction, priority is given to deleting redundant replicas deployed on storage nodes with lower storage bandwidth, higher current load, or higher failure risk to release inefficient resources and maintain service quality; when performing the conversion operation of cold blocks from multi-replica to erasure coding mode, the following sub-steps are executed in sequence: first, all existing replicas of the cold block are aggregated to ensure data integrity; then, the data is encoded based on the selected erasure coding scheme to generate multiple coded blocks; next, the coded blocks are distributed to multiple target storage nodes that meet the bandwidth requirements of the cold block; finally, after confirming that all coded blocks are successfully persisted, the original multi-replica data is safely deleted to complete the smooth migration of storage modes.

[0089] A multinational corporation uses the method of this invention to manage its global document library.

[0090] Scaling up scenario: A financial report is being accessed intensively by European users, requiring the system to add new replicas. The optimal deployment locations are idle nodes in Frankfurt (high bandwidth), Singapore (geographically dispersed), and Dallas (low network latency) to improve the global access experience.

[0091] Downsizing scenario: When the popularity of an old version of a contract decreases, the system identifies that one of its replicas is located on a high-load node with CPU continuously >85%, and prioritizes deleting that replica, while retaining the remaining replicas deployed on healthy nodes.

[0092] Cold block conversion: Convert a 3-year-old audit report (originally 3 copies) into erasure coding: First, aggregate the 3 copies to verify consistency → encode 6 blocks as RS(4,2) → distribute to 6 low-bandwidth but stable archive nodes → delete the original 3 copies after confirming successful write, thus achieving a safe and smooth migration.

[0093] Application Example 7

[0094] During the dynamic expansion, contraction, or cold block storage mode conversion of replicas, the system applies a lightweight metadata lock to the target data block and employs a two-phase commit protocol to ensure the atomicity and consistency of the operation. During the expansion phase, the replica location metadata of the data block is only updated after the new replica has been written to the target storage node and returned a persistent confirmation. During the contraction phase, redundant replicas are only allowed to be deleted if it is confirmed that the number and distribution of the remaining replicas still meet the availability and bandwidth requirements. During the cold block mode conversion phase, the deletion operation of the original multiple replicas is delayed until all erasure coding blocks have been successfully distributed and verified, thereby avoiding data loss or service unavailability due to operation interruption or storage node failure.

[0095] A bank's core transaction system strictly adheres to the consistency protocol when performing replica adjustments.

[0096] When expanding a real-time clearing data (hot block), the system first adds a lightweight metadata lock to the data block;

[0097] Write the replica to the new node and wait for it to return a "successful persistence" confirmation;

[0098] The new copy location is only registered in the metadata after this confirmation of arrival;

[0099] If the write operation fails (e.g., due to node crash), the operation is rolled back without unlocking to avoid metadata inconsistency.

[0100] During cold block conversion, the deletion operation of the original 3 replicas is delayed until all 6 erasure coding blocks have passed verification. This mechanism ensures that even if a power outage or network partition occurs during the operation, the system can still maintain data integrity and service availability, meeting financial-grade strong consistency requirements.

[0101] In view of the description and exemplary embodiments of the invention disclosed herein, other embodiments of the invention will be apparent to those skilled in the art. These descriptions and embodiments are considered as examples only, and any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the scope of protection of the invention.

Claims

1. A method for dynamically scaling up and down replica storage based on access popularity and storage bandwidth, characterized in that, The method includes the following specific steps: Step 1: The "Bandwidth Assessment Module" completes the storage bandwidth measurement of the storage node; Step 2: The "Node Acquisition Module" works in conjunction with the "Popularity Analysis Module" to record data access popularity. Step 3: The "Replica Strategy Calculation Module" performs dynamic calculation of the number of replicas; Step 4: The "Replica Expansion Execution Module" completes the dynamic expansion operation of the replicas; Step 5: The "Replica Shrinkage and Mode Conversion Module" completes the dynamic shrinkage of replicas and the conversion to cold block storage; Step 6: The "Bandwidth and Popularity Verification Module" completes dynamic verification of bandwidth and popularity. Step 7: Program execution ends; in, Step 1, the method of "'Bandwidth Assessment Module' completing storage node storage bandwidth measurement" is as follows: The "Bandwidth Assessment Module" collects the hardware performance parameters of each storage node, calculates and determines the actual storage bandwidth of each storage node; at the same time, the "Bandwidth Assessment Module" determines the required storage bandwidth threshold for each level of data block according to the system's preset hot block, warm block, and cold block classification standards, ensuring that the total bandwidth of the storage nodes for subsequent replica deployment meets the corresponding threshold requirements, and providing basic parameters for replica deployment and reliability calculation; Step 2, the method for "the 'node acquisition module' working with the 'heat analysis module' to record data access heat" is as follows: The "node acquisition module" built into each storage node collects local data block access logs in real time and reports them to the system's "heat analysis module" at preset time intervals; the "heat analysis module" aggregates and deduplicates the reported logs, and counts the total number of successful reads of each data block in the current system epoch, which serves as the dynamic access heat of that data block; at the same time, the "heat analysis module" classifies each data block into hot blocks, warm blocks, and cold blocks based on preset heat thresholds, providing a basis for subsequent bandwidth threshold matching and replica calculation; Step 3, the method for the "Replica Strategy Calculation Module" to dynamically calculate the number of replicas is as follows: The "Replica Strategy Calculation Module" retrieves and clarifies the system's preset availability target. For hot blocks and warm blocks after classification, it first matches the storage bandwidth threshold of their corresponding level, selects candidate storage nodes from the storage node pool whose actual total bandwidth is higher than the corresponding threshold and meets the performance requirements, and then it iteratively calculates through the built-in reliability model to obtain the minimum number of replicas that meets the system availability target. For cold blocks, it first matches the storage bandwidth threshold corresponding to the cold block, selects storage nodes that meet the bandwidth requirements, determines the erasure coding storage scheme, and calculates the required number of coded blocks and the corresponding storage node requirements. Step 4, the method of "'Replica Expansion Execution Module' completing the dynamic expansion operation of replicas" is as follows: The "Replica Expansion Execution Module" first checks the current number of replicas of hot blocks and warm blocks, the deployed storage nodes, and the total bandwidth of each deployed storage node. It compares the minimum number of replicas and the corresponding bandwidth threshold obtained by the "Replica Policy Calculation Module" in Step 3. If it is determined that the current number of replicas is insufficient, or the total bandwidth of some deployed storage nodes has not reached the corresponding threshold, the "Replica Expansion Execution Module" selects the optimal storage node from the candidate storage nodes to deploy the new replica. The "Replica Expansion Execution Module" ensures that the total bandwidth of the deployed storage nodes of the replicas continues to be higher than the corresponding threshold after expansion, thus completing the expansion operation and ensuring data access performance. Step 5, the method by which the "Replica Scaling Down and Mode Conversion Module" completes dynamic replica scaling down and cold block storage conversion is as follows: The "Replica Scaling Down and Mode Conversion Module" checks the current number of replicas of hot blocks and warm blocks, the deployed storage nodes, and the total bandwidth of each deployed storage node. It compares this with the minimum number of replicas and the corresponding bandwidth threshold obtained by the "Replica Policy Calculation Module" in Step 3. If it is determined that the current number of replicas exceeds the limit, and the total bandwidth of the remaining deployed storage nodes after deleting redundant replicas can still meet the corresponding threshold requirements, then the redundant replicas are deleted, the scaling down operation is completed, and storage resources are saved. At the same time, the "Replica Scaling Down and Mode Conversion Module" checks the current status of cold blocks and the total bandwidth of the corresponding storage nodes to ensure that the total bandwidth of the storage nodes meets the cold block threshold requirements. Then, according to the number of coded blocks obtained by the "Replica Policy Calculation Module" and the storage node requirements, it performs the conversion operation from cold blocks to erasure coding storage mode to ensure the security and bandwidth adaptability of cold block storage. Step 6, the method of "'Bandwidth and Popularity Verification Module' to complete dynamic verification of bandwidth and popularity" is as follows: After the replica expansion, reduction and cold block conversion operations are completed, the "Bandwidth and Popularity Verification Module" performs a secondary verification of the total bandwidth of the storage node corresponding to each data block to confirm that the total bandwidth of the storage node is continuously higher than the bandwidth threshold of the corresponding level.

2. The method according to claim 1, characterized in that, In the second step, the preset heat threshold system adopts a three-layer dynamic partitioning mechanism: Let the maximum access heat of all data blocks in the current epoch of the system be H max , then the determination condition for a hot block is the access heat x • H max ≤ H i , the determination condition for a warm block is y • H max ≤ H i < x • H max , and the determination condition for a cold block is 0 ≤ H i < y • H max , where 0 < y < x < 1 is a configurable normalized heat coefficient, x is the normalized heat coefficient for hot blocks, and y is the normalized heat coefficient for warm blocks. Both are constants configured manually according to the requirements of the business scenario and satisfy the value range of 0 < y < x < 1; H i is the dynamic access heat of the i-th data block in the current system epoch. i is the unique identification serial number of the data block, and its value is a positive integer, that is, i = 1, 2, 3, …, n, where n is the total number of data blocks in the current system.

3. The method according to claim 1, characterized in that, The specific steps for measuring the storage bandwidth of the storage nodes include: the "bandwidth evaluation module" collecting hardware performance-related indicators of each storage node, including but not limited to disk throughput, network interface speed, CPU processing load, and cache hit efficiency; assigning weights to the above indicators based on the analytic hierarchy process to form a multidimensional evaluation system; then using a multiplicative synthesis mechanism to normalize and fuse the indicators to generate a comprehensive storage bandwidth score for each storage node; finally, based on a preset scoring threshold, the entire network of storage nodes is divided into three levels: high bandwidth, medium bandwidth, and low bandwidth, and corresponding storage node pools are constructed for subsequent resource deployment matching according to data popularity levels.

4. The method according to claim 1, characterized in that, The data access heat recording steps include: each storage node's "node acquisition module" continuously monitors the read operations of the data blocks it carries locally, records successful access events in real time, and generates access logs; the access logs are reported in batches to the "heat analysis module" according to a preset time period; after receiving logs from multiple storage nodes, the "heat analysis module" performs aggregation and deduplication processing to eliminate duplicate or redundant records; based on this, it counts the cumulative number of successful reads for each data block in the current system epoch and uses the number of reads as the dynamic access heat of the data block; subsequently, according to a preset heat classification threshold, all data blocks are divided into three levels: hot blocks, warm blocks, and cold blocks, providing input basis for subsequent replica strategy decisions.

5. The method according to claim 1, characterized in that, The specific steps for dynamically calculating the number of replicas include: first, clarifying the overall availability target preset by the system; for data blocks divided into hot or warm blocks, selecting candidate storage nodes that meet the bandwidth and performance requirements from the storage node pool that matches the heat level of the data blocks, and iteratively evaluating them based on the reliability model until the minimum number of replicas that can meet the availability target is determined; for data blocks divided into cold blocks, instead of using a multi-replica redundancy method, switching to erasure coding storage mode, calculating the required number of original data fragments and parity fragments based on the reliability requirements of cold blocks and available storage node resources, and determining the total number of coded blocks to be deployed and the corresponding number of storage nodes accordingly.

6. The method according to claim 1, characterized in that, The dynamic expansion, contraction, and storage mode conversion operations of the replicas further include the following optimization strategies: When performing expansion, priority is given to selecting storage nodes with higher storage bandwidth, lower network latency, and geographically dispersed from other replica storage nodes from idle storage nodes to improve concurrent access capabilities and disaster recovery capabilities; when performing contraction, priority is given to deleting redundant replicas deployed on storage nodes with lower storage bandwidth, higher current load, or higher failure risk to release inefficient resources and maintain service quality; when performing the conversion operation of cold blocks from multi-replica to erasure coding mode, the following sub-steps are executed in sequence: first, all existing replicas of the cold block are aggregated to ensure data integrity; then, the data is encoded based on the selected erasure coding scheme to generate multiple coded blocks; next, the coded blocks are distributed to multiple target storage nodes that meet the bandwidth requirements of the cold block; finally, after confirming that all coded blocks are successfully persisted, the original multi-replica data is safely deleted to complete the smooth migration of storage modes.

7. The method according to claim 1, characterized in that, During the process of dynamically expanding, shrinking, or converting the cold block storage mode, the system applies a lightweight metadata lock to the target data block and uses a two-phase commit protocol to ensure the atomicity and consistency of the operation. During the expansion phase, the replica location metadata of the data block is only updated after the new replica has been written to the target storage node and returned a persistence confirmation. During the scaling-down phase, redundant replicas are only allowed to be deleted if it is confirmed that the number and distribution of the remaining replicas still meet the availability and bandwidth requirements. During the cold block mode conversion phase, the deletion operation of the original multiple replicas is delayed until all erasure coding blocks have been successfully distributed and verified, thereby avoiding data loss or service unavailability due to operation interruption or storage node failure.

Images