An HDFS optimization method based on dynamic load balancing and adaptive partitioning

Abstract

The application discloses a kind of HDFS optimization method based on dynamic load balancing and adaptive partitioning, belong to distributed storage technical field.The method realizes HDFS performance optimization by four-step cooperation: collect DataNode timing data and train load prediction model, trigger data migration;Statistical block access frequency marks hot data, optimizes consistent hash ring allocation storage location;Adopt exponential smoothing method to divide cold and hot temperature data, implement hierarchical storage and automated migration;According to directory hash value, fragment metadata, through distributed cooperation and cache prefetch optimization operation.The method realizes cluster load balancing, improves data read-write and metadata operation efficiency, optimizes storage resource configuration, guarantees data access continuity and system stability, applicable to massive data storage scene.

Description

Technical Field

[0001] This invention relates to the field of distributed storage technology, and in particular to an HDFS optimization method based on dynamic load balancing and adaptive partitioning. Background Technology

[0002] HDFS (Hadoop Distributed File System), as one of the core technologies in the field of distributed storage, has been widely used in various industry scenarios such as massive data storage, big data analysis, and cloud computing. Its master-slave architecture (NameNode is responsible for metadata management, and DataNodes are responsible for data block storage) has become a recognized mature deployment model in the industry. With the continuous expansion of data scale and the diversification of access requirements, the industry has carried out a lot of technical exploration around HDFS performance optimization, forming a variety of basic optimization solutions. For example, optimizing the storage structure through small file merging and erasure coding storage, realizing the collaborative use of different media through tiered storage technology, and adjusting the distribution of data blocks through load balancing tools based on fixed thresholds to adapt to the storage and access requirements in different scenarios. At the same time, the integration of technologies such as machine learning and distributed collaboration with the storage field is gradually deepening, providing new technical ideas for improving HDFS performance. All kinds of optimization solutions aim to adapt to large-scale data processing and improve system stability, driving the continuous development of distributed storage technology.

[0003] In practical applications, existing technologies still have many unresolved issues, making it difficult to fully meet the efficient and stable storage and access requirements of large-scale data scenarios. First, data distribution lacks dynamic adaptability. Traditional data block allocation often employs random or fixed strategies, failing to flexibly adjust based on data access frequency and real-time node load, resulting in insufficient matching between data and storage resources. Second, load adjustment mechanisms lack foresight. Existing load balancing relies heavily on post-event threshold triggering, unable to anticipate node load changes, easily leading to over-concentration of load and delayed adjustments. Third, data storage lacks refined heat-based hierarchical management; mixed storage of hot and cold data results in inefficient use of high-efficiency storage resources, increasing unnecessary I / O consumption. Finally, metadata management faces single-point bottleneck risks. Traditional metadata storage and processing models struggle to cope with the operational pressure from massive amounts of data, leading to increased metadata operation latency and impacting overall system response speed. These problems are interconnected and mutually influential, restricting HDFS's load balancing effectiveness, read / write response efficiency, storage resource utilization, and metadata management capabilities, necessitating a collaborative and refined optimization solution for systematic resolution. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide an HDFS optimization method based on dynamic load balancing and adaptive partitioning.

[0005] The objective of this invention is achieved through the following technical solution: A method for optimizing HDFS based on dynamic load balancing and adaptive partitioning is provided, which includes the following steps: S1. Collect time-series data from DataNodes, standardize the time-series data and extract features, build a load prediction model, set the loss function and optimizer of the load prediction model, train the load prediction model using historical load data, compare the difference between the current load and the predicted load based on the future load status output by the load prediction model, and execute the data migration trigger operation. S2. Calculate the access frequency of each data block, mark hot data according to the access frequency, calculate the weight of hot data by combining the access frequency and the data block size, perform virtual node mapping on the consistent hash ring, adjust the number of virtual nodes according to the node performance, allocate data block storage locations based on the virtual node mapping, and handle the data migration process in the case of node addition and failure scenarios. S3. The exponential smoothing method is used to calculate the heat value of the data block. The heat value calculation combines the current period access count and the historical heat value. The data is divided into hot data, warm data and cold data according to the heat value. The corresponding storage media is allocated to different types of data to implement hierarchical storage. The heat value of the data block is scanned periodically, and automatic migration operation is performed according to the change of heat value. S4. Perform metadata sharding on the HDFS Namespace according to the directory tree structure. The sharding rules are based on the directory hash value. The consensus protocol is used to achieve collaborative management of metadata sharding. The client caching and prefetching mechanism is adopted to perform parallel processing of metadata creation and deletion operations across shards.

[0006] Furthermore, step S1 includes the following sub-steps: S1.1. Collect time-series data related to CPU utilization, disk I / O throughput, network bandwidth utilization, and data block access frequency. Standardize the raw time-series data and extract sliding window features, which include the mean and variance of the time-series data. S1.2. Construct a load prediction model, set the loss function and optimizer of the load prediction model, adjust the network structure parameters of the load prediction model, and iteratively train the load prediction model using historical load data; S1.3. Set a load deviation threshold based on the cluster size, calculate the difference between the predicted load and the current load, determine whether the difference exceeds the load deviation threshold, sort the DataNodes with overload according to the degree of deviation, determine the type of data blocks to be migrated based on the sorting results, and clarify the priority order of data migration.

[0007] Furthermore, step S2 includes the following sub-steps: S2.1. Calculate the access frequency of each data block within a set period, mark the hot data whose access frequency meets the set conditions, and use the product of access frequency and data block size as the weight of hot data. S2.2. Map each DataNode to a virtual node on the hash ring. Node performance includes hardware configuration parameters and running status parameters. Adjust the number of virtual nodes based on the node performance evaluation results. S2.3. Distribute hot data blocks to the adjacent intervals of virtual nodes of DataNodes whose load meets the set requirements. Establish the correspondence between hot data blocks and DataNodes whose load meets the set requirements by dividing the intervals of virtual nodes on the hash ring. S2.4. When adding a new node, insert the corresponding virtual node on the hash ring, identify the data blocks affected by the insertion of the virtual node and perform migration. When a node fails, locate the virtual node range of the failed node and migrate the data blocks in that range to healthy nodes in a preset order.

[0008] Furthermore, step S3 includes the following sub-steps: S3.1. The popularity value of the data block is calculated using the exponential smoothing method. The historical popularity value is the popularity result calculated in the previous period. The number of visits in the current period and the historical popularity value are used in the popularity value calculation according to a set ratio. S3.2. Data with a heat value reaching the first threshold is defined as hot data, data with a heat value between the first threshold and the second threshold is defined as warm data, and data with a heat value below the second threshold is defined as cold data. Hot data is stored in high-speed storage media, warm data is stored in conventional storage media and access priority is set, and cold data is stored in large-capacity storage media. S3.3. Scan the data block heat value according to the set period, determine whether the heat value continuously meets the migration conditions, start data migration when the migration conditions are met, and determine the data migration direction according to the trend of heat value change; S3.4. Merge migration requests for data blocks that meet the set criteria to reduce the number of metadata operations. When the cold data heat value reaches the standard for warm or hot data, perform data migration operation according to the storage level corresponding to the warm or hot data.

[0009] Furthermore, step S4 includes the following sub-steps: S4.1. The HDFS Namespace is sharded according to the directory tree structure. The sharding rules are based on the directory hash value to ensure that all metadata under the same directory belongs to the same shard. S4.2. Each metadata shard is managed by an independent management node. The management node synchronizes metadata modification information through a consistency protocol, records the addition, deletion and change of metadata, and maintains the consistency of metadata in each shard. S4.3. The client caches the metadata of frequently accessed directories, analyzes the client's historical access records to obtain access patterns, and preloads the relevant metadata to the client based on the access patterns; S4.4. Metadata creation and deletion operations across shards are handled jointly by multiple management nodes. Each management node is assigned a corresponding processing task according to its shard affiliation, and metadata operations for each shard are carried out synchronously.

[0010] Furthermore, in step S1, the load prediction model adopts a two-layer neural network structure. The input layer receives the time-series features extracted by the sliding window. The time-series features include the statistical features and trend features of the data. The output layer outputs the load status for future periods. The load status covers indicators related to CPU utilization, disk I / O throughput, and network bandwidth utilization. Based on the comparison between the output load status and the current load, it is determined whether to execute the data migration trigger operation.

[0011] Furthermore, in step S2.2, the number of virtual nodes is dynamically adjusted based on the hardware performance parameters of the nodes, including CPU processing power, disk read / write speed, and network transmission efficiency. These hardware performance parameters are comprehensively evaluated, and the number of virtual nodes allocated is determined based on the evaluation results. More virtual nodes are allocated to nodes whose hardware performance meets the set level, allowing nodes whose hardware performance meets the set level to undertake more data storage and access-related operations.

[0012] Furthermore, in step S3, when the access frequency of cold data increases, causing the heat value to exceed the second threshold, and the heat value reaches the standard for warm data or hot data, the storage association between the cold data and the original storage medium is removed, the data reading process and the data migration process are started, the data is transferred from the original storage medium to the storage medium corresponding to the warm data or hot data, the storage identification information of the data is updated, and the cold data migration operation is performed.

[0013] Furthermore, in step S4, the operation of writing metadata modification logs to the shared storage medium adopts an asynchronous method. The shared storage medium includes a distributed database and a distributed file system. The metadata modification logs are temporarily stored in the local cache in the order of generation, and then written to the shared storage medium in batches according to a set schedule. This reduces the synchronization waiting process between management nodes and fully records the relevant information of metadata modification.

[0014] Furthermore, in step S4.3, when the client caches the metadata of the frequently accessed directory, it sets an expiration time, verifies the cached metadata according to the set expiration time, performs update or clear operations based on the verification results, and records the access records of the cached metadata to provide data support for subsequent cache optimization.

[0015] The beneficial effects of this invention are: (1) Through the coordinated operation of load prediction, adaptive data partitioning, hot and cold data hierarchical storage and metadata sharding, the cluster load is balanced, data access latency is reduced, and the overall operating efficiency of HDFS is improved. (2) Based on data popularity identification and dynamic storage strategy, optimize storage resource allocation, reduce ineffective resource consumption, and ensure access continuity after data popularity changes; (3) By leveraging the distributed collaborative management of metadata and the client cache prefetching mechanism, the bottleneck of single-point management is broken, the efficiency of metadata operation is improved, and data consistency and system stability are guaranteed. Attached Figure Description

[0016] Figure 1 A flowchart illustrating the steps of an HDFS optimization method based on dynamic load balancing and adaptive partitioning; Figure 2 The following is a flowchart illustrating the specific steps of an HDFS optimization method based on dynamic load balancing and adaptive partitioning, provided as an example. Detailed Implementation

[0017] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Example 1 See Figure 1 This embodiment provides an HDFS optimization method based on dynamic load balancing and adaptive partitioning, which includes the following steps: S1. Collect time-series data from DataNodes, standardize the time-series data and extract features, build a load prediction model, set the loss function and optimizer of the load prediction model, train the load prediction model using historical load data, compare the difference between the current load and the predicted load based on the future load status output by the load prediction model, and execute the data migration trigger operation. S2. Calculate the access frequency of each data block, mark hot data according to the access frequency, calculate the weight of hot data by combining the access frequency and the data block size, perform virtual node mapping on the consistent hash ring, adjust the number of virtual nodes according to the node performance, allocate data block storage locations based on the virtual node mapping, and handle the data migration process in the case of node addition and failure scenarios. S3. The exponential smoothing method is used to calculate the heat value of the data block. The heat value calculation combines the current period access count and the historical heat value. The data is divided into hot data, warm data and cold data according to the heat value. The corresponding storage media is allocated to different types of data to implement hierarchical storage. The heat value of the data block is scanned periodically, and automatic migration operation is performed according to the change of heat value. S4. Perform metadata sharding on the HDFS Namespace according to the directory tree structure. The sharding rules are based on the directory hash value. The consensus protocol is used to achieve collaborative management of metadata sharding. The client caching and prefetching mechanism is adopted to perform parallel processing of metadata creation and deletion operations across shards.

[0019] In some embodiments, step S1 includes the following sub-steps: S1.1. Collect time-series data related to CPU utilization, disk I / O throughput, network bandwidth utilization, and data block access frequency. Standardize the raw time-series data and extract sliding window features, which include the mean and variance of the time-series data. S1.2. Construct a load prediction model, set the loss function and optimizer of the load prediction model, adjust the network structure parameters of the load prediction model, and iteratively train the load prediction model using historical load data; S1.3. Set a load deviation threshold based on the cluster size, calculate the difference between the predicted load and the current load, determine whether the difference exceeds the load deviation threshold, sort the DataNodes with overload according to the degree of deviation, determine the type of data blocks to be migrated based on the sorting results, and clarify the priority order of data migration.

[0020] In some embodiments, step S2 includes the following sub-steps: S2.1. Calculate the access frequency of each data block within a set period, mark the hot data whose access frequency meets the set conditions, and use the product of access frequency and data block size as the weight of hot data. S2.2. Map each DataNode to a virtual node on the hash ring. Node performance includes hardware configuration parameters and running status parameters. Adjust the number of virtual nodes based on the node performance evaluation results. S2.3. Distribute hot data blocks to the adjacent intervals of virtual nodes of DataNodes whose load meets the set requirements. Establish the correspondence between hot data blocks and DataNodes whose load meets the set requirements by dividing the intervals of virtual nodes on the hash ring. S2.4. When adding a new node, insert the corresponding virtual node on the hash ring, identify the data blocks affected by the insertion of the virtual node and perform migration. When a node fails, locate the virtual node range of the failed node and migrate the data blocks in that range to healthy nodes in a preset order.

[0021] In some embodiments, step S3 includes the following sub-steps: S3.1. The popularity value of the data block is calculated using the exponential smoothing method. The historical popularity value is the popularity result calculated in the previous period. The number of visits in the current period and the historical popularity value are used in the popularity value calculation according to a set ratio. S3.2. Data with a heat value reaching the first threshold is defined as hot data, data with a heat value between the first threshold and the second threshold is defined as warm data, and data with a heat value below the second threshold is defined as cold data. Hot data is stored in high-speed storage media, warm data is stored in conventional storage media and access priority is set, and cold data is stored in large-capacity storage media. S3.3. Scan the data block heat value according to the set period, determine whether the heat value continuously meets the migration conditions, start data migration when the migration conditions are met, and determine the data migration direction according to the trend of heat value change; S3.4. Merge migration requests for data blocks that meet the set criteria to reduce the number of metadata operations. When the cold data heat value reaches the standard for warm or hot data, perform data migration operation according to the storage level corresponding to the warm or hot data.

[0022] In some embodiments, step S4 includes the following sub-steps: S4.1. The HDFS Namespace is sharded according to the directory tree structure. The sharding rules are based on the directory hash value to ensure that all metadata under the same directory belongs to the same shard. S4.2. Each metadata shard is managed by an independent management node. The management node synchronizes metadata modification information through a consistency protocol, records the addition, deletion and change of metadata, and maintains the consistency of metadata in each shard. S4.3. The client caches the metadata of frequently accessed directories, analyzes the client's historical access records to obtain access patterns, and preloads the relevant metadata to the client based on the access patterns; S4.4. Metadata creation and deletion operations across shards are handled jointly by multiple management nodes. Each management node is assigned a corresponding processing task according to its shard affiliation, and metadata operations for each shard are carried out synchronously.

[0023] In some embodiments, in step S1, the load prediction model adopts a two-layer neural network structure. The input layer receives the time-series features extracted by the sliding window. The time-series features include the statistical features and trend features of the data. The output layer outputs the load status for future periods. The load status covers indicators related to CPU utilization, disk I / O throughput, and network bandwidth utilization. Based on the comparison between the output load status and the current load, it is determined whether to perform a data migration trigger operation.

[0024] In some embodiments, in step S2.2, the number of virtual nodes is dynamically adjusted according to the hardware performance parameters of the nodes. The hardware performance parameters include CPU processing power, disk read / write speed and network transmission efficiency. These hardware performance parameters are comprehensively evaluated, and the number of virtual nodes to be allocated is determined based on the evaluation results. More virtual nodes are allocated to nodes whose hardware performance meets the set level, so that nodes whose hardware performance meets the set level can undertake more data storage and access-related operations.

[0025] In some embodiments, in step S3, when the access frequency of cold data increases, causing the heat value to exceed the second threshold and the heat value to reach the standard for warm data or hot data, the storage association between the cold data and the original storage medium is terminated, the data reading process and the data migration process are started, the data is transferred from the original storage medium to the storage medium corresponding to the warm data or hot data, the storage identification information of the data is updated, and the cold data migration operation is performed.

[0026] In some embodiments, in step S4, the operation of writing metadata modification logs to the shared storage medium is carried out asynchronously. The shared storage medium includes a distributed database and a distributed file system. The metadata modification logs are temporarily stored in the local cache in the order of generation, and then written to the shared storage medium in batches according to a set batch, reducing the process of synchronization waiting between management nodes and completely recording the relevant information of metadata modification.

[0027] In some embodiments, in step S4.3, when the client caches the metadata of the frequently accessed directory, it sets an expiration time, verifies the cached metadata according to the set expiration time, performs update or clear operations based on the verification result, and records the access records of the cached metadata to provide data support for subsequent cache optimization.

[0028] Example 2 This embodiment provides a specific implementation process for an HDFS optimization method based on dynamic load balancing and adaptive partitioning. Through the coordinated operation of dynamic load prediction, adaptive data partitioning, separate storage of hot and cold data, and parallel metadata management, the operating performance of HDFS is comprehensively optimized. Figure 2 As shown, the specific implementation steps are as follows: S1. Dynamic Load Prediction and Migration Triggering: Dynamic load prediction and migration triggering are prerequisites for optimization methods. By collecting data on the load status of DataNodes in real time, predicting the load using models, and scheduling migrations, performance bottlenecks caused by excessive load concentration can be avoided in advance, laying the foundation for subsequent data partitioning and storage optimization.

[0029] S1.1. Time Series Data Acquisition and Preprocessing: Time-series data is periodically collected from DataNodes, including metrics such as CPU utilization, disk I / O throughput, network bandwidth utilization, and data block access frequency, ensuring that the data comprehensively reflects the hardware operating status and data access characteristics of the DataNodes. The collected raw time-series data is standardized using Z-Score normalization to eliminate dimensional differences between different metrics, providing a unified analytical benchmark for all data types. Based on this, sliding window features are extracted, including the mean and variance of the time-series data. Statistical analysis of data over a fixed time span accurately captures the patterns and fluctuations in data changes, providing high-quality input data for load forecasting models. In some embodiments, time series data preprocessing may also include an outlier handling stage. By setting reasonable value ranges for each indicator, outlier data points that exceed the range are removed, avoiding interference from outliers on feature extraction and model training. For example, the types of outlier data to be processed may include outliers in CPU utilization, disk I / O throughput, network bandwidth utilization, and data block access frequency, ensuring the accuracy of the data input to the model. Sliding window features can also be enhanced by adding statistics such as the rate of change and peak values ​​of the data as needed, further enriching the feature dimensions.

[0030] S1.2. Load Prediction Model Construction and Training: A load prediction model is constructed using a two-layer LSTM network structure. Each layer contains a predetermined number of neurons. The input layer receives preprocessed temporal features, with the time step determined based on the data acquisition cycle and prediction requirements. The output layer predicts the load status for future periods, encompassing metrics such as CPU utilization, disk I / O throughput, and network bandwidth occupancy. The loss function and optimizer for the load prediction model are defined. The loss function uses Mean Absolute Error (MAE) to measure the deviation between the model's predictions and actual load data. The optimizer uses Adam to adjust model parameters and reduce deviation. The network structure parameters of the load prediction model are adjusted, including key configurations such as the number of neurons, learning rate, and number of iterations, to ensure the model has good fitting and generalization capabilities. The load prediction model is iteratively trained using historical load data. The preprocessed temporal feature data is proportionally divided into training, validation, and test sets. The training set is used for model parameter learning, the validation set is used to monitor overfitting and adjust parameters during training, and the test set is used to evaluate the model's final prediction performance. In some embodiments, the load prediction model can be built on a mature deep learning framework, supporting distributed training to improve training efficiency on large-scale historical data. During model training, an early stopping mechanism can be used to avoid overfitting. When the loss value on the validation set no longer decreases for several consecutive iterations, training is stopped and the optimal model parameters are saved. Key parameters involved may include the initial learning rate, the upper limit of the iteration period, and the batch size, which can be dynamically adjusted according to the training effect.

[0031] S1.3. Setting and prioritizing migration trigger thresholds: The load deviation threshold is dynamically set based on the cluster size. The cluster size mainly refers to core indicators such as the number of nodes, total storage capacity, and data processing capabilities to ensure that the threshold setting matches the actual operating capacity of the cluster. The difference between the future load status output by the load prediction model and the current load is calculated. This difference is compared with the set load deviation threshold to determine whether a DataNode is overloaded. Overloaded DataNodes are sorted according to the degree of deviation, with nodes having greater deviation ranking higher. Based on the sorting results, the type of data blocks to be migrated is determined, prioritizing hot data blocks for migration. This clarifies the priority order of data migration, ensuring that the migration operation can specifically alleviate the pressure on high-load nodes. In some embodiments, the load deviation threshold can be dynamically adjusted based on the real-time operating status of the cluster. When the overall cluster load is low, the threshold is appropriately lowered to prevent load concentration in advance; when the overall cluster load is high, the threshold is appropriately increased to avoid additional overhead caused by frequent migrations. The determination of data migration priority can also combine factors such as the access frequency of data blocks, the importance of data blocks, and the correlation between data blocks to form a multi-dimensional priority evaluation system.

[0032] S2. Adaptive Data Partitioning Configuration: Adaptive data partitioning configuration identifies hot data and optimizes hash ring mapping to achieve reasonable data distribution, reduce cross-node data access, and flexibly respond to node addition and failure scenarios, ensuring data storage stability and access efficiency.

[0033] S2.1. Hot Data Identification and Weight Calculation: The system analyzes the access frequency of each data block within a set period, accurately capturing the access frequency of data blocks by recording the number of accesses and timestamps. Hot data blocks are marked based on their access frequency meeting set criteria. When a data block's access frequency exceeds a set threshold, it is marked as hot data for targeted storage and allocation optimization. A weight value is assigned to each hot data block, calculated as: Weight = Access Frequency × Data Block Size. This weight value is used in subsequent virtual node allocation in the hash ring, comprehensively considering both the access frequency and data volume of the data block to ensure that high-weight hot data receives better storage resources. In some embodiments, the statistics of data block access frequency can adopt a sliding window mechanism to update the number of accesses within a set period in real time, avoiding the lag in popularity judgment caused by fixed-period statistics. The threshold for hot data marking can be dynamically adjusted according to the data access characteristics of the cluster. For clusters with dense access requests, the threshold should be appropriately increased; for clusters with smooth access requests, the threshold should be appropriately decreased. The data types involved include data block access count, data block size, and data block weight value, ensuring that the number of data types does not exceed 5.

[0034] S2.2. Consistent Hash Ring Optimization and Virtual Node Mapping: Each DataNode is mapped to multiple virtual nodes on a hash ring. The introduction of virtual nodes reduces the impact of data migration and improves the uniformity of data distribution. Node performance includes hardware configuration parameters and operational status parameters. Hardware configuration parameters cover CPU processing power, disk storage capacity, network transmission bandwidth, etc., while operational status parameters include current load level, response latency, and fault history. These hardware performance parameters are comprehensively evaluated, and the number of virtual nodes is adjusted based on the evaluation results to match the distribution of virtual nodes with the actual performance of the DataNodes. In some embodiments, the mapping of virtual nodes can be implemented using a consistent hashing algorithm. The position of a DataNode on the hash ring is determined by hashing its identification information. Adjustments to the number of virtual nodes can be made by setting minimum and maximum thresholds to avoid uneven data distribution due to too few virtual nodes, or increased management overhead due to too many. For example, hardware performance evaluation can select five indicators: CPU processing power, disk read / write speed, network transmission efficiency, current load level, and failure frequency, to ensure compliance with data quantity limits.

[0035] S2.3. Establishment of thermal data distribution and correspondence: Hot data blocks are preferentially allocated to the adjacent range of virtual nodes of DataNodes whose load meets the set requirements. The storage range corresponding to each virtual node is defined by the range division of virtual nodes on the hash ring. A correspondence between hot data blocks and DataNodes whose load meets the set requirements is established to ensure that hot data blocks with high access frequency can be allocated to nodes with low load and strong response capability, reducing cross-node transmission during hot data access and improving data read and write efficiency. In some embodiments, hot data blocks can be allocated using a hash mapping method. The identifier information of the hot data block is hashed and mapped to the corresponding interval on a hash ring, thereby determining its corresponding virtual node and its associated DataNode. To ensure the rationality of the allocation, the load status of the DataNode can be periodically monitored, and the allocation relationship of hot data blocks can be readjusted when the node load changes.

[0036] S2.4. Handling dynamic changes in nodes: When a new DataNode joins the cluster, a corresponding virtual node is inserted into the hash ring. The number of virtual nodes is determined based on the performance evaluation results of the new DataNode. Data blocks affected by the insertion of virtual nodes are identified. These data blocks were originally stored within the virtual node range adjacent to the new node, and their storage locations need to be replanned with the insertion of the new virtual node. Only the affected data blocks are migrated to the virtual node range corresponding to the new node, avoiding network overhead and performance degradation caused by large-scale data migration. When a DataNode fails, the virtual node range of the failed node is located, clarifying the storage range it was responsible for. Data blocks within this range are automatically migrated to the next healthy node in a clockwise direction. Healthy nodes are selected to receive the migrated data in a preset order, ensuring rapid recovery of data from the failed node and guaranteeing data availability and access continuity. In some embodiments, the insertion position of the newly added virtual node can be determined by a random algorithm to further improve the uniformity of data distribution. Node fault detection can employ a heartbeat mechanism, periodically receiving heartbeat signals from DataNodes. When no heartbeat signal is received for several consecutive cycles, a node fault is determined and data migration is triggered.

[0037] S3. Tiered storage and automated migration of hot and cold data: Tiered storage and automated migration of hot and cold data implements differentiated storage strategies based on data popularity, optimizes storage resource allocation, reduces storage costs, and ensures access performance after changes in data popularity through an automated migration mechanism.

[0038] S3.1. Calculation of data block heat value: The exponential smoothing method is used to calculate the popularity value of a data block. This method smooths short-term fluctuations in data access by combining the current period's access count with historical popularity values, accurately reflecting the long-term popularity trend of the data block. The calculation formula is: St=α⋅Xt+(1−α)⋅St−1 Where St is the popularity value of the data block in the current period, Xt is the number of times the data block is accessed in the current period, St−1 is the popularity result calculated in the previous period, and α is an empirical value used to adjust the weight ratio of the number of accesses in the current period and the historical popularity value in the popularity calculation. In some embodiments, the value of α can be adjusted according to the data access characteristics of the cluster. For clusters with rapidly changing access patterns, the value of α can be increased appropriately to make the impact of the current period's access count more significant; for clusters with stable access patterns, the value of α can be decreased appropriately to highlight the reference significance of historical popularity values. The data types involved include the current period's access count, historical popularity values, current popularity values, and the α parameter, ensuring that the number of data types does not exceed 5.

[0039] S3.2. Data Classification and Hierarchical Storage: Data blocks are categorized based on calculated heat values. Data with heat values ​​reaching a first threshold are defined as hot data, data with heat values ​​between the first and second thresholds are defined as warm data, and data with heat values ​​below the second threshold are defined as cold data. A tiered storage strategy is implemented for different data types. Hot data is stored on high-speed storage media, which features fast read / write speeds and low response latency, meeting the high-frequency, high-real-time access requirements of hot data. Warm data is retained on conventional storage media with access priority set. Conventional storage media balances storage performance and cost, and by setting access priorities, it ensures that warm data receives priority access to system resources when accessed. Cold data is migrated to high-capacity storage media, which has the advantages of high storage density and low unit storage cost, making it suitable for storing cold data with low access frequency and low response speed requirements. In some embodiments, high-speed storage media may be SSDs or memory caches, conventional storage media may be HDDs, and large-capacity storage media may be high-capacity HDDs or archive storage devices. The settings of the first threshold and the second threshold can be dynamically adjusted according to the storage resource status of the cluster and the data access distribution characteristics to ensure the rationality of data classification.

[0040] S3.3. Periodic Scanning and Migration Triggering: The system scans the popularity values ​​of all data blocks at set intervals and periodically reassesses the popularity status of data blocks to ensure that data classification is consistent with actual access patterns. It determines whether the popularity values ​​of data blocks continuously meet migration conditions. Continuously meeting these conditions avoids frequent migrations caused by short-term data access fluctuations, ensuring the rationality and effectiveness of migration operations. Data migration is initiated when the popularity values ​​continuously meet the migration conditions. The migration direction is determined based on the trend of popularity value changes: if the data popularity value continues to rise, it migrates from a lower-level storage medium to a higher-level storage medium; if the data popularity value continues to fall, it migrates from a higher-level storage medium to a lower-level storage medium. In some embodiments, the scanning cycle can be adjusted according to the frequency of data popularity changes. For clusters with rapid data popularity changes, the scanning cycle is shortened; for clusters with gradual data popularity changes, the scanning cycle is extended. The migration conditions can be set by combining factors such as the magnitude and duration of data block popularity value changes to form a multi-dimensional migration judgment standard.

[0041] S3.4. Migration process optimization and migration back operation: Merging migration requests for data blocks that meet set criteria consolidates multiple migration requests for data blocks needing to be migrated to the same target storage medium or the same DataNode into a single batch migration task. This reduces the number of metadata operations, lowers the pressure on metadata management, and improves migration efficiency. When the access frequency of cold data increases, causing the hotness value to exceed the second threshold and reach the standard for warm or hot data, a cold data rollback operation is triggered. This disconnects the cold data from the original high-capacity storage medium, initiates the data reading and migration processes, and transfers the data from the original storage medium to the storage medium corresponding to the warm or hot data. The storage identification information of the data is updated to ensure that the data can obtain the corresponding storage resources and access performance after the increase in data hotness. In some embodiments, the integration of batch migration tasks can be based on factors such as migration target and migration time window. Small data blocks with the same migration target are prioritized for batch integration. Cold data migration operations can be prioritized. Cold data with a sudden increase in popularity is given a higher migration priority to ensure that it can be quickly restored to high-efficiency storage media.

[0042] S4. Parallelized Metadata Management and Optimization: Parallelized metadata management and optimization addresses the bottleneck of single-point metadata management by splitting metadata management tasks, implementing distributed collaborative management, and accelerating caching, thereby improving the efficiency of metadata operations and ensuring the consistency and availability of metadata.

[0043] S4.1. Metadata Fragmentation: HDFS Namespaces are sharded according to a directory tree structure, adhering to the hierarchical relationships and logical connections between directories to ensure that sharding maintains the integrity of the directory structure. Sharding rules are based on directory hash values. The name or identifier information of each directory is hashed, and the directory is assigned to the corresponding metadata shard according to the hash result. This ensures that all metadata under the same directory belongs to the same shard, avoiding directory data scattering caused by sharding and guaranteeing the continuity of data access. In some embodiments, the number of metadata shards can be dynamically adjusted based on the total amount of metadata and the number of directories in the cluster. When the total amount of metadata increases to a set threshold, new metadata shards are added to alleviate the management pressure. Hash calculations can employ mature hash algorithms to ensure a uniform distribution of directories across the shards.

[0044] S4.2. Fragmented Collaborative Management and Consistency Guarantee: Each metadata shard is managed by an independent management node. Each management node specializes in handling metadata operations for its corresponding shard, including adding, deleting, querying, and modifying metadata. This distributed approach to metadata management tasks avoids excessive pressure on a single point of failure. Metadata consistency is achieved among management nodes through a consistency protocol. This protocol ensures that all management nodes' metadata modification operations remain synchronized, preventing data conflicts or inconsistencies. All additions, deletions, and changes to metadata are recorded, forming a complete metadata operation log. The log includes key information such as operation type, time, target, and result, facilitating data traceability and fault recovery. In some embodiments, the Raft protocol can be used as the consistency protocol. This protocol achieves consistent synchronization of metadata among management nodes through mechanisms such as leader election, log replication, and security guarantees. Management nodes can be configured with backup nodes. When the primary management node fails, the backup node can quickly take over metadata management, ensuring the continuity of metadata management. The metadata operation types involved include adding, deleting, modifying, and querying metadata, ensuring that the number of data types does not exceed five.

[0045] S4.3. Client-side caching and prefetching optimization: The client caches metadata for frequently accessed directories, storing the metadata of directories frequently accessed by the client within a set period in a local cache. This reduces the number of communications between the client and the management node, lowering metadata access latency. Access patterns are obtained by analyzing historical client access records. By mining historical access data, characteristics such as frequently used directories, access time patterns, and access to associated directories are identified. Based on these access patterns, relevant metadata is pre-loaded onto the client. Before the client initiates an access request, potentially needed metadata is pre-cached locally, further shortening metadata access response time. An expiration time is set for the client's cached metadata for frequently accessed directories. The cached metadata is validated according to the set expiration time to determine if the cached data has expired. Update or clear operations are performed based on the validation result. If the cached data has expired, the latest metadata is retrieved from the management node and the cache is updated; if the cached data is valid, it is used directly. Access records for cached metadata are also recorded, including access time, access count, and cache hit rate, providing data support for subsequent cache optimization. In some embodiments, the cache expiration time can be adjusted according to the update frequency of metadata. For frequently updated metadata, a shorter expiration time is set; for less frequently updated metadata, a longer expiration time is set. Cache prefetching can employ an algorithm based on association rules, predicting potentially subsequently accessed associated directories based on the directories accessed by the client and preloading their metadata.

[0046] S4.4. Parallel processing of metadata across shards: Metadata creation and deletion operations across shards are handled collaboratively by multiple management nodes. Each management node is only responsible for handling metadata operations within its assigned shard, avoiding excessive cross-shard operation load on a single node. Processing tasks are assigned to each management node based on shard affiliation, clearly defining the responsibilities of each node and ensuring a reasonable and balanced task allocation. Metadata operations across shards are performed synchronously. By coordinating the execution progress of each management node, cross-shard metadata operations are ensured to be completed efficiently and systematically, improving the overall throughput of metadata operations. In some embodiments, the coordination of cross-shard metadata operations can be achieved through a unified scheduling module. The scheduling module is responsible for receiving operation requests, splitting tasks, allocating tasks, monitoring execution progress, and returning results to the client after all shard operations are completed.

[0047] S4.5. Asynchronous storage of metadata modification logs: The metadata modification log is written to the shared storage medium asynchronously. The logs are temporarily stored in the local cache of the management node in the order they are generated, ensuring no data loss. Then, they are written to the shared storage medium in batches according to a set schedule. Batch writing reduces the number of interactions with the shared storage medium, improving log writing efficiency. The shared storage medium includes a distributed database and a distributed file system, possessing high availability, high reliability, and high scalability. This reduces the synchronization waiting time between management nodes; management nodes can continue processing subsequent metadata operations without waiting for the logs to be written to the shared storage medium completely, providing a complete record of metadata modification information for metadata recovery, auditing, and traceability. In some embodiments, a maximum cache capacity can be set for locally cached log data. When the cache capacity reaches the set threshold, a batch write operation is triggered to avoid local cache overflow. The choice of shared storage medium can be based on the cluster's storage needs and performance requirements. For scenarios with frequent log queries, a distributed database can be selected; for scenarios with large log storage volume and low access frequency, a distributed file system can be selected.

[0048] The HDFS optimization method based on dynamic load balancing and adaptive partitioning provided in this embodiment uses a dynamic load prediction model to anticipate changes in DataNode load. Combined with a migration triggering mechanism and priority sorting, it effectively avoids performance bottlenecks caused by excessive load concentration and achieves balanced distribution of cluster load. The adaptive data partitioning strategy optimizes the hash ring mapping relationship based on data popularity and node performance, reducing cross-node data access and improving the response speed and efficiency of data read and write. Tiered storage for hot and cold data rationally allocates different types of storage media according to data popularity, optimizing the utilization of storage resources and reducing storage costs. At the same time, through automated migration and rollback mechanisms, it ensures the stability of access performance after changes in data popularity. Parallel metadata management breaks through the bottleneck of single-point metadata management through sharding, distributed collaboration, cache prefetching, and parallel processing, improving the throughput and efficiency of metadata operations and ensuring the consistency and availability of metadata. The overall solution, through close collaboration and efficient operation of each link, comprehensively optimizes the operating performance of HDFS, enhances the stability and adaptability of the system under different operating scenarios, and provides reliable technical support for the storage and access of massive amounts of data.

[0049] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. An HDFS optimization method based on dynamic load balancing and adaptive partitioning, characterized in that, Includes the following steps: S1. Collect time-series data from DataNodes, standardize the time-series data and extract features, build a load prediction model, set the loss function and optimizer of the load prediction model, train the load prediction model using historical load data, compare the difference between the current load and the predicted load based on the future load status output by the load prediction model, and execute the data migration trigger operation. S2. Calculate the access frequency of each data block, mark hot data according to the access frequency, calculate the weight of hot data by combining the access frequency and the data block size, perform virtual node mapping on the consistent hash ring, adjust the number of virtual nodes according to the node performance, allocate data block storage locations based on the virtual node mapping, and handle the data migration process in the case of node addition and failure scenarios. S3. The exponential smoothing method is used to calculate the heat value of the data block. The heat value calculation combines the current period access count and the historical heat value. The data is divided into hot data, warm data and cold data according to the heat value. The corresponding storage media is allocated to different types of data to implement hierarchical storage. The heat value of the data block is scanned periodically, and automatic migration operation is performed according to the change of heat value. S4. Perform metadata sharding on the HDFS Namespace according to the directory tree structure. The sharding rules are based on the directory hash value. The consensus protocol is used to achieve collaborative management of metadata sharding. The client caching and prefetching mechanism is adopted to perform parallel processing of metadata creation and deletion operations across shards.

2. The method according to claim 1, characterized in that, Step S1 includes the following sub-steps: S1.

1. Collect time-series data related to CPU utilization, disk I / O throughput, network bandwidth utilization, and data block access frequency. Standardize the raw time-series data and extract sliding window features, which include the mean and variance of the time-series data. S1.

2. Construct a load prediction model, set the loss function and optimizer of the load prediction model, adjust the network structure parameters of the load prediction model, and iteratively train the load prediction model using historical load data; S1.

3. Set a load deviation threshold based on the cluster size, calculate the difference between the predicted load and the current load, determine whether the difference exceeds the load deviation threshold, sort the DataNodes with overload according to the degree of deviation, determine the type of data blocks to be migrated based on the sorting results, and clarify the priority order of data migration.

3. The method according to claim 1, characterized in that, Step S2 includes the following sub-steps: S2.

1. Calculate the access frequency of each data block within a set period, mark the hot data whose access frequency meets the set conditions, and use the product of access frequency and data block size as the weight of hot data. S2.

2. Map each DataNode to a virtual node on the hash ring. Node performance includes hardware configuration parameters and running status parameters. Adjust the number of virtual nodes based on the node performance evaluation results. S2.

3. Distribute hot data blocks to the adjacent intervals of virtual nodes of DataNodes whose load meets the set requirements. Establish the correspondence between hot data blocks and DataNodes whose load meets the set requirements by dividing the intervals of virtual nodes on the hash ring. S2.

4. When adding a new node, insert the corresponding virtual node on the hash ring, identify the data blocks affected by the insertion of the virtual node and perform migration. When a node fails, locate the virtual node range of the failed node and migrate the data blocks in that range to healthy nodes in a preset order.

4. The method according to claim 1, characterized in that, Step S3 includes the following sub-steps: S3.

1. The popularity value of the data block is calculated using the exponential smoothing method. The historical popularity value is the popularity result calculated in the previous period. The number of visits in the current period and the historical popularity value are used in the popularity value calculation according to a set ratio. S3.

2. Data with a heat value reaching the first threshold is defined as hot data, data with a heat value between the first threshold and the second threshold is defined as warm data, and data with a heat value below the second threshold is defined as cold data. Hot data is stored in high-speed storage media, warm data is stored in conventional storage media and access priority is set, and cold data is stored in large-capacity storage media. S3.

3. Scan the data block heat value according to the set period, determine whether the heat value continuously meets the migration conditions, start data migration when the migration conditions are met, and determine the data migration direction according to the trend of heat value change; S3.

4. Merge migration requests for data blocks that meet the set criteria to reduce the number of metadata operations. When the cold data heat value reaches the standard for warm or hot data, perform data migration operation according to the storage level corresponding to the warm or hot data.

5. The method according to claim 1, characterized in that, Step S4 includes the following sub-steps: S4.

1. The HDFS Namespace is sharded according to the directory tree structure. The sharding rules are based on the directory hash value to ensure that all metadata under the same directory belongs to the same shard. S4.

2. Each metadata shard is managed by an independent management node. The management node synchronizes metadata modification information through a consistency protocol, records the addition, deletion and change of metadata, and maintains the consistency of metadata in each shard. S4.

3. The client caches the metadata of frequently accessed directories, analyzes the client's historical access records to obtain access patterns, and preloads the relevant metadata to the client based on the access patterns; S4.

4. Metadata creation and deletion operations across shards are handled jointly by multiple management nodes. Each management node is assigned a corresponding processing task according to shard affiliation, and metadata operations of each shard are carried out synchronously.

6. The method according to claim 1, characterized in that, In step S1, the load prediction model adopts a two-layer neural network structure. The input layer receives the time-series features extracted by the sliding window. The time-series features include the statistical features and trend features of the data. The output layer outputs the load status for future periods. The load status covers indicators related to CPU utilization, disk I / O throughput, and network bandwidth utilization. Based on the comparison between the output load status and the current load, it is determined whether to execute the data migration trigger operation.

7. The method according to claim 3, characterized in that, In step S2.2, the number of virtual nodes is dynamically adjusted based on the hardware performance parameters of the nodes. These hardware performance parameters include CPU processing power, disk read / write speed, and network transmission efficiency. These hardware performance parameters are comprehensively evaluated, and the number of virtual nodes to be allocated is determined based on the evaluation results. More virtual nodes are allocated to nodes whose hardware performance meets the set level, allowing nodes whose hardware performance meets the set level to undertake more data storage and access-related operations.

8. The method according to claim 4, characterized in that, In step S3, when the access frequency of cold data increases, causing the heat value to exceed the second threshold and the heat value to reach the standard for warm data or hot data, the storage association between the cold data and the original storage medium is removed, the data reading process and the data migration process are started, the data is transferred from the original storage medium to the storage medium corresponding to the warm data or hot data, the storage identification information of the data is updated, and the cold data migration operation is performed.

9. The method according to claim 5, characterized in that, In step S4, the operation of writing metadata modification logs to the shared storage medium is carried out asynchronously. The shared storage medium includes a distributed database and a distributed file system. The metadata modification logs are temporarily stored in the local cache in the order of generation, and then written to the shared storage medium in batches according to the set batches, which reduces the process of synchronization waiting between management nodes and fully records the relevant information of metadata modification.

10. The method according to claim 5, characterized in that, In step S4.3, when the client caches the metadata of the frequently accessed directory, it sets an expiration time, verifies the cached metadata according to the set expiration time, performs update or clear operations based on the verification results, and records the access records of the cached metadata to provide data support for subsequent cache optimization.

Images