A method and system for optimizing seamless upgrade of large-scale HDFS storage system

Abstract

The application discloses a method and system for optimizing seamless upgrade of a large-scale HDFS storage system, wherein the method comprises the following steps: defining a new replica placement strategy to automatically distribute three replicas of each data block on three different racks in a cluster; automatically adjusting the replica placement strategy when the state of the cluster changes; optimizing the replica replication logic of the NameNode to identify the expected temporary stop state of the DataNode during the upgrade; and implementing a hierarchical replication strategy according to the severity of different scenarios during the expected temporary stop, and combining a gradual data migration strategy. The method and system ensure that the three replicas of the data block are distributed on three different racks by defining a new replica placement strategy, thereby reducing the risk of data loss; the temporary stop state of the DataNode is marked to avoid unnecessary replica replication, and the hierarchical replication and gradual data migration strategies ensure the high availability and performance stability of the system during the upgrade, so that the seamless upgrade of the large-scale HDFS storage system is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of large-scale HDFS storage system upgrade, and particularly relates to a method and system for optimizing seamless upgrade of large-scale HDFS storage system. BACKGROUND

[0002] In the process of upgrading a large-scale HDFS storage system, some challenges will be faced:

[0003] 1. Data loss risk caused by DataNode service stop.

[0004] During the upgrade process, the operating system or software version of each node needs to be updated, which will inevitably cause the DataNode service to stop. Although HDFS ensures data availability through rack awareness and three-copy mechanism, there is still a risk of data loss due to the following reasons.

[0005] (1) Too long time for upgrading one by one: Although upgrading one by one does not have the risk of data loss, the way of upgrading and restarting the storage server one by one is very time-consuming, especially in a large-scale cluster (e.g. with thousands of nodes). This approach is not feasible because it significantly slows down the upgrade process.

[0006] (2) Risk of upgrading by rack: If upgrading by rack, when two replicas in a rack are unavailable at the same time, the permanent loss of data will occur once the remaining third replica fails.

[0007] 2. Performance and resource burden caused by a large number of data block replication operations.

[0008] During the upgrade process, when HDFS detects a large number of DataNode service stops, it will trigger data block replication operations. This brings the following problems:

[0009] (1) Network and storage resource consumption: A large number of data block replication will significantly increase the consumption of network bandwidth and storage resources, especially during the upgrade period. This may cause network congestion and storage device overload, further affecting other normally running services.

[0010] (2) Cluster performance degradation: Data block replication operations also negatively impact the overall performance and response time of the cluster, making the upgrade process slower and affecting the availability of the cluster and user experience. SUMMARY

[0011] To solve the above problems existing in the upgrading process of the large-scale HDFS storage system, the application provides a method and system for optimizing seamless upgrading of the large-scale HDFS storage system, a new replica placement strategy is developed to ensure that the three replicas of a data block are distributed on three different racks, and the risk of data loss is reduced; the temporary stop state of the DataNode is marked to avoid unnecessary replica replication, and the hierarchical replication and gradual data migration strategies are used to ensure the high availability and performance stability of the system during the upgrading process, so that the seamless upgrading of the large-scale HDFS storage system is realized.

[0012] To achieve the above purposes, the application adopts the following technical solutions:

[0013] In an embodiment of the application, a method for optimizing seamless upgrading of a large-scale HDFS storage system is provided, which comprises:

[0014] A new replica placement strategy is defined to automatically distribute the three replicas of each data block on three different racks in the cluster; when the cluster state changes, the replica placement strategy is automatically adjusted;

[0015] The replica replication logic of the NameNode is optimized to identify the expected temporary stop state of the DataNode during the upgrading process; during the expected temporary stop state, a hierarchical replication strategy is implemented according to the severity of different scenarios, and a gradual data migration strategy is combined.

[0016] Further, the new replica placement strategy comprises:

[0017] The first replica is selected to be placed on the same rack as the data write node;

[0018] The second replica is selected from a node with low load on a rack different from the first replica;

[0019] The third replica is selected from a node with low load on a third rack different from the racks where the first two replicas are located;

[0020] The fallback mechanism: when there are only two racks in the cluster, two replicas are placed on one rack, and the other replica is placed on the other rack.

[0021] Further, based on the real-time collected node state information, the node load and available resources are evaluated, and the most suitable node for storing the replica is dynamically selected; the node selection is optimized in combination with the network topology information;

[0022] The load distribution of each node in the cluster is regularly checked and evaluated to determine whether the replica redistribution operation needs to be triggered; in the redistribution operation, the principle of distributing the three replicas on three different racks is still followed.

[0023] Further, when a cluster deactivates a certain DataNode during upgrade, the node can be marked as a short-term stop state in advance;

[0024] The NameNode checks whether the node is in the short-term stop state before triggering the replica replication;

[0025] When the NameNode implements the replica replication, the replicas distributed on low-load racks or nodes are preferentially selected for migration.

[0026] In an embodiment of the present application, a system for optimizing seamless upgrade of a large-scale HDFS storage system is also provided, and the system comprises:

[0027] A replica placement strategy customization module is configured to customize a new replica placement strategy, so that three replicas of each data block are automatically distributed on three different racks in the cluster; and the replica placement strategy is automatically adjusted when the state of the cluster changes.

[0028] A replica replication logic optimization module is configured to optimize the replica replication logic of the NameNode, so that the NameNode can identify the short-term stop state of the DataNode during upgrade; and during the short-term stop state, a hierarchical replication strategy is implemented according to the severity of different scenarios, and a gradual data migration strategy is combined.

[0029] Further, the new replica placement strategy includes:

[0030] The first replica is selected to be placed on the same rack as the data write-in node;

[0031] The second replica is selected from a node on a rack different from that of the first replica;

[0032] The third replica is selected from a node on a third rack different from those of the first and second replicas;

[0033] A fallback mechanism is configured to place two replicas on one rack and the other replica on the other rack when there are only two racks in the cluster.

[0034] Further, according to the real-time collected node state information, the node load and available resources are evaluated, and the most suitable node for storing the replica is dynamically selected; and the node selection is optimized in combination with the network topology information.

[0035] The load distribution of each node in the cluster is periodically checked and evaluated to determine whether the replica redistribution operation needs to be triggered; and in the redistribution operation, the principle of distributing three replicas on three different racks is still followed.

[0036] Further, when the cluster disables a certain DataNode during the upgrade, the node can be marked in advance as an expected temporary stop state;

[0037] The NameNode checks whether the node is in the expected temporary stop state before triggering the replica replication;

[0038] When the NameNode implements the replica replication, the replicas distributed on low-load racks or nodes are preferentially selected for migration.

[0039] In an embodiment of the present application, a computer device is also provided, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the method for optimizing the seamless upgrade of the large-scale HDFS storage system.

[0040] In an embodiment of the present application, a computer readable storage medium is also provided, which stores a computer program for executing the method for optimizing the seamless upgrade of the large-scale HDFS storage system.

[0041] Advantages:

[0042] 1. The present application develops a new replica placement strategy by self-definition, which greatly reduces the risk of data loss caused by rack-level failures, and improves the reliability and security of data.

[0043] 2. The present application supports automatic adjustment of the replica placement strategy when the cluster state changes (such as node failure or rack load fluctuation). This flexible response mechanism ensures that the system can always maintain high efficiency and stability under different operating conditions.

[0044] 3. The present application optimizes the replica replication logic of the NameNode, which can identify the state of the DataNode during the upgrade, avoiding unnecessary replica replication operations, thereby reducing the system load.

[0045] 4. The present application implements a hierarchical replication strategy according to the severity of the scene, combined with a gradual data migration method, effectively reducing the consumption of network and storage resources, and ensuring the high availability of data and the overall performance of the system. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 is a flowchart of the method for optimizing the seamless upgrade of the large-scale HDFS storage system according to an embodiment of the present application;

[0047] Figure 2 is a schematic diagram of the default three-replica two-rack placement of the replica placement strategy according to an embodiment of the present application;

[0048] Figure 3Is the three-copy three-rack placement schematic diagram of the copy placement strategy customized by the application;

[0049] Figure 4 Is the system structure schematic diagram of the application for optimizing seamless upgrade of large-scale HDFS storage system;

[0050] Figure 5 Is the computer device structure schematic diagram of the application. DETAILED DESCRIPTION

[0051] The principles and spirits of the application will be described below with reference to several exemplary embodiments, and it should be understood that these embodiments are given only to enable those skilled in the art to better understand and design the application, and do not limit the scope of the application in any way. On the contrary, these embodiments are provided to make the disclosure more thorough and complete, and to fully convey the scope of the disclosure to those skilled in the art.

[0052] Those skilled in the art know that the embodiments of the application can be designed as a system, a system, a device, a method or a computer program product. Therefore, the disclosure can be specifically designed in the following forms: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0053] According to the embodiments of the application, a method for optimizing seamless upgrade of large-scale HDFS storage system is proposed, by customizing the development of a new copy placement strategy, ensuring that the three copies of the data block are distributed on three different racks, reducing the risk of data loss; marking the temporary stop state of DataNode to avoid unnecessary copy replication, and through the hierarchical replication and gradual data migration strategy, guaranteeing the high availability and performance stability of the system during the upgrade process, to realize the seamless upgrade of large-scale HDFS storage system.

[0054] The principles and spirits of the application will be described below with reference to several exemplary embodiments, and it should be understood that these embodiments are given only to enable those skilled in the art to better understand and design the application, and do not limit the scope of the application in any way. On the contrary, these embodiments are provided to make the disclosure more thorough and complete, and to fully convey the scope of the disclosure to those skilled in the art.

[0055] The method for optimizing seamless upgrade of large-scale HDFS storage system of the application comprises:

[0056] 1. Customizing the copy placement strategy: developing a new data copy placement strategy, so that the three copies of each data block are automatically distributed on three different racks (by default, the three copies of each data block are automatically distributed on two different racks, as shown in Figure 2 ), as shown in Figure 3 , reducing the risk of data loss, and introducing a strategy rollback mechanism to ensure safe data storage when the conditions are not met.

[0057] 2. Optimizing the NameNode replica replication logic: Modify the replica replication logic of the NameNode to recognize the expected temporary stop state of the DataNode during the upgrade, avoid triggering unnecessary replica replication operations, and reduce excessive consumption of cluster performance and resources by optimizing the data migration strategy.

[0058] It should be noted that although the operations of the method of the present application are described in a specific order in the above embodiments and drawings, this does not require or imply that the operations must be performed in this specific order, or that all of the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps can be omitted, multiple steps can be combined into one step, and / or one step can be divided into multiple steps.

[0059] In order to more clearly explain the above-mentioned method for optimizing seamless upgrade of large-scale HDFS storage systems, a specific embodiment will be described below, however, it should be noted that this embodiment is only for better illustration of the present application and does not constitute an improper limitation on the present application.

[0060] Embodiment:

[0061] As shown in the following steps: Figure 1

[0062] I. Customized replica placement strategy

[0063] The replica configuration strategy in Hadoop clusters can only support modification of the number of replicas, such as changing from three replicas to two replicas or four replicas, and does not support the rack situation of the replicas. In order to ensure that the three replicas of each data block are automatically distributed on three different racks, a new replica placement strategy needs to be developed.

[0064] 1. Design a new replica placement strategy

[0065] First replica selection: placed on the same rack as the data write node to reduce data write delay and transmission time.

[0066] Second replica selection: select a node with low load from a different rack than the first replica.

[0067] Third replica selection: select a node with low load from a third rack different from the racks of the first two replicas.

[0068] Fallback mechanism: when there are only two racks in the cluster, place two replicas on one rack and the other replica on the other rack to ensure that the risk of data loss is minimized.

[0069] 2. Implement the new replica placement strategy ​

[0070] (1) Dynamic Node Selection and Load Balancing

[0071] Based on real-time node status and resource usage, dynamically select the most suitable node to store the replica, ensuring the load balancing of the cluster and efficient use of resources. Implementation steps:

[0072] a. Real-time monitoring and collection of node status information:

[0073] Use Hadoop's heartbeat mechanism or integrated monitoring tools (such as Apache Ambari, Cloudera Manager) to regularly collect the status information of each DataNode (node).

[0074] Monitoring indicators include:

[0075] CPU load: the current CPU usage of the node.

[0076] Memory usage: the current memory usage of the node.

[0077] Storage space utilization: the remaining storage space of the node.

[0078] Node health status: whether the node is running normally or has faults.

[0079] Real-time aggregation of these information to NameNode for replica placement strategy decision-making.

[0080] b. Evaluate node load and available resources

[0081] Define load evaluation criteria:

[0082] Set threshold values for each indicator, for example: if the CPU usage exceeds 80% or the storage space utilization exceeds 90%, the node is considered to be in an overload state.

[0083] Calculate node load:

[0084] Use real-time data to calculate the load score of each node, sort the nodes according to the score, and identify the node with the lowest load.

[0085] c. Dynamic node selection strategy:

[0086] Select the first replica location:

[0087] Place the first replica on the rack where the source node of the data write request is located. In this rack, select the node with the lowest load and good health status to ensure the highest write efficiency.

[0088] Select the second replica location:

[0089] Exclude the rack that already contains the first replica from the remaining racks in the cluster. Among the remaining racks, select the node with the lowest load and most resources to place the second replica.

[0090] Select the location for the third replica:

[0091] Again, exclude racks that already contain replicas from the remaining racks in the cluster. Select the node with the lowest load to store the third replica, ensuring that the replicas are distributed across three different racks.

[0092] d. Realize real-time load balancing:

[0093] Periodically check and evaluate the load distribution of each node in the cluster to determine whether to trigger the redistribution of data block replicas. When the load of some nodes is significantly higher than that of other nodes, start the replica redistribution operation to migrate some replicas from high-load nodes to low-load nodes. In the redistribution operation, the principle of distributing three replicas across three different racks is still followed to ensure high availability of data.

[0094] e. Optimize node selection combined with network topology information:

[0095] When selecting a node each time, not only consider the load and resource conditions of the node, but also combine network topology information (such as network bandwidth, delay between racks, etc.) to make decisions. Prefer to choose paths with good network conditions to reduce data transmission delay and bandwidth occupation, and improve the overall performance of the cluster.

[0096] f. Dynamically adjust replica placement strategy:

[0097] When the state of the cluster changes (such as node failure, rack load fluctuation, etc.), the strategy can automatically adjust the replica placement decision. For example, when the load of a rack becomes too high, it can reselect nodes on other racks to store new written data replicas to avoid excessive concentration.

[0098] 3、Integrate and configure new placement strategy

[0099] (1) Put the newly developed replica placement strategy class file into the source code directory of Hadoop.

[0100] The source code directory of Hadoop usually contains multiple modules and packages, each corresponding to a different function. The new policy class file needs to be placed in the same package as the existing policy class file to ensure that it can be correctly referenced and compiled. Place the policy class file in the org.apache.hadoop.hdfs.server.blockmanagement package (a Java package in the Hadoop project that manages functions such as block allocation, storage, and replica placement policies in the Hadoop Distributed File System). Copy the new policy class file to the above package directory.

[0101] (2) Modify the configuration file hdfs-site.xml of Hadoop.

[0102] In order to make Hadoop use the new replica placement policy, it is necessary to modify its configuration file hdfs-site.xml. Because it determines which policy class Hadoop will load when starting to manage the distribution of data replicas. In the installation directory of Hadoop, find the etc / hadoop folder, which usually contains the hdfs-site.xml file. This file defines many core behaviors of HDFS, including replica placement policy. Open the hdfs-site.xml file and find or add the following configuration item.

[0103] <property>

[0104] <name>dfs.block.replicator.classname< / name>

[0105] <value>com.yourcompany.hadoop.NewPlacementPolicy< / value>

[0106] < / property>

[0107] Among them, dfs.block.replicator.classname is the configuration item that specifies the replica placement policy class, and the name of the new policy class is filled in the value tag. After modification, distribute the file to all nodes in the cluster that need to use the new policy to ensure consistency of the configuration.

[0108] (3) Verify the effectiveness of the policy configuration

[0109] After completing the configuration of the policy, first verify whether the configuration takes effect and ensure that the Hadoop system has correctly loaded the new replica placement policy.

[0110] Check the correctness of the configuration file: Ensure that the correct policy class is specified in the hdfs-site.xml file, and that the modification of the file has been distributed and taken effect on all related nodes.

[0111] Check the startup log: After restarting the Hadoop cluster, immediately check the startup log of each node. The log should contain information about loading the new policy class, confirming that the new policy has been correctly loaded.

[0112] Loading information for the policy class: In the logs, information like the following indicates that Hadoop successfully found and loaded the policy class.

[0113] INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Using BlockPlacementPolicy class com.yourcompany.hadoop.NewPlacementPolicy

[0114] Sign of successful initialization: Information like the following indicates that the initialization of the policy class is complete, and it can start processing data replica placement.

[0115] INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: BlockPlacementPolicy initialized successfully

[0116] 4、Test and verify the effectiveness of the new policy

[0117] After confirming that the policy has been correctly loaded, it is necessary to comprehensively test the functionality of the new policy to ensure that it behaves as expected in actual operation. These tests include:

[0118] (1) Functional testing:

[0119] Use test scripts or tools to simulate various data write scenarios and check whether the three replicas of each data block are distributed on three different racks according to the design logic.

[0120] (2) Fault scenario testing:

[0121] Simulate various fault conditions (such as node failure, rack failure, etc.), and observe the data recovery and replica rebalancing situation.

[0122] For example, shut down all DataNodes on a rack, verify that the system can quickly detect the failure and redistribute the replicas on other racks to ensure high availability of data.

[0123] (3) Performance testing:

[0124] Test the impact of the new policy on the overall performance of the cluster, including:

[0125] Data write delay: Monitor the response time of data write operations to ensure that the new policy does not introduce significant delays.

[0126] Network Bandwidth Usage: Observe the amount of data transferred between racks to verify that the new strategy does not result in unnecessary network overhead.

[0127] Data Replication Time: Measure the data block replication time under different loads to assess the impact of the strategy on data replication performance.

[0128] Example Data:

[0129] Assume the cluster has three racks, each with two DataNode nodes.

[0130] Rack1: DataNode1, DataNode2

[0131] Rack2: DataNode3, DataNode4

[0132] Rack3: DataNode5, DataNode6

[0133] Execution Flow:

[0134] Data Write Request:

[0135] A client initiates a data write request to the cluster, and the system detects that the request originates from DataNode1, which is located in Rack1.

[0136] Replica Selection and Placement:

[0137] Selecting the First Replica: Select the least loaded DataNode1 on Rack1 for the first replica, as this is the write origin node, resulting in the lowest write latency.

[0138] Selecting the Second Replica: Exclude Rack1 and select DataNode3 on Rack2 from the remaining racks (Rack2 and Rack3) because it has lower load and is in good health.

[0139] Selecting the Third Replica: Exclude racks that already contain replicas (Rack1 and Rack2) and select DataNode5 on Rack3, which has the lowest load.

[0140] Results:

[0141] The three replicas of the data block are distributed across DataNode1 on Rack1, DataNode3 on Rack2, and DataNode5 on Rack3, meeting the three-rack distribution requirement of the new strategy.

[0142] Load Balancing and Optimization:

[0143] In daily operation, the load of nodes is periodically evaluated. When it is found that the load of nodes on Rack2 is too high, some data block replicas may be migrated from DataNode3 to nodes with lower load on Rack1 or Rack3, such as DataNode2 or DataNode6, to achieve better load balancing.

[0144] II. Optimization of NameNode replica replication logic

[0145] During the upgrade of a Hadoop cluster, the temporary downtime of DataNode services can cause a replica shortage, resulting in unnecessary replica replication operations. In order to improve the efficiency and stability of the system, the optimized NameNode replica replication logic needs to cover the following improvements:

[0146] 1. Analysis of replica shortage scenarios

[0147] Even a short DataNode downtime can lead to the following scenarios where replicas are insufficient:

[0148] (1) Rack failure or rack-wide maintenance

[0149] In some cases, it may be necessary to maintain or upgrade the DataNode nodes in an entire rack. If two replicas in these nodes are unavailable at the same time, and the node where the other replica is located also fails, this will result in a data block replica shortage.

[0150] Example: Suppose the three replicas of data block X are stored on different nodes in Rack1, Rack2, and Rack3. If the nodes on Rack1 are down due to a short upgrade, and the nodes on Rack2 are also offline due to an unexpected failure at the same time, only one replica remains on Rack3, resulting in a replica shortage.

[0151] (2) Network isolation or temporary network failure

[0152] The network in the cluster may experience temporary partitioning or isolation issues, causing the NameNode to be unable to communicate with certain DataNodes. If these nodes experience such network problems during a short downtime, the NameNode may not be able to obtain heartbeat information from these nodes in time, mistakenly considering these replicas as unavailable.

[0153] Example: If all nodes in Rack1 are unable to communicate with the NameNode due to network isolation, and these nodes happen to be marked as "expected short downtime" during the upgrade, the NameNode may not be able to correctly determine the availability of these replicas, triggering replica replication operations.

[0154] (3) Multiple nodes restart simultaneously

[0155] In a large-scale upgrade, multiple nodes might be restarted simultaneously. The replicas on these nodes might be temporarily unavailable. If the NameNode detects that all replicas of a certain data block are unavailable (e.g. the nodes of two replicas are restarted simultaneously, and the node of the third replica is being upgraded) during this period, it will trigger a replica replication.

[0156] Example: The replicas of data block Y are distributed in DataNode1 (Rack1), DataNode3 (Rack2), and DataNode5 (Rack3). If DataNode1 and DataNode3 are restarted simultaneously, and DataNode5 is performing a short upgrade, the NameNode will not be able to access any replica, resulting in an under-replicated state.

[0157] (4) Hardware failure overlaps

[0158] A short outage might overlap with other unexpected hardware failures. For example, a node is scheduled for an in-planned outage for upgrade, but a disk failure or memory corruption occurs, making it impossible for the system to recover data smoothly, resulting in an under-replicated state.

[0159] Example: DataNode2 (Rack2) is scheduled for an outage for maintenance, but due to a disk failure, the outage time is extended. If DataNode4 (Rack1) experiences a memory failure and cannot be recovered during this period, even though DataNode6 (Rack3) is still online, the system will determine that the data block is under-replicated.

[0160] 2、Optimized data migration strategy

[0161] (1) Introduce an expected short outage state

[0162] Add a new state identifier to the DataNode: expected short outage. When the cluster is deactivating a DataNode during an upgrade, the node can be marked as expected short outage in advance.

[0163] When the NameNode detects this state, it will determine that it is a temporary outage within expectations, and will not immediately trigger a replica replication.

[0164] (2) Adjust the replica replication trigger condition

[0165] In the standard HDFS, replica replication is usually based on the absence of DataNode heartbeat signals to determine whether the node is failed. After optimization, the NameNode will first check whether the node is in the expected short outage state before deciding to trigger replica replication.

[0166] If the node is in this state, the NameNode will wait for a certain period of time (e.g., 5 minutes) before checking the node's status update again. If the node resumes normal operation within the waiting time, the NameNode will skip the replica replication operation for this node.

[0167] If the waiting time exceeds the threshold (e.g., the node cannot resume within 10 minutes), the NameNode will trigger replica replication, but the priority of replica replication will be graded according to the situation to alleviate network and storage pressure.

[0168] (3) Graded replication strategy

[0169] According to the severity of different scenarios, a graded replication strategy is implemented. During the expected short downtime, if the above replica shortage situation is detected, the NameNode will trigger emergency replica replication, but the priority of replication will be graded according to the situation.

[0170] High priority: If multiple replica shortage situations occur (such as network isolation or multiple node restarts), the NameNode will immediately trigger replica replication to ensure data safety.

[0171] Low priority: For the case of only single node short downtime or hardware failure, the NameNode will delay replica replication and try to recover the node within a short time to reduce unnecessary resource consumption.

[0172] (4) Gradual data migration

[0173] To avoid excessive consumption of network and storage resources during the upgrade, the optimized NameNode will adopt a gradual data migration strategy. Replica replication will be done in stages, with a certain percentage of data blocks being replicated in each stage, rather than all replicas being replicated at once.

[0174] For example, during peak hours, the NameNode will limit the speed of replica replication and only migrate replicas of critical data blocks; during low-peak hours, the system will speed up replica replication to ensure high availability of data within the shortest time.

[0175] (5) Replica location priority

[0176] When implementing replica replication, those replicas distributed on low-load racks or nodes are preferred for migration. This can effectively avoid the problem of excessive local load caused by replica replication. The system will first evaluate all potential target nodes and select the optimal node according to its current load situation. For example, when selecting target nodes, those nodes with CPU usage below 50% and sufficient storage space are preferred.

[0177] Example data:

[0178] Suppose one of the racks in the cluster (Rack2) is undergoing planned short downtime maintenance, while another rack in the cluster (Rack1) experiences network isolation issues.

[0179] (1) Initial state:

[0180] Copy distribution of data block Z: DataNode2 (Rack2), DataNode4 (Rack1), DataNode6 (Rack3)

[0181] (2) Fault process:

[0182] DataNode2 on Rack2 is temporarily down and marked as an expected short stop.

[0183] At the same time, Rack1 experiences network isolation issues, causing DataNode4 to be unable to communicate with the NameNode.

[0184] The NameNode can only access the DataNode6 copy on Rack3 at this time, determining that the data block Z copy is insufficient.

[0185] (3) Optimized processing process:

[0186] The NameNode first triggers high-priority copy replication, copying the data block Z copy from DataNode6 to a node with lower load (such as DataNode5 on Rack3), to ensure high availability of data. (At this time, the system will not immediately replicate a third copy, but will wait until the cluster load decreases or enters a business low period before replicating the third copy)

[0187] The replication operation is performed in a gradual manner to avoid excessive consumption of network and storage resources.

[0188] After the nodes on Rack2 are restored, the NameNode continues to monitor the cluster load and further balances the distribution of data block copies when the network load decreases.

[0189] (4) Final state:

[0190] The copy of data block Z is restored to three and distributed on three different racks, ensuring high availability of data and stable operation of the system.

[0191] Based on the same inventive concept, the present application also proposes a system for optimizing seamless upgrade of large-scale HDFS storage system. The implementation of the system can refer to the implementation of the above method, and the repeated parts will not be described here. The term "module" used below can be a combination of software and / or hardware designed with predetermined functions. Although the system described in the following embodiments is preferably designed in software, the design of hardware or a combination of software and hardware is also possible and contemplated.

[0192] Figure 4 is a schematic diagram of the system structure of the present application for optimizing seamless upgrade of large-scale HDFS storage system. As shown in Figure 4 , the system comprises:

[0193] The replica placement strategy customization module 101 is used to customize a new replica placement strategy, so that three replicas of each data block are automatically distributed on three different racks in the cluster; when the cluster state changes, the replica placement strategy is automatically adjusted.

[0194] Customizing a new replica placement strategy includes:

[0195] First replica selection: placed on the same rack as the data write node;

[0196] Second replica selection: select a node with lower load from a rack different from the first replica;

[0197] Third replica selection: select a node with lower load from a third rack different from the racks where the first two replicas are located;

[0198] Fallback mechanism: when there are only two racks in the cluster, place two replicas on one rack and the other replica on the other rack.

[0199] According to the real-time collected node state information, evaluate the node load and available resources, and dynamically select the node most suitable for storing replicas; combine network topology information to optimize node selection;

[0200] Periodically check and evaluate the load distribution of each node in the cluster to determine whether to trigger replica redistribution operation; in the redistribution operation, the principle of distributing three replicas on three different racks is still followed.

[0201] The replica replication logic optimization module 102 is used to optimize the replica replication logic of NameNode, so that it can identify the expected temporary stop state of DataNode during the upgrade; during the expected temporary stop, implement a hierarchical replication strategy according to the severity of different scenarios, and combine a gradual data migration strategy.

[0202] When the cluster deactivates a certain DataNode during the upgrade, the node can be marked in advance as an expected temporary stop state;

[0203] The NameNode checks whether the node is in the expected temporary stop state before triggering the replica replication;

[0204] When the NameNode implements the replica replication, the replicas distributed on low-load racks or nodes are preferentially selected for migration.

[0205] It should be noted that although several modules of the system for optimizing seamless upgrade of large-scale HDFS storage systems are mentioned in the foregoing detailed description, such division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided into embodiments by multiple modules.

[0206] Based on the foregoing inventive concept, as Figure 5 shown, the present application also proposes a computer device 200, comprising a memory 210, a processor 220, and a computer program 230 stored on the memory 210 and executable on the processor 220, wherein the processor 220 executes the computer program 230 to implement the foregoing method for optimizing seamless upgrade of large-scale HDFS storage systems.

[0207] Based on the foregoing inventive concept, the present application also proposes a computer-readable storage medium, which stores a computer program for executing the foregoing method for optimizing seamless upgrade of large-scale HDFS storage systems.

[0208] The method and system for optimizing seamless upgrade of large-scale HDFS storage systems proposed by the present application have the following highlights:

[0209] 1. By custom-developing a new replica placement strategy, the risk of data loss due to rack-level failures is greatly reduced, and the reliability and security of data are improved.

[0210] 2. Support for automatically adjusting replica placement decisions when cluster state changes (such as node failure or rack load fluctuations). This flexible response mechanism ensures that the system can always maintain high efficiency and stability under different operating conditions.

[0211] 3. Optimizing the replica replication logic of the NameNode enables it to recognize the state of the DataNode during the upgrade, avoiding unnecessary replica replication operations and thus reducing system load.

[0212] 4. The hierarchical replication strategy is implemented according to the scene severity, and the consumption of network and storage resources is effectively reduced by combining with the progressive data migration mode, so as to ensure the high availability of data and the overall performance of the system.

[0213] Although the spirit and principles of the present application have been described with reference to several specific embodiments, it should be understood that the present application is not limited to the disclosed specific embodiments, and the division of aspects does not mean that the features in these aspects cannot be combined for benefit, but only for the convenience of expression. The present application is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the appended claims.

[0214] The scope of protection of the present application should be understood by those skilled in the art that various modifications or changes can be made to the technical solutions of the present application without departing from the spirit and principles of the present application, and such modifications or changes should be within the scope of protection of the present application.

Claims

1. A method for optimizing seamless upgrades of large-scale HDFS storage systems, characterized in that, The method includes: Define a new replica placement strategy so that the three replicas of each data block are automatically distributed across three different racks in the cluster; automatically adjust the replica placement strategy when the cluster state changes; the new replica placement strategy includes: First replica selection: Place it on the same rack as the data write node; Second replica selection: Select a node with lower load from a different rack than the first replica; Third replica selection: Select a node with lower load from a third rack, which is different from the racks where the first two replicas are located; Rollback mechanism: When there are only two racks in the cluster, place two replicas on one rack and the other replica on the other rack; Based on real-time collected node status information, assess node load and available resources, dynamically select the most suitable node for storing replicas, and optimize node selection by combining network topology information; Regularly check and evaluate the load distribution of each node in the cluster to determine whether a replica redistribution operation needs to be triggered; during the redistribution operation, the principle of distributing three replicas on three different racks should still be followed. Optimize the NameNode's replication logic to recognize expected brief downtime of DataNodes during upgrades. During these expected brief downtimes, implement a tiered replication strategy based on the severity of different scenarios. If a replica shortage is detected during an expected brief downtime, the NameNode will trigger emergency replication, but the replication priority will be tiered according to the situation: High priority: If multiple replicas are insufficient, the NameNode will immediately trigger replication; Low priority: For cases of only a single node experiencing a short downtime or hardware failure, the NameNode will delay replication and attempt to recover the node within a short time. Combined with a progressive data migration strategy, replication will be performed in stages, replicating a certain proportion of data blocks in each stage, rather than replicating all replicas at once. When a DataNode is deactivated during a cluster upgrade, the node can be marked in advance as expected to be in a temporary stop state. Before triggering replica replication, the NameNode checks whether the node is in the expected short-term stop state. When implementing replica replication on the NameNode, replicas distributed on low-load racks or nodes are preferred for migration.

2. A system for optimizing seamless upgrades of large-scale HDFS storage systems, characterized in that, The system includes: The replica placement strategy customization module is used to define new replica placement strategies, automatically distributing the three replicas of each data block across three different racks in the cluster; automatically adjusting the replica placement strategy when the cluster state changes; and customizing new replica placement strategies, including: First replica selection: Place it on the same rack as the data write node; Second replica selection: Select a node with lower load from a different rack than the first replica; Third replica selection: Select a node with lower load from a third rack, which is different from the racks where the first two replicas are located; Rollback mechanism: When there are only two racks in the cluster, place two replicas on one rack and the other replica on the other rack; Based on real-time collected node status information, assess node load and available resources, dynamically select the most suitable node for storing replicas, and optimize node selection by combining network topology information; Regularly check and evaluate the load distribution of each node in the cluster to determine whether a replica redistribution operation needs to be triggered; during the redistribution operation, the principle of distributing three replicas on three different racks should still be followed. The replica replication logic optimization module optimizes the NameNode's replica replication logic, enabling it to recognize expected brief downtime of DataNodes during upgrades. During these expected brief downtimes, a tiered replication strategy is implemented based on the severity of different scenarios. If a replica shortage is detected during an expected brief downtime, the NameNode will trigger emergency replica replication, but the replication priority will be tiered according to the situation: High priority: If multiple replicas are insufficient, the NameNode will immediately trigger replica replication; Low priority: For situations involving only a single node's brief downtime or hardware failure, the NameNode will delay replica replication and attempt to recover the node within a short period. Combined with a progressive data migration strategy, replica replication will be performed in stages, replicating a certain proportion of data blocks in each stage, rather than replicating all replicas at once. When a DataNode is deactivated during a cluster upgrade, the node can be marked in advance as expected to be in a temporary stop state. Before triggering replica replication, the NameNode checks whether the node is in the expected short-term stop state. When implementing replica replication on the NameNode, replicas distributed on low-load racks or nodes are preferred for migration.

3. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to design the method of claim 1.

4. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that performs the method of claim 1.

Images