Method and system for optimizing seamless upgrade of large-scale HDFS (Hadoop Distributed File System) storage system

Abstract

The invention discloses a method and a system for optimizing seamless upgrade of a large-scale HDFS (Hadoop Distributed File System) storage system, and the method comprises the steps: customizing a new copy placement strategy, and enabling three copies of each data block to be automatically distributed on three different racks in a cluster; when the cluster state changes, a copy placement strategy is automatically adjusted; the copy replication logic of the NameNode is optimized, so that the NameNode recognizes the expected transient stop state of the DataNode in the upgrading period; during an expected transient stop period, a hierarchical replication strategy is implemented according to the severity of different scenes, and a progressive data migration strategy is combined. According to the method and the system, three copies of a data block are ensured to be distributed on three different racks by customizing a new copy placement strategy, so that the risk of data loss is reduced; according to the method, the transient stop state of the DataNode is marked to avoid unnecessary copy replication, and high availability and performance stability of the system in the upgrading process are guaranteed through hierarchical replication and a progressive data migration strategy, so that seamless upgrading of the large-scale HDFS storage system is realized.

Description

Technical Field

[0001] The present invention relates to the field of large-scale HDFS storage system upgrade, and in particular to a method and system for optimizing seamless upgrade of large-scale HDFS storage system. Background Art

[0002] There are some challenges in upgrading the HDFS storage system on a large scale:

[0003] 1. Data loss risk caused by DataNode service shutdown.

[0004] During the upgrade process, the operating system or software version of each node needs to be updated, which will inevitably lead to the suspension of DataNode services. Although HDFS ensures data availability through rack awareness and three-copy mechanism, there is still a risk of data loss due to the following reasons.

[0005] (1) Upgrading each server one by one takes too long: Although there is no risk of data loss when upgrading each server one by one, it is very time-consuming to upgrade and restart each storage server one by one, especially in a large-scale cluster (for example, with thousands of nodes). This method is not feasible because it will significantly slow down the upgrade process.

[0006] (2) Risk of upgrading in rack units: If upgrading in rack units, when two replicas in a rack are simultaneously unavailable, once the remaining third replica fails, it will cause permanent data loss.

[0007] 2. Performance and resource burden caused by large-scale data block copying operations.

[0008] During the upgrade process, when HDFS detects that a large number of DataNode services have stopped, it triggers the replication of data blocks. This brings the following problems:

[0009] (1) Network and storage resource consumption: The replication of a large number of data blocks will significantly increase the consumption of network bandwidth and storage resources, especially during the upgrade period. This may cause network congestion and storage device overload, further affecting other normal services.

[0010] (2) Cluster performance degradation: Data block replication operations also have a negative impact on the overall performance and response time of the cluster, making the upgrade process slower and affecting cluster availability and user experience. Summary of the invention

[0011] In order to solve the above problems existing in the upgrade process of HDFS storage system with large data scale, the present invention provides a method and system for optimizing the seamless upgrade of large-scale HDFS storage system. By customizing and developing a new replica placement strategy, it is ensured that three replicas of data blocks are distributed on three different racks to reduce the risk of data loss. The DataNode is marked as temporarily stopped to avoid unnecessary replica replication. The high availability and performance stability of the system during the upgrade process are guaranteed through hierarchical replication and progressive data migration strategies, so as to realize the seamless upgrade of large-scale HDFS storage system.

[0012] To achieve the above purpose, the present invention adopts the following technical solutions:

[0013] In one embodiment of the present invention, a method for optimizing seamless upgrade of a large-scale HDFS storage system is proposed, the method comprising:

[0014] Customize a new replica placement strategy so that three replicas of each data block are automatically distributed on three different racks in the cluster; automatically adjust the replica placement strategy when the cluster status changes;

[0015] Optimize the replica replication logic of NameNode to enable it to recognize the expected brief downtime of DataNode during the upgrade; during the expected brief downtime, implement a hierarchical replication strategy based on the severity of different scenarios, combined with a gradual data migration strategy.

[0016] Furthermore, you can customize new replica placement strategies, including:

[0017] The first replica selection is placed on the same rack as the data writing node.

[0018] Second replica selection: select a node with lower load from a different rack than the first replica;

[0019] Third replica selection: Select a node with lower load from a third rack that is different from the racks where the first two replicas are located;

[0020] Fallback mechanism: When there are only two racks in the cluster, place two replicas on one rack and the other replica on the other rack.

[0021] Furthermore, based on the node status information collected in real time, the node load and available resources are evaluated, and the most suitable node for storing replicas is dynamically selected; the node selection is optimized in combination with the network topology information;

[0022] Regularly check and evaluate the load distribution of each node in the cluster to determine whether it is necessary to trigger the replica redistribution operation; in the redistribution operation, the principle of distributing three replicas on three different racks is still followed.

[0023] Furthermore, when a cluster deactivates a DataNode during an upgrade, the node can be marked as expected short-term downtime in advance;

[0024] Before triggering the replication of the replica, the NameNode checks whether the node is in the expected short down state;

[0025] When NameNode implements replica replication, replicas distributed on low-load racks or nodes are preferentially selected for migration.

[0026] In one embodiment of the present invention, a system for optimizing seamless upgrade of a large-scale HDFS storage system is also proposed, the system comprising:

[0027] The replica placement strategy customization module is used to customize a new replica placement strategy so that three replicas of each data block are automatically distributed on three different racks in the cluster; when the cluster status changes, the replica placement strategy is automatically adjusted;

[0028] The replica replication logic optimization module is used to optimize the replica replication logic of NameNode so that it can recognize the expected short-term stop state of DataNode during the upgrade; during the expected short-term stop, a hierarchical replication strategy is implemented according to the severity of different scenarios, combined with a gradual data migration strategy.

[0029] Furthermore, you can customize new replica placement strategies, including:

[0030] The first replica selection is placed on the same rack as the data writing node.

[0031] Second replica selection: select a node with lower load from a different rack than the first replica;

[0032] Third replica selection: Select a node with lower load from a third rack that is different from the racks where the first two replicas are located;

[0033] Fallback mechanism: When there are only two racks in the cluster, place two replicas on one rack and the other replica on the other rack.

[0034] Furthermore, based on the node status information collected in real time, the node load and available resources are evaluated, and the most suitable node for storing replicas is dynamically selected; the node selection is optimized in combination with the network topology information;

[0035] Regularly check and evaluate the load distribution of each node in the cluster to determine whether it is necessary to trigger the replica redistribution operation; in the redistribution operation, the principle of distributing three replicas on three different racks is still followed.

[0036] Furthermore, when a cluster deactivates a DataNode during an upgrade, the node can be marked as expected short-term downtime in advance;

[0037] Before triggering the replication of the replica, the NameNode checks whether the node is in the expected short down state;

[0038] When NameNode implements replica replication, replicas distributed on low-load racks or nodes are preferentially selected for migration.

[0039] In one embodiment of the present invention, a computer device is also proposed, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the aforementioned method for optimizing the seamless upgrade of a large-scale HDFS storage system is designed.

[0040] In one embodiment of the present invention, a computer-readable storage medium is further proposed, wherein the computer-readable storage medium stores a computer program for executing a method for optimizing seamless upgrade of a large-scale HDFS storage system.

[0041] Beneficial effects:

[0042] 1. The present invention develops a new replica placement strategy through custom development, which greatly reduces the risk of data loss due to rack-level failures and improves data reliability and security.

[0043] 2. The present invention supports automatic adjustment of replica placement strategies when the cluster status changes (such as node failure or rack load fluctuation). This flexible response mechanism ensures that the system can always maintain high efficiency and stability under different operating conditions.

[0044] 3. The present invention optimizes the replica replication logic of NameNode so that it can identify the status of DataNode during the upgrade, avoiding unnecessary replica replication operations, thereby reducing the system load.

[0045] 4. The present invention implements a hierarchical replication strategy according to the severity of the scenario, combined with a progressive data migration method, which effectively reduces the consumption of network and storage resources and ensures high data availability and overall system performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a flow chart of a method for optimizing seamless upgrade of a large-scale HDFS storage system according to an embodiment of the present invention;

[0047] Figure 2 This is a schematic diagram of the default placement of three replicas and two racks for the replica placement strategy of the present invention;

[0048] Figure 3It is a schematic diagram of three-replica three-rack placement customized by the replica placement strategy of the present invention;

[0049] Figure 4 It is a schematic diagram of the system structure of the present invention for optimizing seamless upgrade of a large-scale HDFS storage system;

[0050] Figure 5 It is a schematic diagram of the computer device structure of the present invention. DETAILED DESCRIPTION

[0051] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and design the present invention, and are not intended to limit the scope of the present invention in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0052] Those skilled in the art will appreciate that the embodiments of the present invention may be designed as a system, system, device, method or computer program product. Therefore, the present disclosure may be specifically designed in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0053] According to an implementation of the present invention, a method for optimizing the seamless upgrade of a large-scale HDFS storage system is proposed. By customizing and developing a new replica placement strategy, it is ensured that three copies of a data block are distributed on three different racks to reduce the risk of data loss. The DataNode is marked as temporarily stopped to avoid unnecessary replica replication. The high availability and performance stability of the system during the upgrade process are guaranteed through hierarchical replication and progressive data migration strategies, so as to achieve seamless upgrade of a large-scale HDFS storage system.

[0054] The principle and spirit of the present invention are explained in detail below with reference to several representative embodiments of the present invention.

[0055] The method of optimizing seamless upgrade of a large-scale HDFS storage system of the present invention comprises:

[0056] 1. Customized replica placement strategy: Develop a new data replica placement strategy so that three replicas of each data block are automatically distributed on three different racks (the default is that three replicas of each data block are automatically distributed on two different racks, such as Figure 2 As shown), Figure 3 As shown, the risk of data loss is reduced and a policy fallback mechanism is introduced to ensure that data can still be stored safely when the conditions are not met.

[0057] 2. Optimize NameNode replica replication logic: Modify the NameNode replica replication logic to make it recognize the expected short-term stop state of DataNode during the upgrade, avoid triggering unnecessary replica replication operations, and reduce excessive consumption of cluster performance and resources by optimizing data migration strategies.

[0058] It should be noted that, although the operations of the method of the present invention are described in a specific order in the above embodiments and the accompanying drawings, this does not require or imply that the operations must be performed in the specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0059] In order to explain the above method for optimizing seamless upgrade of large-scale HDFS storage system more clearly, a specific embodiment is used for illustration below. However, it should be noted that this embodiment is only for better illustrating the present invention and does not constitute an improper limitation to the present invention.

[0060] Example:

[0061] like Figure 1 As shown, the specific implementation steps are as follows:

[0062] 1. Customizing replica placement strategies

[0063] The replica configuration strategy in the Hadoop cluster can only support the modification of the number of replicas, such as changing three replicas to two or four replicas, but does not support the racks where the replicas are located. In order to ensure that the three replicas of each data block are automatically distributed on three different racks, a new replica placement strategy needs to be developed.

[0064] 1. Design a new replica placement strategy

[0065] The first replica selection is placed on the same rack as the data writing node to reduce data writing latency and transmission time.

[0066] Second replica selection: select a less loaded node from a different rack than the first replica.

[0067] Third replica selection: Select a node with a lower load from a third rack that is different from the racks where the first two replicas are located.

[0068] Fallback mechanism: When there are only two racks in the cluster, two replicas are placed on one rack and the other replica is placed on the other rack to ensure that the risk of data loss is minimized.

[0069] 2. Implement a new replica placement strategy

[0070] (1) Dynamic node selection and load balancing

[0071] According to the real-time node status and resource usage, dynamically select the node that is most suitable for storing replicas to ensure cluster load balancing and efficient resource utilization. Implementation steps:

[0072] a. Real-time monitoring and collection of node status information:

[0073] Use Hadoop's heartbeat mechanism or integrated monitoring tools (such as Apache Ambari, Cloudera Manager) to regularly collect status information of each DataNode.

[0074] Monitoring indicators include:

[0075] CPU load: The current CPU usage of the node.

[0076] Memory usage: The current memory usage of the node.

[0077] Storage space utilization: the remaining storage space of the node.

[0078] Node health status: whether the node is running normally and whether there are any faults.

[0079] This information is aggregated to the NameNode in real time for replica placement strategy decision making.

[0080] b. Evaluate node load and available resources

[0081] Define the load assessment criteria:

[0082] Set thresholds for various indicators. For example, if the CPU usage exceeds 80% or the storage space utilization exceeds 90%, the node is considered to be overloaded.

[0083] Calculate the node load:

[0084] Use real-time data to calculate the load score of each node, sort the nodes according to the score, and identify the nodes with the lowest load.

[0085] c. Dynamic node selection strategy:

[0086] Select the first replica location:

[0087] The first replica is placed on the rack where the source node of the data write request is located. In this rack, the node with the lowest load and good health is selected to ensure the highest write efficiency.

[0088] Select the second copy location:

[0089] Exclude the rack that already contains the first replica from the remaining racks of the cluster. Among the remaining racks, choose the node with the lowest load and abundant resources to place the second replica.

[0090] Select the third copy location:

[0091] The rack that already contains the replica is again excluded from the remaining racks in the cluster. The node with the lowest load is selected to house the third replica, ensuring that the replicas are distributed across three different racks.

[0092] d. Realize real-time load balancing:

[0093] Regularly check and evaluate the load distribution of each node in the cluster to determine whether it is necessary to trigger the redistribution of data block replicas. When the load of some nodes is significantly higher than that of other nodes, initiate the replica redistribution operation to migrate some replicas from high-load nodes to low-load nodes. During the redistribution operation, the principle of distributing three replicas on three different racks is still followed to ensure high data availability.

[0094] e. Optimize node selection based on network topology information:

[0095] Each time a node is selected, not only the node load and resource conditions are considered, but also the network topology information (such as network bandwidth and latency between racks) is combined to make a decision. Paths with good network conditions are given priority to reduce data transmission latency and bandwidth usage, and improve the overall performance of the cluster.

[0096] f. Dynamically adjust the replica placement strategy:

[0097] When the cluster state changes (such as node failure, rack load fluctuation, etc.), the strategy can automatically adjust the replica placement decision. For example, when the load of a rack becomes too high, nodes on other racks can be reselected to store newly written data replicas to avoid over-concentration.

[0098] 3. Integrate and configure new placement strategies

[0099] (1) Put the newly developed replica placement strategy class file into the Hadoop source code directory.

[0100] Hadoop's source directory usually contains multiple modules and packages, each package corresponding to different functions. You need to put the new policy class file into the same package as the existing placement policy class file to ensure that it can be correctly referenced and compiled. Place the policy class file in the org.apache.hadoop.hdfs.server.blockmanagement package (a Java package in the Hadoop project, which is used to manage block allocation, storage, and replica placement strategies in the Hadoop distributed file system). Copy the new policy class file to the above package directory.

[0101] (2) Modify the Hadoop configuration file hdfs-site.xml.

[0102] In order to make Hadoop use the new replica placement strategy, you need to modify its configuration file hdfs-site.xml. This is because it determines which policy class Hadoop will load when it starts to manage the distribution of data replicas. In the Hadoop installation directory, find the etc / hadoop folder, which usually contains the hdfs-site.xml file. This file defines many core behaviors of HDFS, including the replica placement strategy. Open the hdfs-site.xml file and find or add the following configuration items.

[0103] <property>

[0104] <name> dfs.block.replicator.classname< / name>

[0105] <value> com.yourcompany.hadoop.NewPlacementPolicy< / value>

[0106] < / property>

[0107] Among them, dfs.block.replicator.classname is the configuration item for specifying the replica placement strategy class, and the value tag is filled with the name of the new strategy class. After the modification is completed, distribute the file to all nodes in the cluster that need to use the new strategy to ensure configuration consistency.

[0108] (3) Verify the validity of policy configuration

[0109] After completing the policy configuration, you first need to verify whether the configuration is effective and ensure that the Hadoop system has correctly loaded the new replica placement strategy.

[0110] Check the correctness of the configuration file: Make sure that the correct policy class is specified in the hdfs-site.xml file and that the file modification has been distributed and taken effect on all relevant nodes.

[0111] Check the startup log: After restarting the Hadoop cluster, immediately check the startup log of each node. The log should contain relevant information about loading the new policy class to confirm that the new policy has been loaded correctly.

[0112] Policy class loading information: In the log, the following information indicates that Hadoop successfully found and loaded the policy class.

[0113] INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:UsingBlockPlacementPolicy class com.yourcompany.hadoop.NewPlacementPolicy

[0114] Sign of successful initialization: The following information indicates that the initialization of the policy class has been completed and the placement of data copies can be processed.

[0115] INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy:BlockPlacementPolicy initialized successfully

[0116] 4. Test and verify the effectiveness of new strategies

[0117] After confirming that the policy has been loaded correctly, the functionality of the new policy needs to be fully tested to ensure that it performs as expected in actual operation. These tests include:

[0118] (1) Functional test:

[0119] Use test scripts or tools to simulate various data writing scenarios and check whether the three copies of each data block are distributed on three different racks according to the design logic.

[0120] (2) Fault scenario testing:

[0121] Simulate various failure scenarios (such as node failure, rack failure, etc.) and observe data recovery and replica rebalancing.

[0122] For example, shut down all DataNodes on a rack to verify whether the system can quickly detect failures and redistribute replicas on other racks to ensure high data availability.

[0123] (3) Performance test:

[0124] Test the impact of new strategies on the overall performance of the cluster, including:

[0125] Data write latency: Monitor the response time of data write operations to ensure that the new policy does not introduce significant latency.

[0126] Network bandwidth usage: Observe the amount of data transferred between racks to verify whether the new policy is causing unnecessary network overhead.

[0127] Data replication time: Measure the data block replication time under different loads and evaluate the impact of policies on data replication performance.

[0128] Example data:

[0129] Assume that the cluster has three racks, each rack has two DataNode nodes.

[0130] Rack1: DataNode1, DataNode2

[0131] Rack2: DataNode3, DataNode4

[0132] Rack3: DataNode5, DataNode6

[0133] Execution process:

[0134] Data write request:

[0135] The client initiates a data write request to the cluster, and the system detects that the source node of the request is DataNode1, which is located in Rack1.

[0136] Copy selection and placement:

[0137] Select the first replica: Select DataNode1 with the lowest load on Rack1 to store the first replica, because this is the write source node and has the lowest write latency.

[0138] Select the second replica: Excluding Rack1, among the remaining racks (Rack2 and Rack3), select DataNode3 on Rack2 because it has a lower load and is in good health.

[0139] Select the third replica: Excluding the racks that already contain replicas (Rack1 and Rack2), select DataNode5 on the remaining Rack3 because it has the lowest load.

[0140] result:

[0141] The three copies of the data block are distributed on DataNode1 of Rack1, DataNode3 of Rack2, and DataNode5 of Rack3, respectively, which meets the three-rack distribution requirements of the new strategy.

[0142] Load balancing and optimization:

[0143] In daily operation, node load is evaluated regularly. When it is found that the node load on Rack2 is too high, some data block replicas may be migrated from DataNode3 to nodes with lower load on Rack1 or Rack3, such as DataNode2 or DataNode6, to achieve better load balancing.

[0144] 2. Optimize NameNode replica replication logic

[0145] During the Hadoop cluster upgrade process, a brief DataNode service outage may cause insufficient replicas, resulting in unnecessary replica replication operations. To improve the efficiency and stability of the system, the optimized NameNode replica replication logic needs to cover the following improvements:

[0146] 1. Analysis of scenarios with insufficient copies

[0147] Even a brief DataNode outage may result in insufficient replicas in the following scenarios:

[0148] (1) Rack failure or whole rack maintenance

[0149] In some cases, it may be necessary to perform maintenance or upgrades on the DataNode nodes in the entire rack. If two replicas in these nodes are unavailable at the same time, and the node where the other replica is located also fails, this will result in insufficient data block replicas.

[0150] For example, suppose three copies of data block X are stored on different nodes of Rack1, Rack2 and Rack3. If the node on Rack1 is shut down due to a short upgrade, and the node on Rack2 is also offline due to an unexpected failure, then only Rack3 will retain the only copy, resulting in insufficient copies.

[0151] (2) Network isolation or temporary network failure

[0152] The network in the cluster may experience temporary partition or isolation problems, causing the NameNode to be unable to communicate with some DataNodes. If these nodes experience such network problems during a short downtime, the NameNode may not be able to obtain the heartbeat information of these nodes in time, and thus mistakenly believe that these replicas are unavailable.

[0153] For example, if all nodes in Rack1 cannot communicate with the NameNode due to network isolation, and these nodes happen to be marked as "expected brief stop" during the upgrade, the NameNode may not correctly determine the availability of these replicas, thereby triggering the replica replication operation.

[0154] (3) Restart multiple nodes simultaneously

[0155] In a large-scale upgrade, multiple nodes may be restarted at the same time. The replicas on these nodes may be temporarily unavailable. If the NameNode detects that all replicas of a data block are unavailable during this period (for example, the nodes where two replicas are located are restarted at the same time, and the node where the third replica is located is being upgraded), it will trigger the replica replication.

[0156] For example, the replicas of data block Y are distributed on DataNode1 (Rack1), DataNode3 (Rack2), and DataNode5 (Rack3). If DataNode1 and DataNode3 are restarted at the same time, and DataNode5 is performing a short upgrade, the NameNode will not be able to access any replicas, resulting in insufficient replicas.

[0157] (4) Hardware failure overlap

[0158] Short outages may overlap with other unexpected hardware failures. For example, a node is scheduled to be down for upgrade, and a disk failure or memory corruption occurs, making it impossible for the system to recover data smoothly, resulting in insufficient replicas.

[0159] For example: DataNode2 (Rack2) is scheduled to be shut down for maintenance, but the downtime is extended due to a disk failure. If DataNode4 (Rack1) has a memory failure during this period and cannot be recovered, then even if DataNode6 on Rack3 is still online, the system will determine that the data block replicas are insufficient.

[0160] 2. Optimized data migration strategy

[0161] (1) Introducing an expected short stop state

[0162] Add a new state flag for DataNode: expected short downtime. When a DataNode is decommissioned during the cluster upgrade process, the node can be marked as expected short downtime in advance.

[0163] When NameNode detects this state, it determines that this is an expected temporary outage and does not immediately trigger replica replication.

[0164] (2) Adjust the copy replication trigger conditions

[0165] In standard HDFS, replica replication is usually based on the absence of DataNode heartbeat signals to determine whether a node is failed. After optimization, NameNode will first check whether the node is in an expected short-term down state before deciding to trigger replica replication.

[0166] If the node is in this state, NameNode will wait for a period of time (for example, 5 minutes) before checking for node status updates. If the node resumes normal operation within the waiting time, NameNode will skip the replica replication operation for the node.

[0167] If the waiting time exceeds the threshold (for example, the node cannot be restored within 10 minutes), the NameNode will trigger replica replication, but the priority of replica replication will be graded according to the situation to alleviate network and storage pressure.

[0168] (3) Hierarchical replication strategy

[0169] A hierarchical replication strategy is implemented according to the severity of different scenarios. During an expected short outage, if the above replica shortage is detected, the NameNode will trigger emergency replica replication, but the priority of replication will be graded according to the situation.

[0170] High priority: If there are insufficient replicas (such as network isolation or multi-node restart), NameNode will immediately trigger replica replication to ensure data security.

[0171] Low priority: In the case of a short single-node outage or hardware failure, the NameNode delays replica replication and tries to recover the node in a short time to reduce unnecessary resource consumption.

[0172] (4) Gradual Data Migration

[0173] To avoid excessive consumption of network and storage resources during the upgrade, the optimized NameNode will adopt a gradual data migration strategy. Replica replication will be carried out in stages, with a certain proportion of data blocks being replicated in each stage, rather than completing the replication of all replicas at once.

[0174] For example, during peak hours, NameNode will limit the speed of replica replication and only migrate replicas of key data blocks; during off-peak hours, the system will speed up the replica replication to ensure that high data availability is restored in the shortest time.

[0175] (5) Copy location priority

[0176] When implementing replica replication, priority is given to replicas that are distributed on low-load racks or nodes for migration. This can effectively avoid the problem of local overload caused by replica replication. The system will first evaluate all potential target nodes and select the best node based on its current load. For example, when selecting a target node, priority is given to nodes with a CPU usage rate of less than 50% and sufficient storage space.

[0177] Example data:

[0178] Assume that a rack (Rack2) in the cluster is undergoing a planned short downtime maintenance, and at the same time another rack (Rack1) in the cluster has a network isolation problem.

[0179] (1) Initial state:

[0180] Replica distribution of data block Z: DataNode2 (Rack2), DataNode4 (Rack1), DataNode6 (Rack3)

[0181] (2) Failure process:

[0182] DataNode2 on Rack2 is down briefly and is marked as expected short-term downtime.

[0183] At the same time, a network isolation problem occurred in Rack1, causing DataNode4 to be unable to communicate with NameNode.

[0184] At this time, NameNode can only access the DataNode6 replica on Rack3 and determines that the replica of data block Z is insufficient.

[0185] (3) Optimized processing:

[0186] NameNode first triggers high-priority replica replication, copying the replica of data block Z from DataNode6 to a node with lower load (such as DataNode5 on Rack3) to ensure high data availability. (At this time, the system will not immediately copy the third replica, but will wait until the cluster load decreases or enters a business trough before copying the third replica)

[0187] The replication operation is performed incrementally to avoid excessive consumption of network and storage resources.

[0188] After the Rack2 node is restored, the NameNode continues to monitor the cluster load and further balances the distribution of data block replicas when the network load decreases.

[0189] (4) Final state:

[0190] The number of replicas of data block Z is restored to three and distributed on three different racks, ensuring high data availability and stable operation of the system.

[0191] Based on the same inventive concept, the present invention also proposes a system for optimizing the seamless upgrade of large-scale HDFS storage systems. The implementation of the system can refer to the implementation of the above method, and the repeated parts will not be repeated. The term "module" used below can be a combination of software and / or hardware designed for a predetermined function. Although the system described in the following embodiments is preferably designed with software, the design of hardware, or a combination of software and hardware is also possible and conceivable.

[0192] Figure 4 Schematic diagram of the system structure of the present invention for optimizing seamless upgrade of large-scale HDFS storage system. Figure 4 As shown, the system includes:

[0193] The replica placement strategy customization module 101 is used to customize a new replica placement strategy so that three replicas of each data block are automatically distributed on three different racks in the cluster; when the cluster state changes, the replica placement strategy is automatically adjusted.

[0194] Customize new replica placement strategies, including:

[0195] The first replica selection is placed on the same rack as the data writing node.

[0196] Second replica selection: select a node with lower load from a different rack than the first replica;

[0197] Third replica selection: Select a node with lower load from a third rack that is different from the racks where the first two replicas are located;

[0198] Fallback mechanism: When there are only two racks in the cluster, place two replicas on one rack and the other replica on the other rack.

[0199] Based on the node status information collected in real time, node load and available resources are evaluated, and the most suitable node for storing replicas is dynamically selected; node selection is optimized in combination with network topology information;

[0200] Regularly check and evaluate the load distribution of each node in the cluster to determine whether it is necessary to trigger the replica redistribution operation; in the redistribution operation, the principle of distributing three replicas on three different racks is still followed.

[0201] The replica replication logic optimization module 102 is used to optimize the replica replication logic of the NameNode so that it can recognize the expected short-term stop state of the DataNode during the upgrade; during the expected short-term stop, a hierarchical replication strategy is implemented according to the severity of different scenarios, combined with a progressive data migration strategy.

[0202] When a cluster stops a DataNode during an upgrade, the node can be marked as expected to be down for a short period of time in advance;

[0203] Before triggering the replication of the replica, the NameNode checks whether the node is in the expected short down state;

[0204] When NameNode implements replica replication, replicas distributed on low-load racks or nodes are preferentially selected for migration.

[0205] It should be noted that although several modules of the system for optimizing seamless upgrade of large-scale HDFS storage systems are mentioned in the above detailed description, this division is only exemplary and not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more modules described above can be concretized in one module. Conversely, the features and functions of one module described above can be further divided into multiple modules for concretization.

[0206] Based on the above invention concept, Figure 5 As shown, the present invention also proposes a computer device 200, including a memory 210, a processor 220, and a computer program 230 stored in the memory 210 and executable on the processor 220. When the processor 220 executes the computer program 230, the aforementioned method for optimizing the seamless upgrade of a large-scale HDFS storage system is designed.

[0207] Based on the aforementioned inventive concept, the present invention further proposes a computer-readable storage medium, which stores a computer program for executing the aforementioned method for optimizing seamless upgrade of a large-scale HDFS storage system.

[0208] The method and system for optimizing seamless upgrade of large-scale HDFS storage system proposed in the present invention have the following highlights:

[0209] 1. Through custom development of a new replica placement strategy, this strategy greatly reduces the risk of data loss due to rack-level failures and improves data reliability and security.

[0210] 2. Support automatic adjustment of replica placement decisions when the cluster state changes (such as node failure or rack load fluctuation). This flexible response mechanism ensures that the system can always remain efficient and stable under different operating conditions.

[0211] 3. Optimize the replica replication logic of NameNode so that it can identify the status of DataNode during the upgrade, avoid unnecessary replica replication operations, and thus reduce the system load.

[0212] 4. Implementing a hierarchical replication strategy based on the severity of the scenario, combined with a progressive data migration approach, effectively reduces the consumption of network and storage resources, ensuring high data availability and overall system performance.

[0213] Although the spirit and principle of the present invention have been described with reference to several specific embodiments, it should be understood that the present invention is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present invention is intended to cover various modifications and equivalent arrangements contained in the spirit and scope of the attached claims.

[0214] Regarding the limitation of the protection scope of the present invention, those skilled in the art should understand that, based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the protection scope of the present invention.

Claims

1. A method for optimizing seamless upgrade of a large-scale HDFS storage system, characterized in that: The method includes: Customize a new replica placement strategy so that three replicas of each data block are automatically distributed on three different racks in the cluster; automatically adjust the replica placement strategy when the cluster status changes; Optimize the replica replication logic of NameNode to enable it to recognize the expected brief downtime of DataNode during the upgrade; during the expected brief downtime, implement a hierarchical replication strategy based on the severity of different scenarios, combined with a gradual data migration strategy.

2. The method for optimizing seamless upgrade of large-scale HDFS storage system according to claim 1, characterized in that: Customize new replica placement strategies, including: The first replica selection is placed on the same rack as the data writing node. Second replica selection: select a node with lower load from a different rack than the first replica; Third replica selection: Select a node with lower load from a third rack that is different from the racks where the first two replicas are located; Fallback mechanism: When there are only two racks in the cluster, place two replicas on one rack and the other replica on the other rack.

3. The method for optimizing seamless upgrade of large-scale HDFS storage system according to claim 1, characterized in that: Based on the node status information collected in real time, node load and available resources are evaluated, and the most suitable node for storing replicas is dynamically selected; node selection is optimized in combination with network topology information; Regularly check and evaluate the load distribution of each node in the cluster to determine whether it is necessary to trigger the replica redistribution operation; in the redistribution operation, the principle of distributing three replicas on three different racks is still followed.

4. The method for optimizing seamless upgrade of large-scale HDFS storage system according to claim 1, characterized in that: When a cluster stops a DataNode during an upgrade, the node can be marked as expected to be down for a short period of time in advance; Before triggering the replication of the replica, the NameNode checks whether the node is in the expected short down state; When NameNode implements replica replication, replicas distributed on low-load racks or nodes are preferentially selected for migration.

5. A system for optimizing seamless upgrade of large-scale HDFS storage systems, characterized in that: The system includes: The replica placement strategy customization module is used to customize a new replica placement strategy so that three replicas of each data block are automatically distributed on three different racks in the cluster; when the cluster status changes, the replica placement strategy is automatically adjusted; The replica replication logic optimization module is used to optimize the replica replication logic of NameNode so that it can recognize the expected short-term stop state of DataNode during the upgrade; during the expected short-term stop, a hierarchical replication strategy is implemented according to the severity of different scenarios, combined with a gradual data migration strategy.

6. The device for optimizing seamless upgrade of large-scale HDFS storage system according to claim 5, characterized in that: Customize new replica placement strategies, including: The first replica selection is placed on the same rack as the data writing node. Second replica selection: select a node with lower load from a different rack than the first replica; Third replica selection: Select a node with lower load from a third rack that is different from the racks where the first two replicas are located; Fallback mechanism: When there are only two racks in the cluster, place two replicas on one rack and the other replica on the other rack.

7. The device for optimizing seamless upgrade of large-scale HDFS storage system according to claim 5, characterized in that: Based on the node status information collected in real time, node load and available resources are evaluated, and the most suitable node for storing replicas is dynamically selected; node selection is optimized in combination with network topology information; Regularly check and evaluate the load distribution of each node in the cluster to determine whether it is necessary to trigger the replica redistribution operation; in the redistribution operation, the principle of distributing three replicas on three different racks is still followed.

8. The device for optimizing seamless upgrade of large-scale HDFS storage system according to claim 5, characterized in that: When a cluster stops a DataNode during an upgrade, the node can be marked as expected to be down for a short period of time in advance; Before triggering the replication of the replica, the NameNode checks whether the node is in the expected short down state; When NameNode implements replica replication, replicas distributed on low-load racks or nodes are preferentially selected for migration.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 4 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the method according to any one of claims 1 to 4.

Images