Version upgrade method and device of HDFS cluster, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a version upgrade method and device of an HDFS cluster, electronic equipment and a storage medium, and relate to the technical field of big data storage. The method comprises: migrating and deploying a target NameNode node included in a low-version HDFS cluster to a high-version HDFS cluster, and forming a third NameNode group in the high-version HDFS cluster by using the target NameNode node and a plurality of new NameNode nodes; the third NameNode group manages a third namespace; the low-version HDFS cluster is an HDFS cluster with a single namespace; the high-version HDFS cluster is an HDFS cluster with a plurality of namespaces and an RBF mechanism; and the target NameNode node is a to-be-active NameNode node in the low-version HDFS cluster; migrating critical business data included in the low-version HDFS cluster to the high-version HDFS cluster; migrating and deploying other NameNode nodes included in the low-version HDFS cluster to a first NameNode group in the high-version HDFS cluster; and removing the plurality of new NameNode nodes from the high-version HDFS cluster. In this way, a version upgrade with no user awareness and low risk is achieved.

Description

Technical Field

[0001] This application relates to the field of big data storage technology, and in particular to a method, apparatus, electronic device and storage medium for upgrading an HDFS cluster. Background Technology

[0002] In the field of big data storage, building enterprise-grade data lakes using the Hadoop Distributed File System (HDFS) has become a core infrastructure supporting the storage and access of massive amounts of data. With the continuous growth of business data and the evolution of technical architecture, upgrading low-version HDFS clusters with a single namespace to high-version HDFS clusters that support a federated architecture to overcome single-point performance bottlenecks and improve resource isolation and scalability has become a common requirement for enterprise technical operations and maintenance.

[0003] In the existing technology, the upgrade schemes for upgrading a low-version HDFS cluster with a single namespace to a high-version HDFS cluster that supports a federated architecture are divided into two categories: (1) Dual-cluster parallel scheme: This scheme adopts the "new and old clusters in parallel" mode. First, a brand-new federated HDFS cluster is built based on the high-version Hadoop. Multiple NameNodes are configured to manage different business namespaces. Then, the existing data in the low-version HDFS cluster is migrated in batches to the corresponding namespace of the new federated HDFS cluster through a distributed copy tool. (2) In-situ upgrade of Hadoop version followed by federation scheme: This scheme upgrades components such as DataNode and NameNode to the target high version one by one to ensure that the new and old version components are compatible and communicate. After the entire cluster has completed the version upgrade, multiple independent NameNodes are added through the Router-based Federation (RBF) mechanism. Different namespaces are divided to manage the original data and the new data. At the same time, a unified access point for multiple NameNodes is realized through client configuration or routing layer, and the transition from a single namespace to a federated architecture is gradually completed.

[0004] However, both of the above solutions have significant limitations: (1) Dual-cluster parallel solution: It is necessary to switch the main node (NameNode) for business access. The running tasks submitted by the old client will be interrupted due to the change of connection pointer, and the old client that has not adapted to the new cluster configuration needs to be modified and adjusted. During this process, the business will experience obvious service pauses, task failures or data read and write delays, which will have a particularly prominent impact on businesses with high real-time requirements and be strongly perceived by users. (2) In-place upgrade of Hadoop version and then merge into the federation solution: On the one hand, it relies on the compatibility communication between the old and new versions of components. When the target version is significantly different from the current version (e.g., major changes involving core protocols and metadata structures), the compatibility between components will decrease significantly, directly causing the upgrade process to be interrupted. Even if the upgrade is attempted through special patches or adaptation methods, hidden compatibility issues may leave system vulnerabilities. On the other hand, it is necessary to complete the version replacement of all DataNodes and NameNodes in sequence. During the upgrade process, the cluster is in an unstable state. The failure of any node upgrade may trigger a chain reaction, resulting in serious problems such as data loss and service interruption. Summary of the Invention

[0005] This application provides a method, apparatus, electronic device, and storage medium for upgrading an HDFS cluster, enabling seamless and low-risk version upgrades for users while maintaining business continuity and system security.

[0006] In a first aspect, embodiments of this application provide a method for upgrading an HDFS cluster, the method comprising: The target NameNode node in the low-version HDFS cluster is migrated and deployed to the high-version HDFS cluster, and the target NameNode node and multiple new NameNode nodes are combined to form the third NameNode group in the high-version HDFS cluster. The third NameNode group manages the third namespace. The low-version HDFS cluster is an HDFS cluster with a single namespace, and the high-version HDFS cluster is an HDFS cluster with multiple namespaces and RBF mechanism. The multiple namespaces include at least the first namespace and the second namespace. The target NameNode node is the NameNode node to be activated in the low-version HDFS cluster. Migrate critical business data contained in the lower version HDFS cluster to the higher version HDFS cluster; Migrate and deploy other NameNode nodes contained in the lower version HDFS cluster to the third NameNode group in the higher version HDFS cluster, and remove multiple new NameNode nodes from the higher version HDFS cluster.

[0007] In one optional embodiment, the target NameNode node and multiple new NameNode nodes constitute a third NameNode group in the high-version HDFS cluster, including: Introduce multiple new NameNode nodes into the high-version HDFS cluster; Configure the target NameNode node and multiple new NameNode nodes to form a third NameNode group and manage the third namespace.

[0008] In one optional embodiment, migrating critical business data contained in a lower-version HDFS cluster to a higher-version HDFS cluster includes: The business data contained in the low-version HDFS cluster is divided into critical business data and non-critical business data. Migrate non-critical business data to an object storage system, release the storage resources of the corresponding target DataNode nodes in the low-version HDFS cluster, and migrate and deploy the target DataNode nodes to a high-version HDFS cluster. Migrate and store critical business data to the target DataNode nodes contained in the high-version HDFS cluster.

[0009] In an optional embodiment, the method further includes: During the migration of non-critical business data to the object storage system, the integrity of the migration result is verified by comparing the digital fingerprints of the source data and the target data. When the migration result is verified to be successful, the storage resources of the corresponding target DataNode in the low-version HDFS cluster are released.

[0010] In one optional embodiment, migrating and deploying other NameNode nodes contained in the lower-version HDFS cluster to a third NameNode group in the higher-version HDFS cluster includes: After the migration verification of critical business data is passed, the other NameNode nodes contained in the low-version HDFS cluster will be migrated and deployed to the third NameNode group in the high-version HDFS cluster, so that the target NameNode node, multiple new NameNode nodes and other NameNode nodes constitute the third NameNode group and manage the third namespace.

[0011] In one alternative embodiment, multiple new NameNode nodes are removed from the high-version HDFS cluster, including: Once it is confirmed that the high-version HDFS cluster is running stably and the data consistency verification is passed, multiple new NameNode nodes are removed from the high-version HDFS cluster, and the cluster management configuration and resource allocation information corresponding to the multiple new NameNode nodes are updated.

[0012] Secondly, embodiments of this application also provide an HDFS cluster version upgrade device, the device comprising: The first processing module is used to migrate and deploy the target NameNode node contained in the low-version HDFS cluster to the high-version HDFS cluster, and to form a third NameNode group in the high-version HDFS cluster with the target NameNode node and multiple new NameNode nodes. The third NameNode group manages the third namespace. The low-version HDFS cluster is an HDFS cluster with a single namespace, and the high-version HDFS cluster is an HDFS cluster with multiple namespaces and RBF mechanism. The multiple namespaces include at least the first namespace and the second namespace. The target NameNode node is the NameNode node to be activated in the low-version HDFS cluster. The data migration module is used to migrate critical business data contained in a low-version HDFS cluster to a high-version HDFS cluster. The second processing module is used to migrate and deploy other NameNode nodes contained in the low-version HDFS cluster to the third NameNode group in the high-version HDFS cluster, and remove multiple new NameNode nodes from the high-version HDFS cluster.

[0013] In an optional embodiment, when the target NameNode node and multiple new NameNode nodes form the third NameNode group in the high-version HDFS cluster, the first processing module is further configured to: Introduce multiple new NameNode nodes into the high-version HDFS cluster; Configure the target NameNode node and multiple new NameNode nodes to form a third NameNode group and manage the third namespace.

[0014] In an optional embodiment, when migrating critical business data contained in a lower-version HDFS cluster to a higher-version HDFS cluster, the data migration module is further configured to: The business data contained in the low-version HDFS cluster is divided into critical business data and non-critical business data. Migrate non-critical business data to an object storage system, release the storage resources of the corresponding target DataNode nodes in the low-version HDFS cluster, and migrate and deploy the target DataNode nodes to a high-version HDFS cluster. Migrate and store critical business data to the target DataNode nodes contained in the high-version HDFS cluster.

[0015] In an optional embodiment, the data migration module is further configured to: During the migration of non-critical business data to the object storage system, the integrity of the migration result is verified by comparing the digital fingerprints of the source data and the target data. When the migration result is verified to be successful, the storage resources of the corresponding target DataNode in the low-version HDFS cluster are released.

[0016] In an optional embodiment, when migrating and deploying other NameNode nodes contained in a lower-version HDFS cluster to a third NameNode group in a higher-version HDFS cluster, the second processing module is further configured to: After the migration verification of critical business data is passed, the other NameNode nodes contained in the low-version HDFS cluster will be migrated and deployed to the third NameNode group in the high-version HDFS cluster, so that the target NameNode node, multiple new NameNode nodes and other NameNode nodes constitute the third NameNode group and manage the third namespace.

[0017] In one optional embodiment, multiple new NameNode nodes are removed from the high-version HDFS cluster, and the second processing module is further configured to: Once it is confirmed that the high-version HDFS cluster is running stably and the data consistency verification is passed, multiple new NameNode nodes are removed from the high-version HDFS cluster, and the cluster management configuration and resource allocation information corresponding to the multiple new NameNode nodes are updated.

[0018] Thirdly, embodiments of this application also provide an electronic device, including: Processor; and Stored program memory, The program includes instructions that, when executed by the processor, cause the processor to perform the version upgrade method for the HDFS cluster as described in the first aspect.

[0019] Fourthly, embodiments of this application also provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the version upgrade method for the HDFS cluster as described in the first aspect.

[0020] Fifthly, this application provides a computer program product that, when invoked by a computer, causes the computer to execute the HDFS cluster version upgrade method steps as described in the first aspect.

[0021] The beneficial effects of this application are as follows: In the HDFS cluster version upgrade method provided in this application embodiment, the target NameNode node contained in the low-version HDFS cluster is migrated and deployed to the high-version HDFS cluster, and the target NameNode node and multiple new NameNode nodes constitute a third NameNode group in the high-version HDFS cluster; wherein, the third NameNode group manages a third namespace, the low-version HDFS cluster is an HDFS cluster with a single namespace, and the high-version HDFS cluster is an HDFS cluster with multiple namespaces and RBF mechanism, the multiple namespaces include at least: a first namespace and a second namespace, the target NameNode node is the NameNode node to be activated in the low-version HDFS cluster; the critical business data contained in the low-version HDFS cluster is migrated to the high-version HDFS cluster; other NameNode nodes contained in the low-version HDFS cluster are migrated and deployed to the third NameNode group in the high-version HDFS cluster, and multiple new NameNode nodes are removed from the high-version HDFS cluster. This achieves a seamless and low-risk version upgrade for users, balancing business continuity and system security. By constructing a seamless switching mechanism, the low-version HDFS cluster is merged into the new namespace of the high-version HDFS cluster 102, enabling business tasks to continue running without modifying old clients. This completely eliminates the user's awareness of the upgrade process. At the same time, a phased and modular operation logic is adopted to disperse upgrade risks, ensuring a smooth and controllable upgrade process from a technical perspective.

[0022] Furthermore, other features and advantages of this application will be set forth in the following description and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described herein are used to provide a further understanding of this application, constitute a part of this application, and do not constitute an improper limitation of this application. In the accompanying drawings: Figure 1 This is a schematic diagram of an optional application scenario applicable to the embodiments of this application.

[0024] Figure 2 This is a schematic diagram illustrating the implementation process of an HDFS cluster version upgrade method provided in this application embodiment.

[0025] Figure 3 This is a first schematic diagram illustrating a version upgrade of an HDFS cluster, as provided in an embodiment of this application.

[0026] Figure 4 This is a second schematic diagram illustrating a version upgrade of an HDFS cluster, as provided in an embodiment of this application.

[0027] Figure 5 This is a schematic diagram of the structure of an HDFS cluster version upgrade device provided in an embodiment of this application.

[0028] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0029] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0030] It should be understood that the steps described in the method embodiments of this application may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this application is not limited in this respect.

[0031] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc., mentioned in this application are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0032] It should be noted that the terms "a" and "a plurality of" used in this application are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0033] The names of the messages or information exchanged between multiple devices in the embodiments of this application are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0034] The following explanations of some terms used in the embodiments of this application are provided to facilitate understanding by those skilled in the art.

[0035] (1) Router-based Federation (RBF): This provides a unified access point for multiple NameNodes. By introducing a component called Router, the mount table is extracted from the client, solving the problem of mount table inconsistency. Router is responsible for resolving the mount table and forwarding client requests to the correct sub-cluster. Router communicates with the StateStore service, collects the heartbeat information of the NameNode, and reports it to the StateStore.

[0036] (2) The Hadoop Distributed File System (HDFS) is used to store massive amounts of data and provide high-throughput access.

[0037] (3) NameNode (NN): The master node of HDFS, which manages file system metadata (such as directory structure, file attributes, etc.).

[0038] (4) Data Node (DN): The slave node of HDFS, responsible for storing actual data blocks and responding to read and write requests.

[0039] (5) HDFS High Availability Mode (HA): HDFS HA ​​is a high availability solution designed to solve the problem of single point of failure of NameNode in HDFS cluster. This feature was introduced in Hadoop version 2.0.0. It usually consists of multiple NameNodes, one in active state and another in standby state.

[0040] (6) Journal Node (JN): Used to synchronize NameNode metadata logs in HDFS HA ​​architecture to ensure metadata consistency.

[0041] (7) HDFS Federation Architecture: This includes multiple independent namespaces (NS) and a shared DataNode cluster. Each namespace is independent and managed by its own dedicated NameNode group. Each namespace manages its own independent file system metadata (such as directory structure, file permissions, etc.). A namespace can be understood as an independent, logical "small HDFS cluster". All DataNodes register with the NameNodes of all namespaces in the cluster, and DataNodes receive commands from all Active NameNodes.

[0042] (8) Distributed coordination service Zookeeper (ZK): used to implement functions such as automatic failover of NameNode in HDFS HA.

[0043] The design concept of the embodiments of this application is briefly introduced below: In the field of big data storage, building enterprise-grade data lakes using HDFS has become a core infrastructure supporting the storage and access of massive amounts of data. With the continuous growth of business data and the evolution of technical architecture, upgrading low-version HDFS clusters with a single namespace to high-version HDFS clusters that support a federated architecture to overcome single-point performance bottlenecks and improve resource isolation and scalability has become a common requirement for enterprise technical operations and maintenance.

[0044] In the existing technology, the upgrade schemes for upgrading a low-version HDFS cluster with a single namespace to a high-version HDFS cluster that supports a federated architecture are divided into two categories: (1) Dual-cluster parallel scheme: This scheme adopts the "new and old clusters in parallel" mode. First, a brand-new federated HDFS cluster is built based on the high-version Hadoop. Multiple NameNodes are configured to manage different business namespaces. Then, the existing data in the low-version HDFS cluster is migrated in batches to the corresponding namespace of the new federated HDFS cluster through a distributed copy tool. (2) In-situ upgrade of Hadoop version followed by integration into the federated scheme: This scheme upgrades components such as DataNode and NameNode to the target high version one by one to ensure that the new and old versions of the components are compatible and communicate. After the entire cluster has completed the version upgrade, multiple independent NameNodes are added through the RBF mechanism to divide different namespaces to manage the original data and the new data. At the same time, a unified access point for multiple NameNodes is realized through client configuration or routing layer, and the transition from a single namespace to a federated architecture is completed gradually.

[0045] However, both of the above solutions have significant limitations: (1) Dual-cluster parallel solution: It is necessary to switch the NameNode accessed by the business. The running tasks submitted by the old client will be interrupted due to the change of connection pointer, and the old client that has not adapted to the new cluster configuration needs to be modified and adjusted. During this process, the business will experience obvious service pauses, task failures or data read and write delays, which will have a particularly prominent impact on businesses with high real-time requirements and will be strongly perceived by users. (2) In-place upgrade of Hadoop version and then merge into the federation solution: On the one hand, it relies on the compatibility communication between the old and new versions of components. When the target version is significantly different from the current version (e.g., major changes involving core protocols and metadata structures), the compatibility between components will decrease significantly, directly causing the upgrade process to be interrupted. Even if the upgrade is attempted through special patches or adaptation methods, hidden compatibility issues may leave system vulnerabilities. On the other hand, it is necessary to complete the version replacement of all DataNodes and NameNodes in sequence. During the upgrade process, the cluster is in an unstable state. The failure of any node upgrade may trigger a chain reaction, resulting in serious problems such as data loss and service interruption. Moreover, after the upgrade, the federation architecture transformation needs to be connected immediately. The superposition of the two operations further amplifies the technical risks. Once a problem occurs, it is difficult to roll back quickly.

[0046] In view of this, this application provides a method, apparatus, electronic device, and storage medium for upgrading an HDFS cluster. The method involves migrating and deploying a target NameNode node from a lower-version HDFS cluster to a higher-version HDFS cluster, and forming a third NameNode group within the higher-version HDFS cluster with the target NameNode node and multiple new NameNode nodes. The third NameNode group manages a third namespace. The lower-version HDFS cluster is a single-namespace HDFS cluster, while the higher-version HDFS cluster is a multi-namespace HDFS cluster with an RBF mechanism. The multi-namespace includes at least a first namespace and a second namespace. The target NameNode node is a NameNode node from the lower-version HDFS cluster that is yet to be activated. Then, critical business data from the lower-version HDFS cluster is migrated to the higher-version HDFS cluster. Finally, other NameNode nodes from the lower-version HDFS cluster are migrated and deployed to the third NameNode group in the higher-version HDFS cluster, and multiple new NameNode nodes are removed from the higher-version HDFS cluster. In this way, a version upgrade that is seamless and low-risk for users is achieved, taking into account both business continuity and system security. By constructing a seamless switching mechanism, business tasks can continue to run without modifying old clients, completely eliminating the user's perception of the upgrade process. At the same time, the phased and modular operation logic is adopted to disperse upgrade risks, ensuring the smooth and controllable upgrade process from a technical perspective. Specifically, the beneficial effects include: (1) Achieving a "seamless" upgrade: By using the phased approach of "building a new environment first, then migrating data, and finally integrating resources", the access entry point (RBF routing layer) of the high-version HDFS cluster can be established in advance and kept stable during the long process of data migration and node integration. Furthermore, old clients can access the IP addresses of other NameNode nodes contained in the low-version HDFS cluster. Old clients do not need to modify their configurations. Their requests can be processed by the routing layer of the high-version HDFS cluster or by other NameNode nodes in the low-version HDFS cluster, thereby completely eliminating the pain point of having to stop or interrupt services in traditional upgrade schemes. (2) Significantly reduced upgrade risk: The high-risk single, full-cluster "big bang" upgrade is decoupled into multiple low-risk, reversible steps. The new environment is built by borrowing the master node first, so even if it fails, the impact is limited. This "phased and modular" operation logic allows each step to be independently verified and rolled back, which greatly improves the reliability and controllability of the entire upgrade process.(3) It makes efficient use of existing hardware resources and saves costs: The core is "migration and reuse" rather than "complete construction". It effectively integrates the existing NameNode and DataNode hardware resources of the low-version HDFS cluster into the high-version HDFS cluster, and only borrows a small number of master nodes temporarily at critical stages. This not only avoids the hardware cost of a complete new cluster, but also improves resource utilization, which meets the enterprise's demand for cost reduction and efficiency improvement. (4) It breaks through the version compatibility limitation: The upgrade is not achieved by direct communication between the old and new version components, but by migrating the data to a completely new, high-version environment. This method fundamentally avoids the common protocol and metadata compatibility problems when upgrading across major versions, making it possible to upgrade from a lower version to a higher version federated architecture.

[0047] In particular, the preferred embodiments of this application will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments of this application and the features in the embodiments can be combined with each other without conflict.

[0048] See Figure 1 As shown, it is a schematic diagram of an optional application scenario applicable to the embodiments of this application. The application scenario may include: a low-version HDFS cluster 101, a high-version HDFS cluster 102, and an object storage system 103.

[0049] In this embodiment, the low-version HDFS cluster 101 is an HDFS cluster containing a single namespace and multiple NameNode nodes, such as Hadoop 3.2; the high-version HDFS cluster 102 is a federated HDFS cluster that has been stably running in at least two namespaces and has established an RBF mechanism, such as federated cluster version 3.4. With business expansion, the low-version HDFS cluster 101 faces problems such as insufficient architectural elasticity and limited version iteration. To achieve cluster resource integration, improve architectural scalability, and reduce operation and maintenance costs, it is planned to merge the low-version HDFS cluster 101 into the newly added namespace (i.e., the third namespace) of the high-version HDFS cluster 102. The object storage system 103 can be BOS.

[0050] In this embodiment, the target NameNode node in the low-version HDFS cluster 101 is migrated and deployed to the high-version HDFS cluster 102, and the target NameNode node and multiple new NameNode nodes constitute a third NameNode group in the high-version HDFS cluster 102. The third NameNode group manages a third namespace, and the target NameNode node is the NameNode node to be activated in the low-version HDFS cluster 101. Non-critical business data in the low-version HDFS cluster 101 is migrated and stored in the object storage system 102, and the storage resources of the corresponding target DataNode node in the low-version HDFS cluster 101 are released. The target DataNode node is then migrated and deployed to the high-version HDFS cluster 102. Critical business data in the low-version HDFS cluster 101 is migrated to the high-version HDFS cluster 102. Other NameNode nodes in the low-version HDFS cluster 101 are migrated and deployed to the third NameNode group in the high-version HDFS cluster 102, and multiple new NameNode nodes are removed from the high-version HDFS cluster. Meanwhile, low-version HDFS cluster 101 and high-version HDFS cluster 102 will archive infrequently accessed data to object storage system 103.

[0051] The following describes the HDFS cluster version upgrade method provided by the exemplary embodiments of this application in conjunction with the above application scenarios and with reference to the accompanying drawings. It should be noted that the above system architecture is only shown for the purpose of understanding the spirit and principles of this application, and the implementation of this application is not limited in any way in this respect.

[0052] See Figure 2 The diagram shown illustrates the implementation flow of an HDFS cluster version upgrade method provided in this application embodiment. The specific implementation flow of this method is as follows: S20: Migrate and deploy the target NameNode node contained in the low-version HDFS cluster to the high-version HDFS cluster, and form the target NameNode node and multiple new NameNode nodes into the third NameNode group in the high-version HDFS cluster.

[0053] The third NameNode group manages the third namespace. The low-version HDFS cluster is an HDFS cluster with a single namespace, while the high-version HDFS cluster is an HDFS cluster with multiple namespaces and RBF mechanism. The multiple namespaces include at least the first namespace and the second namespace. The first namespace corresponds to the first NameNode group, and the second namespace corresponds to the second NameNode group. The target NameNode node is the NameNode node to be activated in the low-version HDFS cluster.

[0054] In this embodiment, the target NameNode node contained in the low-version HDFS cluster is first migrated and deployed to the high-version HDFS cluster, and the target NameNode node and multiple new NameNode nodes constitute the third NameNode group in the high-version HDFS cluster.

[0055] For example, see Figure 3 The diagram shown is a first schematic diagram of the version upgrade of the HDFS cluster in this embodiment of the application. It is assumed that the low-version HDFS cluster contains NameNode1 (NN1), NameNode2 (NN2), NameNode3 (NN3), JournalNode4 (JN4), and JournalNode5 (JN5). NN1 and NN2 belong to the first data center, NN3 and JN4 belong to the second data center, JN5 belongs to the third data center, and NN2 is the active NameNode node. Then, NN1 or NN3 is used as the target NameNode node to migrate and deploy to the high-version HDFS cluster. For example, NN1 is used as the target NameNode node to migrate and deploy to the high-version HDFS cluster.

[0056] Optionally, in this embodiment of the application, a possible implementation is provided for forming a third NameNode group in a high-version HDFS cluster by the target NameNode node and multiple new NameNode nodes, specifically by performing the following operations: S200: Introduces multiple new NameNode nodes into the high-version HDFS cluster.

[0057] The number of new NameNode nodes can be two or more, and this application embodiment does not impose any restrictions on this.

[0058] In this embodiment of the application, while migrating and deploying the target NameNode node contained in the low-version HDFS cluster to the high-version HDFS cluster, multiple new NameNode nodes are borrowed simultaneously.

[0059] For example, such as Figure 3 As shown, two new NameNode nodes are borrowed simultaneously: NameNode6 (NN6) and NameNode7 (NN7). At this time, NN1 and NN6 belong to the first data center and NN7 belongs to the second data center in the high-version HDFS cluster.

[0060] S201: Configure the target NameNode node and multiple new NameNode nodes to form a third NameNode group and manage the third namespace.

[0061] The configuration operations include: basic network configuration, permission mapping, and metadata initialization.

[0062] In this embodiment, the purpose of the basic network configuration is to ensure that the newly formed third NameNode group can communicate normally with the existing infrastructure of the high-version HDFS cluster. Specific operations include: (1) configuring network settings (e.g., IP address, hostname, DNS mapping, etc.) that conform to the high-version HDFS cluster specifications for the target NameNode node and multiple new NameNode nodes; (2) opening necessary ports to ensure that the target NameNode node and multiple new NameNode nodes can communicate with the following components of the high-version HDFS cluster: Router, ZooKeeper, JournalNode, and the shared DataNode cluster. The purpose of permission mapping is to solve the authentication and authorization problems during cross-cluster integration and achieve seamless access control. The purpose of metadata initialization is to establish an initial file system structure on the third namespace to prepare for receiving data.

[0063] This successfully merged the lower-version HDFS cluster into the new namespace of the higher-version HDFS cluster, ensuring the high availability of the new namespace. By introducing multiple new master nodes and configuring them together with the target master node being migrated, a third NameNode group was formed, successfully joining the higher-version HDFS cluster and ready to operate as a new, independent namespace, providing a highly reliable service foundation for businesses.

[0064] S21: Migrate critical business data contained in the low-version HDFS cluster to the high-version HDFS cluster.

[0065] Optionally, in this application embodiment, a possible implementation is provided for migrating critical business data contained in a low-version HDFS cluster to a high-version HDFS cluster, specifically by performing the following operations: S210: Divide the business data contained in the low-version HDFS cluster into critical business data and non-critical business data.

[0066] Critical business data refers to data that is highly popular, important, or frequently accessed; its importance is related to the type of business to which the data belongs. Critical business data can also be disaster recovery data.

[0067] S211: Migrate non-critical business data to an object storage system, release the storage resources of the corresponding target DataNode in the low-version HDFS cluster, and migrate and deploy the target DataNode to a high-version HDFS cluster.

[0068] In this embodiment of the application, non-critical business data is migrated and stored to an object storage system (e.g., BOS) using a distributed data migration tool (e.g., DistCp).

[0069] Optionally, in this embodiment of the application, in order to ensure the absolute reliability of data migration and thus guarantee the reliability of the upgrade, during the process of migrating non-critical business data to the object storage system, the integrity of the migration result is verified by comparing the digital fingerprints of the source data and the target data. When it is determined that the migration result verification is successful, the storage resources of the corresponding target DataNode in the low-version HDFS cluster are released, and then the target DataNode is migrated and deployed to the high-version HDFS cluster.

[0070] Specifically, in this embodiment, when verifying the integrity of the migration results, the first step is as follows: Before the migration begins, the tool scans the directories that need to be migrated in the low-version HDFS cluster and generates a file list. For each file in the file list, the tool calculates its checksum. The second step is as follows: The tool begins to copy the files from the low-version HDFS cluster to the object storage system in parallel. The third step is as follows: After the migration task is completed, the tool does not immediately consider the task successful, but starts a verification task. This verification task will traverse the migrated files again, but this time it will make a comparison: read the file that was just stored on the object storage system, calculate a new checksum for it, and compare this new checksum with the checksum of the source file generated in the first step byte by byte. The fifth step is as follows: If the checksum matches, the file has been successfully migrated; if the checksum does not match, it means that the file may have been corrupted during the migration process, and the tool will automatically try to re-migrate the file.

[0071] Optionally, in this embodiment, when migrating and deploying the target DataNode to a higher version HDFS cluster, the expansion and access are completed according to the resource requirements of the higher version HDFS cluster, and the cluster resource scheduling policy is updated to ensure that the newly added DataNode in the higher version HDFS cluster is compatible with the existing cluster components for communication. Among them, updating the cluster resource scheduling policy ensures that: (1) load balancing: the newly added DataNode can be recognized by the cluster, and the new data blocks can be written evenly to all DataNodes (including the newly added DataNode); (2) storage policy effectiveness: ensuring that data in a specific namespace can be correctly stored on the newly added nodes.

[0072] S212: Migrate and store critical business data to the target DataNode nodes contained in the high-version HDFS cluster.

[0073] In this embodiment, an incremental synchronization mechanism is used to ensure data consistency by migrating and storing critical business data to target DataNode nodes within a higher-version HDFS cluster. Similarly, in this embodiment, to ensure the absolute reliability of data migration and thus guarantee the reliability of the upgrade, the integrity of the migration result is verified by comparing the digital fingerprints of the source data and the target data during the migration of critical business data to the higher-version HDFS cluster.

[0074] For example, such as Figure 3 As shown, non-critical business data in the first data center of the low-version HDFS cluster is migrated and stored in the object storage system, releasing the storage resources of the corresponding target DataNode nodes in the low-version HDFS cluster. The target DataNode nodes are then migrated and deployed to the high-version HDFS cluster. Then, critical business data in the second data center of the low-version HDFS cluster is migrated and stored in the target DataNode nodes contained in the high-version HDFS cluster.

[0075] By migrating non-critical data to the object storage system first, storage resources of the low-version HDFS cluster can be quickly released, and the released storage resources can be accurately adapted to the high-version HDFS cluster, balancing migration efficiency and resource utilization, and effectively reducing operation and maintenance costs.

[0076] Furthermore, in this embodiment of the application, after migrating the critical business data contained in the low-version HDFS cluster to the high-version HDFS cluster, the business data stored in the high-version HDFS cluster can be archived as cold data through a data cooling strategy to optimize the cluster storage efficiency.

[0077] S22: Migrate and deploy other NameNode nodes contained in the low-version HDFS cluster to the third NameNode group in the high-version HDFS cluster, and remove multiple new NameNode nodes from the high-version HDFS cluster.

[0078] In this context, "other NameNode nodes" refers to the NameNode nodes in the lower-version HDFS cluster other than the target NameNode node. After migrating and deploying the other NameNode nodes in the lower-version HDFS cluster to the third NameNode group in the higher-version HDFS cluster, the other NameNode nodes, the target NameNode node, and multiple new NameNode nodes constitute the third NameNode group, which manages the third namespace. After removing the multiple new NameNode nodes from the higher-version HDFS cluster, the other NameNode nodes and the target NameNode node constitute the third NameNode group, which manages the third namespace.

[0079] In this embodiment, after the migration and verification of critical business data passes, other NameNode nodes in the lower-version HDFS cluster are migrated and deployed to a third NameNode group in the higher-version HDFS cluster. This allows the target NameNode node, multiple new NameNode nodes, and other NameNode nodes to form a third NameNode group and manage the third namespace. This further controls risk; the final master node integration is only performed after the critical business data migration and verification are successful. This avoids complex metadata synchronization operations when the data status is unclear, ensuring operational robustness.

[0080] Optionally, in this embodiment, when migrating and deploying other NameNode nodes in a low-version HDFS cluster to the third NameNode group in a high-version HDFS cluster, all metadata and installation environments of the other NameNode nodes are deleted. After migrating and deploying the other NameNode nodes in the low-version HDFS cluster to the third NameNode group in the high-version HDFS cluster, metadata synchronization between master nodes (synchronizing the metadata corresponding to the third NameNode group to other NameNode nodes) and routing rule configuration (registering the addresses of other NameNode nodes to the third namespace service list) are performed to achieve centralized management and control of cluster resources.

[0081] In this embodiment, once it is determined that the high-version HDFS cluster is running stably and data consistency verification has passed, multiple new NameNode nodes are removed from the high-version HDFS cluster, and the cluster management configuration and resource allocation information corresponding to the multiple new NameNode nodes are updated. This ensures timely recovery of cluster resources after the upgrade, preventing resource idleness, achieving cost savings, and updating the cluster management configuration and resource allocation information ensures the accuracy of the operation and maintenance system information.

[0082] The stable operation of the cluster refers to the third namespace acting as an independent service unit, with all its core indicators functioning normally and providing continuous and reliable service. Data consistency verification includes metadata consistency verification and data block consistency verification. Metadata consistency means that the file system directory tree (metadata) in the NameNode's memory is completely consistent with the edit logs (Edits) and snapshots (FsImage) persisted in the JournalNode, and the metadata views of all NameNodes (Active and Standby) under the third namespace are synchronized. Data block consistency verification means that the actual data blocks stored on the DataNodes completely match the block mapping relationships recorded in the NameNode's metadata. Cluster management configuration refers to configurations involving the NameNode, such as configuration files, service discovery configurations, monitoring and alarm system configurations, etc. Resource allocation information records the allocation, ownership, and lifecycle of hardware resources.

[0083] For example, see Figure 4 The diagram shown is a second illustration of an HDFS cluster version upgrade in this embodiment. It assumes that the other NameNode nodes in the lower-version HDFS cluster include: NameNode2 (NN2), NameNode3 (NN3), JournalNode4 (JN4), and JournalNode5 (JN5). The target NameNode node in the higher-version HDFS cluster is NameNode1 (NN1), and the multiple new NameNode nodes are NameNode6 (NN6) and NameNode7 (NN7). The third NameNode group in the higher-version HDFS cluster at this time includes: NN1, NN6, and N... If N7 is selected, then NN2, NN3, JN4, and JN5 will be migrated and deployed to the third NameNode group in the high-version HDFS cluster. At this time, the third NameNode group in the high-version HDFS cluster includes: NN1, NN2, NN3, JN4, JN5, NN6, and NN7. NN1, NN2, and NN6 belong to the first data center, NN3, JN4, and NN7 belong to the second data center, and JN5 belongs to the third data center. Then, NN6 and NN7 will be removed from the high-version HDFS cluster. At this time, the third NameNode group in the high-version HDFS cluster includes: NN1, NN2, NN3, JN4, and JN5. NN1 and NN2 belong to the first data center, NN3 and JN4 belong to the second data center, and JN5 belongs to the third data center.

[0084] Additionally, it is worth noting that in this embodiment, before or during the version upgrade method, the old client can read and write data to the low-version HDFS cluster based on the IP address of the low-version HDFS cluster, or it can read and write data to the high-version HDFS cluster using the first and second namespaces based on the IP address of the high-version HDFS cluster. After the version upgrade method is completed, the old client can read and write data to the high-version HDFS cluster using the third namespace based on the IP address of the low-version HDFS cluster, or it can read and write data to the high-version HDFS cluster using the first and second namespaces based on the IP address of the high-version HDFS cluster.

[0085] Based on the above embodiments, this application focuses on the goal of version upgrade of HDFS clusters that are user-unnoticed and low-risk. The innovations are concentrated in four dimensions: (1) In terms of user experience: a client-transparent routing adaptation layer is built, which can realize the smooth migration of business requests from the original cluster to the new namespace without modifying the old client, ensuring the continuous operation of tasks and completely eliminating the user's perception of the upgrade; (2) In terms of risk control: a phased architecture of "master node migration first + resource step-by-step expansion" is adopted to avoid centralized operation of the entire cluster, and a data verification and rollback mechanism is provided to significantly reduce the impact of single-step failures; (3) In terms of version compatibility breakthrough: through metadata format adaptive conversion and component communication adaptation module, the restrictions on merging low-version clusters are broken, and cross-version secure migration of data from the original low-version cluster to the new namespace of the RBF cluster is supported; (4) In terms of resource utilization optimization, innovative data hierarchical migration and node elastic scaling strategies are adopted to make the released storage resources accurately adapt to the new namespace requirements, taking into account migration efficiency and resource utilization, and effectively reducing operation and maintenance costs.

[0086] Furthermore, based on the same technical concept, embodiments of this application provide an HDFS cluster version upgrade apparatus, which is used to implement the above-described method flow of the embodiments of this application. For example, see [link to relevant documentation]. Figure 5 As shown, the HDFS cluster version upgrade device 500 may include: a first processing module 501, a data migration module 502, and a second processing module 503, wherein: The first processing module 501 is used to migrate and deploy the target NameNode node contained in the low-version HDFS cluster to the high-version HDFS cluster, and to form a third NameNode group in the high-version HDFS cluster with the target NameNode node and multiple new NameNode nodes; wherein, the third NameNode group manages the third namespace, the low-version HDFS cluster is an HDFS cluster with a single namespace, and the high-version HDFS cluster is an HDFS cluster with multiple namespaces and RBF mechanism, the multiple namespaces include at least: the first namespace and the second namespace, and the target NameNode node is the NameNode node to be activated in the low-version HDFS cluster; Data migration module 502 is used to migrate critical business data contained in a low-version HDFS cluster to a high-version HDFS cluster; The second processing module 503 is used to migrate and deploy other NameNode nodes contained in the low-version HDFS cluster to the third NameNode group in the high-version HDFS cluster, and remove multiple new NameNode nodes from the high-version HDFS cluster.

[0087] In an optional embodiment, when the target NameNode node and multiple new NameNode nodes constitute the third NameNode group in the high-version HDFS cluster, the first processing module 501 is further configured to: Introduce multiple new NameNode nodes into the high-version HDFS cluster; Configure the target NameNode node and multiple new NameNode nodes to form a third NameNode group and manage the third namespace.

[0088] In an optional embodiment, when migrating critical business data contained in a lower-version HDFS cluster to a higher-version HDFS cluster, the data migration module 502 is further configured to: The business data contained in the low-version HDFS cluster is divided into critical business data and non-critical business data. Migrate non-critical business data to an object storage system, release the storage resources of the corresponding target DataNode nodes in the low-version HDFS cluster, and migrate and deploy the target DataNode nodes to a high-version HDFS cluster. Migrate and store critical business data to the target DataNode nodes contained in the high-version HDFS cluster.

[0089] In an optional embodiment, the data migration module 502 is further configured to: During the migration of non-critical business data to the object storage system, the integrity of the migration result is verified by comparing the digital fingerprints of the source data and the target data. When the migration result is verified to be successful, the storage resources of the corresponding target DataNode in the low-version HDFS cluster are released.

[0090] In an optional embodiment, when migrating and deploying other NameNode nodes contained in a lower-version HDFS cluster to a third NameNode group in a higher-version HDFS cluster, the second processing module 503 is further configured to: After the migration verification of critical business data is passed, the other NameNode nodes contained in the low-version HDFS cluster will be migrated and deployed to the third NameNode group in the high-version HDFS cluster, so that the target NameNode node, multiple new NameNode nodes and other NameNode nodes constitute the third NameNode group and manage the third namespace.

[0091] In an optional embodiment, multiple new NameNode nodes are removed from the high-version HDFS cluster, and the second processing module 503 is further configured to: Once it is confirmed that the high-version HDFS cluster is running stably and the data consistency verification is passed, multiple new NameNode nodes are removed from the high-version HDFS cluster, and the cluster management configuration and resource allocation information corresponding to the multiple new NameNode nodes are updated.

[0092] Based on the description of the method and apparatus embodiments above, an exemplary embodiment of the present invention also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, which, when executed by the at least one processor, causes the electronic device to perform the method according to an embodiment of the present invention.

[0093] This application also provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform a method according to an embodiment of this application.

[0094] This application also provides a computer program product, including a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform a method according to an embodiment of this application.

[0095] See Figure 6The diagram illustrates a structural block diagram of an electronic device 600 that can serve as a server or client in this application, serving as an example of hardware devices applicable to various aspects of this application. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0096] like Figure 6 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. The RAM 603 may also store various programs and data required for the operation of the device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0097] Multiple components in electronic device 600 are connected to I / O interface 605, including: input unit 605, output unit 606, storage unit 608, and communication unit 609. Input unit 605 can be any type of device capable of inputting information to electronic device 600. Input unit 605 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device. Output unit 606 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 608 may include, but is not limited to, disk and optical disk. Communication unit 609 allows electronic device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers and / or chipsets, such as Bluetooth devices, WiFi devices, worldwide interoperability for microwave access (WiMax) devices, cellular communication devices, and / or the like.

[0098] The computing unit 601 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above. For example, in some embodiments, the above-described HDFS cluster version upgrade method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 600 via ROM 602 and / or communication unit 609. In some embodiments, the computing unit 601 can be configured to perform the above-described HDFS cluster version upgrade method by any other suitable means (e.g., by means of firmware).

[0099] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0100] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM) or flash memory, optical fibers, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0101] As used in this application, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device, PLD) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0102] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0103] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0104] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0105] Furthermore, it should be understood that the above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of the invention. Therefore, any equivalent variations made in accordance with the claims of this invention are still within the scope of this application.

Claims

1. A method for upgrading an HDFS cluster, characterized in that, include: The target NameNode node in the low-version HDFS cluster is migrated and deployed to the high-version HDFS cluster, and the target NameNode node and multiple new NameNode nodes constitute the third NameNode group in the high-version HDFS cluster; wherein, the third NameNode group manages the third namespace, the low-version HDFS cluster is an HDFS cluster with a single namespace, and the high-version HDFS cluster is an HDFS cluster with multiple namespaces and RBF mechanism, wherein the multiple namespaces include at least: a first namespace and a second namespace, and the target NameNode node is the NameNode node to be activated in the low-version HDFS cluster; Migrate the critical business data contained in the lower version HDFS cluster to the higher version HDFS cluster; Migrate and deploy the other NameNode nodes contained in the low-version HDFS cluster to the third NameNode group in the high-version HDFS cluster, and remove the multiple new NameNode nodes from the high-version HDFS cluster.

2. The method as described in claim 1, characterized in that, The step of forming the third NameNode group in the high-version HDFS cluster by the target NameNode node and multiple new NameNode nodes includes: Introduce the multiple new NameNode nodes into the high-version HDFS cluster; Configure the target NameNode node and the plurality of new NameNode nodes to form the third NameNode group and manage the third namespace.

3. The method as described in claim 1, characterized in that, The process of migrating critical business data from the lower-version HDFS cluster to the higher-version HDFS cluster includes: The business data contained in the low-version HDFS cluster is divided into critical business data and non-critical business data. The non-critical business data is migrated and stored in the object storage system, and the storage resources of the corresponding target DataNode in the low-version HDFS cluster are released. The target DataNode is then migrated and deployed to the high-version HDFS cluster. The critical business data will be migrated and stored in the target DataNode nodes contained in the high-version HDFS cluster.

4. The method as described in claim 3, characterized in that, The method further includes: During the migration of the non-critical business data to the object storage system, the integrity of the migration result is verified by comparing the digital fingerprints of the source data and the target data. When the migration result is verified to be successful, the storage resources of the corresponding target DataNode node in the low-version HDFS cluster are released.

5. The method as described in claim 1, characterized in that, The step of migrating and deploying other NameNode nodes contained in the low-version HDFS cluster to the third NameNode group in the high-version HDFS cluster includes: After the migration verification of the critical business data is passed, the other NameNode nodes contained in the low-version HDFS cluster are migrated and deployed to the third NameNode group in the high-version HDFS cluster, so that the target NameNode node, the multiple new NameNode nodes and the other NameNode nodes constitute the third NameNode group and manage the third namespace.

6. The method as described in claim 1, characterized in that, The removal of the multiple new NameNode nodes from the high-version HDFS cluster includes: Once it is determined that the high-version HDFS cluster is running stably and the data consistency verification is passed, the multiple new NameNode nodes are removed from the high-version HDFS cluster, and the cluster management configuration and resource allocation information corresponding to the multiple new NameNode nodes are updated.

7. A version upgrade device for an HDFS cluster, characterized in that, include: The first processing module is used to migrate and deploy the target NameNode node contained in the low-version HDFS cluster to the high-version HDFS cluster, and to form a third NameNode group in the high-version HDFS cluster with the target NameNode node and multiple new NameNode nodes; wherein, the third NameNode group manages a third namespace, the low-version HDFS cluster is an HDFS cluster with a single namespace, and the high-version HDFS cluster is an HDFS cluster with multiple namespaces and RBF mechanism, wherein the multiple namespaces include at least: a first namespace and a second namespace, and the target NameNode node is the NameNode node to be activated in the low-version HDFS cluster; The data migration module is used to migrate critical business data contained in the low-version HDFS cluster to the high-version HDFS cluster; The second processing module is used to migrate and deploy other NameNode nodes contained in the low-version HDFS cluster to the third NameNode group in the high-version HDFS cluster, and remove the multiple new NameNode nodes from the high-version HDFS cluster.

8. The apparatus as claimed in claim 7, characterized in that, When migrating critical business data contained in the low-version HDFS cluster to the high-version HDFS cluster, the data migration module is further configured to: The business data contained in the low-version HDFS cluster is divided into critical business data and non-critical business data. The non-critical business data is migrated and stored in the object storage system, and the storage resources of the corresponding target DataNode in the low-version HDFS cluster are released. The target DataNode is then migrated and deployed to the high-version HDFS cluster. The critical business data will be migrated and stored in the target DataNode nodes contained in the high-version HDFS cluster.

9. An electronic device, comprising: processor; as well as Stored program memory, The program includes instructions that, when executed by the processor, cause the processor to perform the method as described in any one of claims 1-6.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6.

Images