A cluster upgrade fault handling method, device, equipment and storage medium

Abstract

The application discloses a cluster upgrade fault processing method and device, equipment and storage medium, wherein the method comprises the following steps: judging whether a fault node appears in a storage cluster upgrade process of an open source platform Kubernetes deployment; when it is determined that the fault node appears, determining whether a new node needs to replace the fault node according to role information of the fault node in the cluster; if yes, deleting a label marked on the fault node by using the open source platform, and marking the deleted label on the new node, so as to complete service replacement processing of the fault node in the cluster upgrade process. The application realizes that the related service of the fault node is smoothly migrated to the new replacement node, avoids that the upgrade time is greatly increased due to the fact that the traditional cluster can only wait for fault repair, meanwhile, the fault node does not need to be removed from the storage cluster in the upgrade process, and the cluster is avoided from being in an unstable state for a long time due to data migration.

Description

Technical Field

[0001] The present invention relates to the technical field of cluster upgrades, and in particular to a cluster upgrade fault handling method, apparatus, device, and storage medium. Background Art

[0002] Kubernetes is an open-source container orchestration platform that automates the deployment, scaling, and management of containerized applications. When a cluster is being upgraded, some components are running new versions while others are running older versions. This creates an unstable state for the cluster, and the shorter this unstable state lasts, the better. This means that cluster upgrades should be performed as quickly as possible.

[0003] Traditional storage clusters are deployed based on bare-metal servers. When a node fails during an upgrade, the upgrade process must continue based on the failed node's role in the storage cluster. If a replacement node is not necessary, the failed node can be removed from the cluster. If a replacement node is required, there is currently no simple, seamless method to achieve this. In the latter case, the upgrade process can generally only continue after the fault is repaired. This significantly increases the upgrade duration due to the failed node blocking the upgrade process. Furthermore, both scenarios require the failed node to be removed from the storage cluster, which results in data migration and significantly increases the upgrade duration.

[0004] Furthermore, when the cluster is not in an upgrade state, deleting nodes from the storage cluster (i.e., scaling the cluster down) is a normal requirement. However, if a node fails while the cluster is in an upgrade state, scaling it down is equivalent to scaling down the already failed cluster, which can make the cluster even more unstable.

[0005] Therefore, how to avoid a significant increase in upgrade time due to faulty nodes during cluster upgrades is a technical problem that urgently needs to be solved. Summary of the Invention

[0006] The main purpose of the present invention is to provide a cluster upgrade fault handling method, device, equipment and storage medium. By achieving the smooth migration of services related to the faulty node to the new replacement node, it avoids the traditional cluster having to wait for the fault to be repaired, which leads to a significant increase in the upgrade time. At the same time, the faulty node does not need to be removed from the storage cluster during the upgrade process, avoiding the cluster being in an unstable upgrade state for a long time due to data migration.

[0007] In a first aspect, the present application provides a cluster upgrade fault handling method, wherein the method comprises the steps of:

[0008] Determine whether faulty nodes occur during the upgrade of a storage cluster deployed on the open source platform Kubernetes.

[0009] When a faulty node is determined to exist, determining whether a new node is needed to replace the faulty node based on the role information of the faulty node in the cluster;

[0010] If so, the open source platform is used to delete the label marked on the failed node, and the deleted label is marked on the new node to complete the service replacement process of the failed node during the cluster upgrade process.

[0011] In conjunction with the first aspect above, as an optional implementation, when it is determined that the faulty node does not need to be replaced, the faulty node is deleted from the open source platform cluster, and the faulty node is retained in the storage cluster embedded in the open source platform cluster;

[0012] After the upgrade of the open source platform cluster is completed, the storage cluster is upgraded, and after the upgrade of the storage cluster is completed, the faulty node in the storage cluster is removed.

[0013] In combination with the first aspect above, as an optional implementation, during the storage cluster upgrade process, when a faulty node is detected, the detected faulty node is filtered.

[0014] In combination with the first aspect above, as an optional implementation, the open source platform sequentially deletes the labels marked on the faulty nodes, and after sequentially deleting the labels marked on the faulty nodes, sequentially marks the new nodes expanded in the open source platform cluster with the labels deleted by the faulty nodes;

[0015] When the open source platform detects the newly added label of the new node, it automatically starts the corresponding service on the new node to complete the smooth migration of the related services of the failed node to the new node.

[0016] In conjunction with the first aspect above, as an optional implementation method, whether an upgrade failure occurs is determined based on the upgrade time and error information of the storage cluster deployed on the open source platform Kubernetes;

[0017] When it is determined that an upgrade failure occurs, it is determined that a faulty node exists in the storage cluster deployed on the open source platform.

[0018] In combination with the first aspect above, as an optional implementation method, when it is determined that there is a faulty node, all nodes are troubleshooted based on the connectivity relationship between each node and the server or the fault signal to determine the faulty node, where the fault signal includes: mainboard fault signal, CPU fault signal and network card fault signal.

[0019] In combination with the first aspect above, as an optional implementation manner, determining whether the faulty node is a master node in the cluster;

[0020] When the master node is determined, it is determined that the failed node needs to be replaced with a new node;

[0021] When it is determined to be a non-primary node, it is determined that there is no need to replace the failed node with a new node.

[0022] In a second aspect, the present application provides a cluster upgrade fault handling device, the device comprising:

[0023] A judgment module is used to determine whether a faulty node occurs during the upgrade of a storage cluster deployed on the open source platform Kubernetes;

[0024] A determination module, configured to determine, when a faulty node is determined to have occurred, whether the faulty node needs to be replaced with a new node based on the role information of the faulty node in the cluster;

[0025] The execution module is used to use the open source platform to delete the label marked on the faulty node and mark the deleted label on the new node to complete the service replacement processing of the faulty node during the cluster upgrade process.

[0026] In combination with the above second aspect, as an optional implementation method,

[0027] In a third aspect, the present application further provides an electronic device comprising: a processor; and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the method described in any one of the first aspects is implemented.

[0028] In a fourth aspect, the present application further provides a computer-readable storage medium storing computer program instructions, which, when executed by a computer, enables the computer to execute any one of the methods described in the first aspect.

[0029] The present application provides a cluster upgrade fault handling method, apparatus, equipment and storage medium, wherein the method includes the steps of: determining whether a faulty node occurs during the upgrade of the storage cluster deployed by the open source platform Kubernetes; when it is determined that a faulty node occurs, determining whether the faulty node needs to be replaced with a new node based on the role information of the faulty node in the cluster; if so, using the open source platform to delete the label marked on the faulty node, and marking the deleted label on the new node to complete the service replacement processing of the faulty node during the cluster upgrade. The present application avoids the substantial increase in upgrade time caused by the traditional cluster having to wait for fault repair by smoothly migrating services related to the faulty node to the new replacement node. At the same time, the faulty node does not need to be removed from the storage cluster during the upgrade process, avoiding the cluster being in an unstable upgrade state for a long time due to data migration.

[0030] It should be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0032] Figure 1 This is a flow chart of a cluster upgrade fault handling method provided in an embodiment of the present application;

[0033] Figure 2 A schematic diagram of a cluster upgrade fault handling device provided in an embodiment of the present application;

[0034] Figure 3 Provides a schematic diagram of replacing a faulty node in an embodiment of the present application;

[0035] Figure 4 A schematic diagram of an electronic device provided in an embodiment of the present application;

[0036] Figure 5 A schematic diagram of a computer-readable program medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. In the following description, when referring to the drawings, like numbers in different figures represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present invention. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present invention, as detailed in the appended claims.

[0038] Furthermore, the drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. Some of the blocks shown in the drawings are functional entities that do not necessarily correspond to physically or logically separate entities.

[0039] The embodiments of the present application are further described in detail below with reference to the accompanying drawings.

[0040] Reference Figure 1 , Figure 1 The figure shows a flow chart of a cluster upgrade fault handling method provided by the present invention. Figure 1 As shown, the method includes the steps of:

[0041] Step S101: Determine whether a faulty node occurs during the upgrade of the storage cluster deployed on the open source platform Kubernetes.

[0042] Specifically, whether an upgrade failure occurs is determined based on the upgrade time and error information of the storage cluster deployed by the open source platform Kubernetes; when it is determined that an upgrade failure occurs, it is determined that there is a faulty node in the storage cluster deployed by the open source platform.

[0043] For easier understanding, let's take an example. When the upgrade progress remains unchanged for a long time during the upgrade process, and combined with the error information of the upgrade process, determine whether the upgrade is faulty. If the upgrade fails, it means that there is a faulty node in the upgrade process, and the way to determine the faulty node is through the connectivity between each node and the server. If it is found that node A is disconnected from the server, it is determined that node A is faulty. When a motherboard fault signal is detected, it is determined that there is a faulty node in the upgrade process, and the faulty node is checked. When a CPU fault signal is detected, it is determined that there is a faulty node in the upgrade process. When a network card fault signal is currently detected, it is determined that there is a faulty node. It should be noted that the faulty node can be checked through the interaction between the node and the server, such as the inability to ping or connect to the server.

[0044] Step S102: When it is determined that a faulty node occurs, determine whether the faulty node needs to be replaced with a new node based on the role information of the faulty node in the cluster.

[0045] Specifically, it is determined whether the faulty node is a master node. If it is a master node, it is determined that the faulty node needs to be replaced with a new node. If it is a non-master node, it is determined that the faulty node does not need to be replaced with a new node.

[0046] For easier understanding, an example is given. The upgrade of a storage cluster deployed on the open source platform Kubernetes can be considered as a two-part upgrade: the upgrade of the Kubernetes cluster and the upgrade of the storage cluster. When a node fails during the upgrade process, based on the role information of the failed node in the cluster, if it is determined to be a master node, it is determined that the failed node needs to be replaced with a new node; if it is determined to be a non-master node, it is determined that the failed node does not need to be replaced with a new node. It should be noted that the Kubernetes cluster can be considered as the foundation, and the storage cluster is deployed on the basis of the Kubernetes cluster, that is, the storage cluster is part of the Kubernetes cluster. The characteristics of the Kubernetes cluster are scalable, and the storage cluster is embedded in it (the storage cluster is a component of the Kubernetes cluster). It can be understood that embedding the storage cluster in Kubernetes is to take advantage of the characteristics of Kubernetes (the characteristic is that the new node A is labeled with the same label as the failed node B, and the business of B is automatically migrated to A).

[0047] Step S103: If yes, use the open source platform to delete the label marked on the faulty node, and mark the deleted label on the new node to complete the service replacement process of the faulty node during the cluster upgrade process.

[0048] Specifically, if a faulty node is discovered and confirmed not to be the master node, there's no need to replace it with a new one. Instead, the faulty node is simply deleted from the Kubernetes cluster. This removes the node from the Kubernetes cluster, preventing it from carrying any services and allowing the upgrade process to proceed smoothly. The node remains in the storage cluster, preventing data migration and increasing the upgrade time. Once the upgrade is complete, the node is removed from the storage cluster.

[0049] If the failed node is identified as the master node, it needs to be replaced with a new node. First, the Kubernetes cluster is expanded with a new node and the new node is labeled with the same label as the failed node. The services carried by the failed node are automatically migrated to the new node. The failed node is then deleted from the Kubernetes cluster, while remaining in the storage cluster. After the upgrade is complete, the failed node is finally deleted from the storage cluster.

[0050] It should be noted that during the upgrade, both Kubernetes and the storage cluster need to be upgraded. Generally speaking, Kubernetes is upgraded first, followed by the storage cluster. However, there are special cases. For example, some components of the Kubernetes cluster may utilize the features of the storage cluster. In this case, these components will be upgraded after the storage cluster.

[0051] In both cases, the failed node must be removed from the Kubernetes cluster. This removes any information related to the failure, ensuring a successful Kubernetes cluster upgrade. The failed node remains in the storage cluster configuration, eliminating the need to remove it from the storage cluster and avoiding data migration, which significantly increases upgrade time.

[0052] During a storage cluster upgrade, the failed node will no longer be running storage cluster-related services because it lacks the corresponding tags. Therefore, the upgrade process does not need to upgrade or check the corresponding services. For upgrades related to the storage cluster and cluster node information, since the failed node still exists in the storage cluster, special processing is required to filter the processing of the failed node to ensure a successful storage cluster upgrade.

[0053] It's important to note that special handling refers to filtering out failed nodes when they are detected during the storage cluster upgrade process. This means the upgrade process involves numerous checks, as the failed node still exists in the storage cluster. If these checks are not ignored, the checks will fail. Therefore, filtering out failed nodes is necessary to ensure a successful storage cluster upgrade.

[0054] Kubernetes is an open-source container orchestration platform that automates the deployment, scaling, and management of containerized applications. For a storage cluster deployed on Kubernetes, expanding capacity requires simply adding nodes to the Kubernetes cluster and assigning appropriate labels to the nodes. Businesses are then automatically migrated to the newly expanded nodes. When deleting nodes from the Kubernetes cluster, if the nodes need to remain in the storage cluster, the process is simple, mature, and smooth.

[0055] If a faulty node needs to be replaced with a new one during the upgrade process, this application can use the Kubernetes label feature to smoothly migrate services related to the faulty node to the new replacement node, avoiding the problem of significantly increased upgrade time caused by traditional clusters having to wait for faults to be repaired.

[0056] When the cluster is not in the upgrade state, deleting nodes from the storage cluster, that is, scaling down the cluster, is a normal requirement for the cluster. However, if a node fails while the cluster is in the upgrade state, scaling down the cluster is equivalent to scaling down the failed cluster, which will make the cluster operation more unstable.

[0057] Therefore, during this application upgrade process, the failed node does not need to be removed from the storage cluster, preventing the cluster from being in an unstable state due to data migration. This also avoids scaling down the cluster when a node has already failed, thus avoiding further instability.

[0058] Reference Figure 2 , Figure 2 FIG. 1 is a schematic diagram of a cluster upgrade fault handling device provided by the present invention, as shown in FIG. Figure 2 As shown, the device includes:

[0059] Judgment module 201: It is used to determine whether a faulty node occurs during the upgrade of the storage cluster deployed on the open source platform Kubernetes.

[0060] Determination module 202: It is used to determine whether it is necessary to replace the faulty node with a new node based on the role information of the faulty node in the cluster when it is determined that a faulty node occurs.

[0061] Execution module 203: If yes, use the open source platform to delete the label marked on the faulty node, and mark the deleted label on the new node to complete the service replacement process of the faulty node during the cluster upgrade process.

[0062] Furthermore, in a possible implementation, the execution module is further configured to, when it is determined that the failed node does not need to be replaced, delete the failed node from the open source platform cluster and retain the failed node in the storage cluster embedded in the open source platform cluster;

[0063] After the upgrade of the open source platform cluster is completed, the storage cluster is upgraded, and after the upgrade of the storage cluster is completed, the faulty node in the storage cluster is removed.

[0064] Furthermore, in a possible implementation manner, a filtering module is configured to filter detected faulty nodes when a faulty node is detected during the storage cluster upgrade process.

[0065] Furthermore, in a possible implementation, the execution module is further configured to cause the open source platform to sequentially delete the labels marked on the faulty nodes, and after sequentially deleting the labels marked on the faulty nodes, sequentially mark the new nodes expanded in the open source platform cluster with the labels deleted by the faulty nodes;

[0066] When the open source platform detects the newly added label of the new node, it automatically starts the corresponding service on the new node to complete the smooth migration of the related services of the failed node to the new node.

[0067] Furthermore, in one possible implementation, the determination module is further configured to determine whether an upgrade failure occurs based on the upgrade time and error information of the storage cluster deployed on the open source platform Kubernetes;

[0068] When it is determined that an upgrade failure occurs, it is determined that a faulty node exists in the storage cluster deployed on the open source platform.

[0069] Furthermore, in a possible implementation, the determination module is also used to perform troubleshooting on all nodes based on the connectivity relationship between each node and the server or the fault signal to determine the faulty node when it is determined that there is a faulty node, wherein the fault signal includes: a mainboard fault signal, a CPU fault signal, and a network card fault signal.

[0070] Furthermore, in a possible implementation, the judgment module is further configured to judge whether the faulty node is a master node in the cluster;

[0071] When the master node is determined, it is determined that the failed node needs to be replaced with a new node;

[0072] When it is determined to be a non-primary node, it is determined that there is no need to replace the failed node with a new node.

[0073] Reference Figure 3 , Figure 3 The figure shows a schematic diagram of replacing a faulty node provided by the present invention, as shown in FIG. Figure 3 As shown:

[0074] The original node can be considered the faulty node. It's important to note that each label on a node in Kubernetes corresponds to a backend service. When Kubernetes scales out a new node, it first removes the label from the original node and then re-labels the newly scaled-out node with the same label. Upon detecting the newly added label, the Kubernetes cluster automatically starts the corresponding service on the replacement node, ensuring that the node is running the corresponding service. Upon detecting a faulty node label check, the Kubernetes cluster automatically stops the corresponding service. This allows the corresponding service to be migrated from the faulty node to a functioning node.

[0075] Furthermore, when replacing labels, they are replaced sequentially rather than all at once. That is, after label 1 is deleted, it is added to the new node, and the following labels are replaced sequentially until all labels are replaced. It's important to note that the service corresponding to one label is successfully switched before the next one is switched. Otherwise, if the services corresponding to different labels are dependent on each other, switching may cause problems.

[0076] Refer to the following Figure 4 An electronic device 400 according to this embodiment of the present invention will be described. Figure 4 The electronic device 400 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0077] like Figure 4 As shown, electronic device 400 is implemented as a general-purpose computing device. Components of electronic device 400 may include, but are not limited to, at least one processing unit 410, at least one storage unit 420, and a bus 430 connecting various system components (including storage unit 420 and processing unit 410).

[0078] The storage unit stores program codes, which can be executed by the processing unit 410, so that the processing unit 410 performs the steps according to various exemplary embodiments of the present invention described in the above “Example Method” section of this specification.

[0079] The storage unit 420 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 421 and / or a cache memory unit 422 , and may further include a read-only memory unit (ROM) 423 .

[0080] The storage unit 420 may also include a program / utility 424 having a set (at least one) of program modules 425, such program modules 425 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0081] Bus 430 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0082] The electronic device 400 may also communicate with one or more external devices (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 400, and / or any device that enables the electronic device 400 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication may occur via an input / output (I / O) interface 450. Furthermore, the electronic device 400 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 460. As shown, the network adapter 460 communicates with other modules of the electronic device 400 via a bus 430. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 400, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0083] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0084] According to the solution of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above-mentioned method of this specification is stored. In some possible implementations, various aspects of the present invention may also be implemented in the form of a program product, which includes program code. When the program product is executed on a terminal device, the program code is used to cause the terminal device to perform the steps according to various exemplary embodiments of the present invention described in the "Exemplary Methods" section of this specification.

[0085] refer to Figure 5 As shown, a program product 500 for implementing the above method according to an embodiment of the present invention is described. The program product 500 may be a portable compact disc read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0086] The program product may be implemented in any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0087] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0088] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0089] The program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0090] Furthermore, the above-described figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention and are not intended to be limiting. It is readily understood that the processes illustrated in the above-described figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0091] The foregoing is merely a list of specific embodiments of the present application, intended to enable those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the broadest scope consistent with the principles and novel features of the present application.

[0092] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

Claims

1. A cluster upgrade fault handling method, characterized in that: include: Determine whether a faulty node occurs during the upgrade of a storage cluster deployed on an open source platform; When a faulty node is determined to exist, determining whether a new node is needed to replace the faulty node based on the role information of the faulty node in the cluster; If so, the open source platform is used to delete the label marked on the failed node and mark the deleted label on the new node to complete the service replacement processing of the failed node during the cluster upgrade process. When the failed node is determined to be the primary node, the open source platform cluster expands the capacity of the new node and marks the newly expanded node with the same label of the failed node. The services carried by the failed node are automatically migrated to the newly expanded node, and then the failed node is deleted from the open source platform cluster. At the same time, the failed node still remains in the storage cluster; When it is determined that the faulty node does not need to be replaced, the faulty node is deleted from the open source platform cluster, and the faulty node is retained in the storage cluster embedded in the open source platform cluster; During the storage cluster upgrade process, when a faulty node is detected, the detected faulty node is filtered. After the open source platform cluster upgrade is completed, the storage cluster is upgraded. After the storage cluster upgrade is completed, the faulty node in the storage cluster is removed.

2. The method according to claim 1, characterized in that The step of utilizing the open source platform to delete the label marked on the faulty node and marking the deleted label on the new node to complete the service replacement process of the faulty node during the cluster upgrade process includes: The open source platform sequentially deletes the labels marked on the faulty nodes, and after sequentially deleting the labels marked on the faulty nodes, sequentially marks the new nodes expanded in the open source platform cluster with the labels deleted by the faulty nodes; When the open source platform detects the newly added label of the new node, it automatically starts the corresponding service on the new node to complete the smooth migration of the related services of the failed node to the new node.

3. The method according to claim 1, characterized in that The determining whether a faulty node occurs during the upgrade of the storage cluster deployed on the open source platform includes: Determine whether an upgrade failure has occurred based on the upgrade time and error information of the storage cluster deployed on the open source platform; When it is determined that an upgrade failure occurs, it is determined that a faulty node exists in the storage cluster deployed on the open source platform.

4. The method according to claim 3, characterized in that include: When a faulty node is determined to exist, all nodes are troubleshooted based on the connectivity between each node and the server or fault signals to determine the faulty node, where the fault signals include: mainboard fault signal, CPU fault signal and network card fault signal.

5. The method according to claim 1, wherein The determining whether it is necessary to replace the faulty node with a new node according to the role information of the faulty node in the cluster includes: Determine whether the faulty node is a master node in the cluster; When the master node is determined, it is determined that the failed node needs to be replaced with a new node; When it is determined to be a non-primary node, it is determined that there is no need to replace the failed node with a new node.

6. A cluster upgrade fault handling device, characterized in that: include: A judgment module is used to determine whether a faulty node occurs during the upgrade of a storage cluster deployed on an open source platform; A determination module, configured to determine, when a faulty node is determined to have occurred, whether the faulty node needs to be replaced with a new node based on the role information of the faulty node in the cluster; an execution module, which is used to, if yes, use the open source platform to delete the label marked on the faulty node, and mark the deleted label on the new node, so as to complete the service replacement processing of the faulty node during the cluster upgrade process, wherein when the faulty node is determined to be the primary node, the open source platform cluster expands the capacity of the new node, and at the same time marks the newly expanded node with the same label of the faulty node, and the business carried by the faulty node is automatically migrated to the newly expanded node, and then the faulty node is deleted from the open source platform cluster, while the faulty node is still retained in the storage cluster; When it is determined that the faulty node does not need to be replaced, deleting the faulty node from the open source platform cluster and retaining the faulty node in the storage cluster embedded in the open source platform cluster; The filtering module is also used to filter the detected faulty nodes when a faulty node is detected during the storage cluster upgrade process. The execution module is also used to upgrade the storage cluster after the open source platform cluster upgrade is completed, and to remove the faulty node in the storage cluster after the storage cluster upgrade is completed.

7. An electronic device, characterized in that: The electronic device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium, characterized in that The computer program instructions are stored therein, and when the computer program instructions are executed by a computer, the computer is caused to execute the method according to any one of claims 1 to 5.

Images