Management node downtime recovery method, electronic equipment and storage medium

Abstract

The invention relates to the field of artificial intelligence, in particular to a downtime recovery method of a management node, electronic equipment and a storage medium. The method comprises the following steps: when a target management node is in a downtime state, configuring a new management node according to configuration information of the target management node; setting stains for the new management node through a preset container arrangement platform component, removing a fault node in a target cluster to which the target management node belongs, and restarting a distributed key value storage system service of the target cluster; and setting the priority of the high-availability daemon process service of the new management node as the lowest level in the target cluster, restarting the high-availability daemon process service of the new management node, and removing stains to complete the recovery action of the target management node. Therefore, the problems that in the prior art, the recovery process is complex and time-consuming, and the availability of running services and platform pages is affected are solved, and rapid and stable recovery of the management node is achieved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a method for recovering a management node from a downtime, an electronic device, and a storage medium. Background Art

[0002] In the field of artificial intelligence, large-scale AI development platforms are often built on high-availability Kubernetes (k8s) clusters to efficiently manage and schedule large-scale computing resources. As the core of a Kubernetes cluster, the management node deploys key components such as the control plane (such as the API Server (Application Programming Interface Server)), the Etcd database, the Docker engine, and high-availability components (such as Keepalived). These nodes are responsible for core functions such as resource scheduling, task orchestration, and state management. The stable operation of the management node is a prerequisite for the proper functioning of the platform. A management node failure could cause the cluster control plane to fail, impacting the entire platform. Therefore, high availability and rapid recovery capabilities of the management node are crucial for AI platform clusters.

[0003] In related technologies, there are two main methods for recovering from management node failures commonly used on the Internet: regularly backing up the management nodes; and leveraging the self-healing capabilities of container orchestration platform components.

[0004] However, the effectiveness of backup and recovery in related technologies relies on the integrity and timeliness of the backups. The self-healing capabilities of container orchestration platform components also place high demands on the initial configuration of the cluster, requiring pre-configured automatic deployment and migration mechanisms. This can lead to data loss, excessive backup times, or impacts on the availability of running services and platform pages, which urgently need to be addressed. Summary of the Invention

[0005] The present application provides a method, electronic device and storage medium for recovering a management node from downtime, so as to solve the problems in the prior art of complex and time-consuming recovery process and impact on the availability of running services and platform pages, thereby achieving rapid and stable recovery of the management node.

[0006] To achieve the above objectives, a first embodiment of the present application provides a method for recovering a management node from downtime, the method comprising the following steps: When the target management node is in a down state, configuring a new management node according to the configuration information of the target management node; Setting a taint for the new management node through a preset container orchestration platform component, removing the faulty node in the target cluster to which the target management node belongs, and restarting the distributed key-value storage system service of the target cluster; The priority of the high availability daemon service of the new management node is set to the lowest level in the target cluster, and the high availability daemon service of the new management node is restarted, and the stain is removed to complete the recovery action on the target management node.

[0007] According to the management node downtime recovery method proposed in the embodiments of the present application, when the target management node is down, a new node is configured based on the target management node, a taint is set for the new node via the container orchestration platform, the failed node in the cluster is removed, and the distributed key-value storage service is restarted. The high-availability daemon process priority of the new node is set to the lowest in the cluster, the service is restarted, the taint is removed, and recovery is completed. This solves the problems of the prior art, such as the complex and time-consuming recovery process and the impact on the availability of running services and platform pages, and achieves rapid and stable recovery of management nodes.

[0008] To achieve the above-mentioned purpose, the second aspect embodiment of the present application proposes an electronic device, comprising: a memory, a processor, and a computer program stored on the memory and runnable on the processor, wherein the processor executes the program to implement the management node downtime recovery method as described in the above embodiment.

[0009] To achieve the above-mentioned purpose, the third embodiment of the present application proposes a computer-readable storage medium on which a computer program is stored. The program is executed by a processor to implement the downtime recovery method of the management node as described in the above embodiment.

[0010] To achieve the above-mentioned objectives, the fourth embodiment of the present application proposes a computer program product, which, when executed by a processor, implements the management node crash recovery method as described in the above-mentioned embodiment.

[0011] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0013] Figure 1 A flowchart of a method for recovering a management node from downtime provided in accordance with an embodiment of the present application; Figure 2 is a flowchart of a method for configuring a new management node according to an embodiment of the present application; Figure 3A flowchart of a stain configuration method according to an embodiment of the present application; Figure 4 is a flowchart of a priority configuration method according to an embodiment of the present application; Figure 5 This is a flow chart of a distributed key-value storage system service recovery method according to one embodiment of the present application; Figure 6 is a schematic diagram of an artificial intelligence training platform according to one embodiment of the present application; Figure 7 This is a flowchart of a method for recovering a management node from downtime according to an embodiment of the present application; Figure 8 Schematic diagram of a method and apparatus for recovering a management node from downtime according to an embodiment of the present application; Figure 9 A schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0014] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0015] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0016] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0017] Those skilled in the art will understand that for AI platform clusters, the normal operation of management nodes is a prerequisite for ensuring stable platform operation. A management node failure can cause the cluster's control plane to fail, impacting the normal operation of the entire platform. For example, developers may be unable to submit new training tasks, ongoing training tasks may be interrupted, the inference service may be unable to respond to external requests, and the platform interface may become inaccessible. Therefore, high availability and rapid recovery capabilities of management nodes are crucial for AI platform clusters.

[0018] There are two key challenges in recovering from an abnormal Kubernetes cluster management node failure: One challenge is ensuring that ongoing containerized services in the cluster, such as development environments, training tasks, and inference services, remain unaffected during the recovery process. Improper operation during the recovery process can cause container restarts and service interruptions, impacting ongoing AI training and inference work and causing unnecessary losses.

[0019] Another challenge is ensuring the normal operation of the platform interface and that all other healthy nodes' services are not restarted. The platform interface is the primary interface for users to interact with the platform, and its availability directly impacts the user experience. Furthermore, unnecessary restarts of services on other healthy nodes can lead to unstable service status and affect the efficiency of the entire cluster.

[0020] In the existing technology, there are two main methods for recovering management node failures commonly used on the Internet: using backup recovery and self-healing capabilities based on k8s.

[0021] Backup recovery involves regularly backing up the management node and restoring it from the backup data in the event of a node failure. Backups include operating system images, configuration files, database data, and more. However, the drawback is that the effectiveness of backup recovery depends on the completeness and timeliness of the backups. If backups are not kept up to date, important data may be lost. Furthermore, the recovery process requires importing the backup data into the new node, which can be time-consuming, especially when dealing with large amounts of data. Furthermore, configuration conflicts and other issues can arise during the recovery process, preventing services from starting properly.

[0022] Based on the self-healing capabilities of Kubernetes: Some Kubernetes clusters utilize their self-healing capabilities to automatically redeploy control plane components on other nodes when a management node is detected to be down. However, this approach places high demands on the initial configuration of the cluster, requiring the automatic deployment and migration mechanisms to be configured in advance. Furthermore, during the automatic recovery process, problems such as improper component deployment order and resource competition may arise, leading to recovery failure or service instability. Furthermore, for some components that rely on local storage (such as Etcd), automatic migration is difficult and prone to data inconsistencies.

[0023] It can be seen that the current management node downtime recovery method has significant technical bottlenecks. Either the recovery process is complicated and time-consuming, or it easily affects the availability of running services and platform pages.

[0024] The following describes a method for recovering a management node from downtime, an electronic device, and a storage medium according to an embodiment of the present application with reference to the accompanying drawings.

[0025] Figure 1 This is a flowchart of a method for recovering a management node from downtime according to an embodiment of the present application.

[0026] For example, Figure 1 As shown in FIG, the method for recovering a management node from downtime includes the following steps: In step S101 , when the target management node is in a down state, a new management node is configured according to the configuration information of the target management node.

[0027] A target management node in a down state refers to a node that originally assumed core system management functions but is unable to continue fulfilling its pre-set management responsibilities due to hardware failure, software anomalies, network outages, or other reasons, and has entered a failed state. Configuration information refers to the set of structured parameters and settings that define the target management node's identity attributes, management responsibilities, operating rules, and coordination logic with other nodes in the system when the node is operating normally.

[0028] Optionally, in some embodiments, a new management node is configured according to the configuration information of the target management node, including: identifying the operating system of the target management node and installing the operating system for the new management node; configuring the Internet Protocol address of the new management node based on the Internet Protocol address of the target management node; generating a key pair on the new management node, and sending the public key in the key pair to the target cluster to which the target management node belongs, and configuring the new management node to communicate with other nodes in the target cluster without secrets; installing a containerized platform component with the same version as the target cluster for the new management node, and configuring the container service corresponding to the containerized platform component to start automatically when the new management node is powered on.

[0029] The operating system refers to the core system software that manages computer hardware and software resources and is the foundational platform for computer system operations. An Internet Protocol address (IP address) is a digital label assigned to a device on a network to identify its location within the network and serves as the core identifier for data communication between devices. A key pair is a pair of interrelated, complementary keys generated through a cryptographic algorithm. The key pair consists of a public key and a private key. The public key is the public portion of the key pair, while the private key is the private portion. A target cluster is a collection of multiple nodes that is originally managed and coordinated by the target management node. Containerized platform components are a collection of core software modules and tools that comprise the containerized platform. Together, they support the full lifecycle of container creation, deployment, operation, management, and scheduling. A container service is a background service process or program generated by the runtime of a containerized platform component to support container lifecycle management.

[0030] Specifically, the embodiment of the present application configures the new management node based on the configuration information of the target management node. It is necessary to prepare a new management node, ensure that the hardware configuration of the new node meets the cluster requirements, and install the same version of the operating system as the original management node to build a compatible underlying operating environment; based on the Internet Protocol address of the target management node, configure the network address of the new management node to ensure that it can be accurately identified and achieve normal communication in the target cluster network; generate a key pair (such as an SSH key pair) in the new management node, and send the public key in the key pair to all other nodes (including management nodes and computing nodes) in the target cluster to which the target management node belongs, so as to achieve secret-free login between the new node and other nodes in the cluster, and ensure the security of node interaction through asymmetric encryption mechanism; install the containerized platform component (such as Docker and other software) consistent with the target cluster version for the new management node, and configure the container service corresponding to the containerized platform component to start automatically at boot, so that the new management node has the core function of participating in cluster container management. Through the above configuration, the new management node can achieve seamless docking with the target cluster based on the configuration information of the target management node, and take over the management functions of the original management node.

[0031] Furthermore, in some embodiments, the Internet Protocol address of the new management node is configured based on the Internet Protocol address of the target management node, including: based on the Internet Protocol address, executing a preset communication test instruction through any node of the target cluster, and determining whether the Internet Protocol address of the target management node is occupied; if the Internet Protocol address of the target management node is occupied, releasing the Internet Protocol address, and after the Internet Protocol address of the target management node is released, configuring the Internet Protocol address of the new management node so that the Internet Protocol address of the new management node is the same as the Internet Protocol address of the target management node.

[0032] The preset communication test instruction may be a communication test instruction pre-set by the user, a communication test instruction obtained through a limited number of experiments, or a communication test instruction obtained through a limited number of computer simulations. The occupied state refers to the state in which a specific Internet Protocol address is registered, bound, or actually used for data transmission by a certain network device through a network protocol in a network environment. Releasing an Internet Protocol address refers to the operation of releasing the binding relationship between the Internet Protocol address and the original occupying device by executing a preset network management instruction or triggering a network protocol interaction when it is detected that the Internet Protocol address of the target management node is in an occupied state, thereby converting the address from an occupied state to an unoccupied state.

[0033] Specifically, when configuring the new management node's Internet Protocol address based on the target management node's Internet Protocol address, a preset communication test command, such as "ping 10.0.0.10" (where 10.0.0.10 is the target management node's IP address), is executed on any node in the target cluster, using the target management node's Internet Protocol address as a reference. This command determines whether the target management node's Internet Protocol address is occupied. If the determination result indicates that the target management node's Internet Protocol address is occupied, the cause needs to be investigated and a release operation performed. After the release is complete, the new management node's Internet Protocol address is configured to be consistent with the target management node's Internet Protocol address. If the determination result indicates that the target management node's Internet Protocol address is unoccupied, it indicates that the Internet Protocol address is unoccupied and can be used. This ensures that the new management node reuses the original Internet Protocol address, maintaining cluster configuration consistency. It also avoids Internet Protocol address conflicts and ensures normal access for the new node.

[0034] To facilitate those skilled in the art to further understand the configuration method of the new management node in the embodiment of the present application, the following is a Figure 2 Further explanation is given.

[0035] like Figure 2 As shown, Figure 2 1 is a flow chart of a method for configuring a new management node according to an embodiment of the present application. The method for configuring a new management node includes the following steps: S201, prepare a new management node and install the operating system.

[0036] S202, execute the ping command on the normal node to check whether the IP is occupied, if yes, execute step S204; otherwise, execute step S203.

[0037] S203: If the IP address is available, proceed to step S205.

[0038] S204: After checking the cause of IP occupation and releasing it, execute step S205.

[0039] S205: Configure the new node IP address to be consistent with the original downtime node IP address.

[0040] S206: Distribute the public key to other normal nodes and configure secret-free communication.

[0041] S207, install the Docker version that is consistent with the original cluster version and set it to start automatically at boot.

[0042] S208, basic configuration is completed.

[0043] In step S102, a taint is set for the new management node through a preset container orchestration platform component, the faulty node in the target cluster to which the target management node belongs is removed, and the distributed key-value storage system service of the target cluster is restarted.

[0044] Container orchestration platform components refer to the core functional modules that make up the container orchestration system. These components work together to manage the entire lifecycle of containerized applications within the cluster, including deployment, scheduling, scaling, node management, and state maintenance. A taint is an exclusionary attribute imposed on a node within the container orchestration platform, which restricts container groups from being scheduled to that node. A faulty node is one that is unable to participate in cluster operations or provide services due to hardware, software, or network issues. A distributed key-value storage system service is a service that runs on a cluster consisting of multiple nodes and is specifically designed to store and manage key-value data.

[0045] Specifically, based on the preset container orchestration platform components, a taint configuration operation is performed on the new management node to restrict the scheduling of unnecessary container groups to it, so as to ensure the operational stability of the new management node during the initialization phase; at the same time, for the faulty nodes identified by the health check mechanism in the target cluster, a removal instruction is issued through the container orchestration platform component to remove them from the node list of the target cluster to prevent the faulty nodes from continuously occupying resources or interfering with the normal scheduling process; a restart operation is initiated to the distributed key-value storage system service of the target cluster through the container orchestration platform component, so that this service can reload and update the information after the node changes in the cluster.

[0046] Optionally, in some embodiments, a taint is set for the new management node using a preset container orchestration platform component, including: adding the new management node to the node list of the target cluster; when initializing the preset container orchestration platform component, a taint is set for the new management node using a preset management tool, and the taint is configured to prohibit the creation of multiple copies of containers on the new management node.

[0047] A pre-configured management tool is a pre-configured tool used to configure node attributes during cluster deployment or node management. A replicated container is a set of multiple identical container instances created from the same container image.

[0048] Specifically, edit the configuration file of the container orchestration component (such as Kubernetes) and add the new node information to the cluster's node list. When using the default management tool (such as kubeadm) to initialize the default container orchestration platform component, use the default management tool (such as kubectl) to set a taint for the new management node to prohibit containers from creating multiple copies on the new management node. The specific command is as follows: kubeadm init --config=kubeadm-config.yaml kubectl taint node ${recover_master_name} inais-recovery=:NoSchedule ${recover_master_name} is the name of the new management node. Setting this taint prevents multi-replica microservices from automatically creating pod containers (the smallest deployment and management unit in a Kubernetes cluster) on the new node before it is fully repaired, potentially leading to page unavailability and other issues.

[0049] To facilitate those skilled in the art to further understand the stain configuration method of the embodiment of the present application, the following is combined with Figure 3 Further explanation is given.

[0050] like Figure 3 As shown, Figure 3 1 is a flowchart of a stain configuration method provided according to an embodiment of the present application, the stain configuration method comprising the following steps: S301, edit the k8s configuration file.

[0051] S302: Execute kubeadm init to initialize the new node.

[0052] S303: Add a taint to the new node and prohibit the creation of multiple replica containers.

[0053] S304, k8s expansion completed.

[0054] Optionally, in some embodiments, the faulty node in the target cluster to which the target management node belongs is removed, and the distributed key-value storage system service of the target cluster is restarted, including: determining any healthy node from the target cluster, and using any healthy node to detect the status of the distributed key-value storage system in the target cluster to obtain a detection result; determining the faulty node based on the detection result, and removing the faulty node from the target cluster; copying at least one certificate from the target management node to a new management node, and obtaining the latest distributed key-value storage system data snapshot from the healthy node of the target cluster, and restoring the latest distributed key-value storage system data snapshot on the new management node, and starting the distributed key-value storage system service of the new management node.

[0055] A healthy node is a node in the cluster that is functioning normally, has stable core components, can communicate normally with the cluster control plane, and is capable of executing pre-set tasks. A certificate is a digital credential used for identity authentication and encrypted communication. It is issued by the cluster's built-in certificate authority and contains key information such as the public key, the certificate holder's identity, the issuing authority, and the validity period. A data snapshot of a distributed key-value storage system is a complete and consistent backup of all key-value data and related metadata in the distributed key-value storage system at a specific point in time.

[0056] Specifically, select a healthy node in the target cluster and use it as the carrier to detect the running status of the distributed key-value storage system (such as the etcd cluster) in the target cluster and identify the faulty node. Use management tools (such as etcdctl) to perform the detection. The specific command is as follows: ETCDCTL_API=3 etcdctl --endpoints=https: / / [healthy node IP]:2379 \ --cacert= / etc / kubernetes / pki / etcd / ca.crt \ --cert= / etc / kubernetes / pki / etcd / healthcheck-client.crt \ --key= / etc / kubernetes / pki / etcd / healthcheck-client.key \ endpoint status --write-out=table After determining the faulty node based on the detection results, execute the following command on a normal management node to remove the faulty node from the target cluster: ETCDCTL_API=3 etcdctl --endpoints=https: / / [healthy node IP]:2379 \ --cacert= / etc / kubernetes / pki / etcd / ca.crt \ --cert= / etc / kubernetes / pki / etcd / healthcheck-client.crt \ --key= / etc / kubernetes / pki / etcd / healthcheck-client.key \ member remove [faulty node ID] Copy at least one certificate from the target management node to the new management node to ensure secure access, obtain the latest data snapshot of the distributed key-value storage system from the healthy node of the target cluster, restore the snapshot on the new management node to ensure data consistency, then restore the snapshot on the new node and start the distributed key-value storage system service of the new node to ensure normal access to the new node and improve recovery reliability and cluster stability.

[0057] In step S103 , the priority of the high availability daemon service of the new management node is set to the lowest level in the target cluster, and the high availability daemon service of the new management node is restarted and the stain is removed to complete the recovery action on the target management node.

[0058] The high availability daemon service is a critical process that runs continuously in the background of the management node and ensures high availability of the cluster's core services. Priority is a parameter used to define the priority of a node or service in resource allocation, fault handling, role election, and other processes. Lowest priority refers to the node or service with the lowest priority in the priority parameter sequence.

[0059] Specifically, you need to install and configure a network failover component (such as Keepalived) and ensure that its configuration is consistent with the original cluster. Set the high-availability daemon service priority of the new management node to the lowest level in the target cluster to prevent interference with the cluster's high availability mechanism when the new node is unstable. Restart the high-availability daemon service to enable the priority configuration to take effect. At the same time, remove the taint from the new management node to allow workloads to be scheduled to it, thus completing the recovery of the target management node.

[0060] Taking Keepalived as an example, execute the command: kubectl taint node ${recover_master_name} inais-recovery- After the new node is fully recovered, the taint is removed and the replicated containers are restarted, ensuring that running containerized services such as the development environment, training tasks, and inference services are not affected. The replicated microservices automatically create pod containers on the new node, completing the entire process of replacing the downed node with the new node.

[0061] Optionally, in some embodiments, the priority of the high availability daemon service of the new management node is set to the lowest level in the target cluster, including: determining a priority set of at least some management nodes in the target cluster; determining an initial priority of the high availability daemon service of the new management node based on the priority set; determining a new priority set of at least some management nodes in the target cluster, and based on the new priority set, verifying the initial priority to obtain a verified priority; and setting the verified priority as the lowest level in the target cluster.

[0062] Specifically, taking Keepalived as an example, when the Keepalived service is restarted, the virtual Internet Protocol address will preferentially drift to the node with the largest priority parameter value (a numerical parameter used to define node priority) in the service configuration. Therefore, to prevent the virtual Internet Protocol address from drifting to the new management node and causing service unavailability, the priority parameter priority in the keepalived.conf configuration of the recovery node needs to be set to the minimum value among all management nodes. The implementation method includes: determining the priority set of at least some management nodes in the target cluster, and setting the initial priority of the high-availability daemon service (such as Keepalived) of the new management node based on this; verifying the initial priority based on the new priority set of some management nodes in the cluster, obtaining the priority that passes the verification and setting it as the minimum value among all management nodes, and writing it to the keepalived.conf configuration of the new management node.

[0063] Optionally, in some embodiments, determining the priority set of at least some management nodes in the target cluster includes: collecting configuration files of high-availability daemon services of at least some management nodes in the target cluster; identifying priority parameter values ​​corresponding to the Internet Protocol addresses of the management nodes based on the configuration files; and obtaining the priority set of at least some management nodes in the target cluster based on the priority parameter values ​​corresponding to the Internet Protocol addresses of the management nodes.

[0064] Specifically, an automated script is used to batch collect the configuration files (i.e., keepalived.conf) of the high-availability daemon service (using Keepalived as an example) for at least some (preferably all) management nodes in the target cluster. Based on this configuration file, the priority parameter values ​​associated with the virtual Internet Protocol addresses corresponding to the Internet Protocol addresses of each management node are identified (assuming the default parameter value range is [1, 255]). Based on the priority parameter values ​​corresponding to the Internet Protocol addresses of these management nodes, a priority set P = [p1, p2, ..., pn] (where n is the number of management nodes) for at least some of the management nodes in the target cluster is obtained. Thus, identifying the priority parameter values ​​corresponding to the Internet Protocol addresses of the management nodes through the configuration file greatly improves the efficiency of priority extraction and facilitates the determination of priority sets.

[0065] Further, in some embodiments, based on the priority set, the initial priority of the high availability daemon service of the new management node is determined, including: based on the priority set, determining the baseline minimum value of the priority configuration parameter; and determining the initial priority of the high availability daemon service of the new management node according to the difference between the baseline minimum value and a preset threshold.

[0066] The baseline minimum value refers to the minimum priority parameter value determined from the collected management node priority set, that is, the element with the smallest value in the set. The preset threshold can be a threshold set by the user, a threshold obtained through a limited number of experiments, or a threshold obtained through a limited number of computer simulations.

[0067] Specifically, based on the priority set of the target cluster management node, the minimum value min_p=min(P) is calculated and determined as the baseline minimum value of the priority configuration parameter. To ensure that the difference between the recovery node priority and other nodes is greater than 1 and to provide buffer space, a safety redundancy value δ is set (δ is a positive integer, recommended to be 2, and the threshold is modifiable) to avoid critical priority conflicts (such as other nodes' priorities suddenly dropping to min_p-1). The initial priority of the high-availability daemon service of the new management node is determined based on the difference between the baseline minimum value and the threshold. The calculation formula is: .

[0068] Therefore, by setting a safety redundancy value (i.e., a preset threshold) to avoid critical priority conflicts, it is effectively ensured that it is much smaller than any possible new value of the remaining nodes in the future, effectively avoiding the problem of service unavailability caused by the virtual Internet Protocol address drifting to the new management node.

[0069] Further, in some embodiments, based on the new priority set, the initial priority is verified to obtain the verified priority, including: determining whether the initial priority is less than any priority in the new priority set; if the initial priority is less than any priority in the new priority set, then taking the initial priority as the verified priority.

[0070] Specifically, taking Keepalived as an example, considering that in the actual scenario, the priority value of Keepalived in the cluster may change dynamically, that is, there may be a delay between the collected data set P and the actual configuration. For example, when collecting data, the priority of a certain node is 100, and it has dropped to 98 during verification, resulting in an inaccurate initial priority set. Therefore, the new priority set P is collected again, and a verification module is added to check whether the initial priority priority_temp satisfies being less than the priorities of all nodes, that is, priority_temp < pn holds for any pn ∈ P. If it is satisfied, then the initial priority is taken as the verified priority. Thus, by determining that the initial priority is less than any priority in the new priority set, the problem of the virtual Internet protocol address drifting to a new management node and causing service unavailability is effectively avoided.

[0071] Further, in some embodiments, when determining whether the initial priority is less than any priority in the new priority set, it includes: if the initial priority is not less than any priority in the new priority set, then based on a preset correction strategy, the initial priority is iteratively corrected until the corrected priority is less than any priority in the new priority set; taking the corrected priority as the verified priority.

[0072] Among them, the preset correction strategy refers to a set of clear and executable rules or methods preset when the initial priority of the new management node does not meet the condition of being less than all priorities in the new priority set. Iterative correction refers to a cyclic process of repeatedly adjusting and verifying the priority according to the preset correction strategy.

[0073] Specifically, taking Keepalived as an example, if the initial priority is not less than any priority in the new priority set, that is, there exists pn ≤ priority_temp, then based on the preset correction strategy (such as priority_temp = priority_temp - 1), iterative correction is performed, and repeated verification is carried out until the condition is satisfied. Taking the corrected priority as the verified priority. Updating the finally verified priority value to the service configuration of the recovery node and restarting the high-availability daemon service. Thus, through iterative correction, the minimum priority value is ensured, and the problem of the virtual Internet protocol address drifting to a new management node and causing service unavailability is effectively avoided.

[0074] To facilitate those skilled in the art to further understand the priority configuration method of the embodiment of the present application, the following Figure 4 Further explanation is given.

[0075] like Figure 4 As shown, Figure 4 1 is a flowchart of a priority configuration method provided according to an embodiment of the present application, the priority configuration method comprising the following steps: S401: Collect cluster priority data.

[0076] S402, setting a reference minimum value min_p and a safety margin value.

[0077] S403: Set the initial priority of the new management node.

[0078] S404: Collect priority data again.

[0079] S405, check whether pn≤priority_temp exists, if yes, execute step S406; otherwise, execute step S407.

[0080] S406, iteratively correct the global minimum value, and then return to step S403.

[0081] S407: Update the Keepalived priority configuration and start the service.

[0082] This approach involves configuring a new node based on the target management node, assigning a taint to the new node through the container orchestration platform, removing the failed node from the cluster, restarting the distributed key-value storage service, setting the high-availability daemon process priority of the new node to the lowest in the cluster, restarting the service, and removing the taint to complete the recovery. This solves the existing issues of complex and time-consuming recovery processes and the impact on the availability of running services and platform pages, achieving rapid and stable recovery of management nodes.

[0083] Furthermore, in order to ensure that the Internet Protocol address of the new node is consistent with the Internet Protocol address of the original downtime node, a check is added during startup to determine whether the Internet Protocol address of the new node is the original address.

[0084] Optionally, in some embodiments, when starting the distributed key-value storage system service of a new management node, it also includes: determining whether the Internet Protocol address of the new management node is consistent with the Internet Protocol address of the target management node; if the Internet Protocol address of the new management node is consistent with the Internet Protocol address of the target management node, restarting the distributed key-value storage system service of the new management node.

[0085] Optionally, in some embodiments, after determining whether the Internet Protocol address of the new management node is consistent with the Internet Protocol address of the target management node, it also includes: if the Internet Protocol address of the new management node is inconsistent with the Internet Protocol address of the target management node, restarting the distributed key-value storage system service of at least some nodes in the target cluster.

[0086] Specifically, if the Internet Protocol address of the new node is consistent with the original Internet Protocol address, only the distributed key-value storage system service of the new management node is restarted; if the Internet Protocol address of the new node is inconsistent with the original Internet Protocol address, it is necessary to restart the distributed key-value storage system service on all nodes of the cluster. The purpose is to refresh the environment variable information required by the distributed key-value storage system service of the cluster and ensure that the new Internet Protocol address information is updated in the distributed key-value storage system database.

[0087] To facilitate those skilled in the art to further understand the distributed key-value storage system service recovery method of the embodiment of the present application, the following is combined with Figure 5 Further explanation is given.

[0088] like Figure 5 As shown, Figure 5 This is a flow chart of a distributed key-value storage system service recovery method provided according to an embodiment of the present application. The distributed key-value storage system service recovery method includes the following steps: S501, check the etcd cluster status and obtain the faulty node ID.

[0089] S502: Remove the faulty node from the etcd cluster.

[0090] S503: Install the etcd service on the new node and copy the certificate.

[0091] S504: Create a snapshot from the healthy node and copy it to the new node.

[0092] S505, determine whether the new node IP is consistent with the original node, if so, execute step S506; otherwise, execute step S507.

[0093] S506: Restart the etcd service of the recovery node (i.e., the new management node).

[0094] S507, restart the etcd service on all nodes.

[0095] S508, etcd service recovery is complete.

[0096] In this way, it is possible to avoid restarting the distributed key-value storage system service of the normal management node during the management node replacement process, thereby ensuring the stability of the distributed key-value storage system cluster.

[0097] Furthermore, in order to facilitate those skilled in the art to further understand the crash recovery method of the management node in the embodiment of the present application, the artificial intelligence training platform involved in the crash recovery method of the management node in the embodiment of the present application is described in detail below.

[0098] like Figure 6 As shown, Figure 6 This is a schematic diagram of an artificial intelligence training platform provided according to one embodiment of the present application. In a high-availability cluster of an artificial intelligence development platform, the components of the management node and compute node are basically as follows. Management nodes typically deploy components such as the API Server, Scheduler, and Controller Manager, the etcd database, the Docker container engine, Keepalived (for virtual IP address migration and high availability), the MariaDB database (for storing platform business data), and Harbor (a container image repository). Compute nodes deploy node services (for communication with the management node), a development environment (for developers to write and debug code), training tasks (for running the artificial intelligence model training process), an inference service (for providing model inference capabilities), and Docker components (for running containers).

[0099] The embodiment of the present application proposes that in the process of replacing the target management node, a series of orderly steps and targeted measures are taken to ensure the stability of the cluster and the availability of services during the recovery process. Specifically, it includes: environmental preparation and basic configuration: prepare a new node with the same information as the original node, then configure the secret-free communication between clusters and install basic software such as docker; restore components such as the k8s control plane: when the recovery node joins the k8s cluster, set a stain to prevent the creation of multiple copies of the cluster business module container on the recovery node, and avoid traffic entering the management node that has not been fully restored and causing service unavailability; restore etcd service: through IP verification, avoid the restart of the etcd service on the normal node during the recovery process of etcd service, causing service interruption; restore Keepalived service: through the Keepalived priority configuration process solution, ensure that the priority of the Keepalived service configuration on the recovery node is the lowest, and avoid the drift of the virtual IP to the restored node when it drifts and causes the page to be unavailable; finally, remove the stain of the recovery node to restart the multiple copies of the business module microservice, and finally complete the recovery of all services on the management node.

[0100] Furthermore, if Figure 7 As shown, Figure 7 This is a flowchart of a method for recovering a management node from a downtime according to an embodiment of the present application. The method for recovering a management node from a downtime includes the following steps: S701, environment preparation and basic configuration.

[0101] S702, restore components such as the k8s control plane.

[0102] S703, restore etcd service.

[0103] S704: Restore the Keepalived service.

[0104] S705: Remove the stain and start all microservices of the recovery node.

[0105] This ensures that the platform can be used normally during the management node recovery process, ensuring that running containerized services such as the development environment, training tasks, and inference services are not affected, thereby improving the stability of the cluster.

[0106] According to the management node downtime recovery method proposed in the embodiments of the present application, when the target management node is down, a new node is configured based on the target management node, a taint is set for the new node via the container orchestration platform, the failed node in the cluster is removed, and the distributed key-value storage service is restarted. The high-availability daemon process priority of the new node is set to the lowest in the cluster, the service is restarted, the taint is removed, and recovery is completed. This solves the problems of the prior art, such as the complex and time-consuming recovery process and the impact on the availability of running services and platform pages, and achieves rapid and stable recovery of management nodes.

[0107] Next, a crash recovery device for a management node proposed according to an embodiment of the present application will be described with reference to the accompanying drawings.

[0108] Figure 8 It is a block diagram of an edge multi-source flow aggregation and transmission device according to an embodiment of the present application.

[0109] like Figure 8 As shown, the edge multi-source flow aggregation transmission device 10 includes: a matching module 100, a processing module 200 and a determination module 300.

[0110] Wherein, the matching module 100 configures a new management node according to the configuration information of the target management node when the target management node is in a down state.

[0111] Processing module 200 sets a taint for the new management node through a preset container orchestration platform component, removes the faulty node in the target cluster to which the target management node belongs, and restarts the distributed key-value storage system service of the target cluster; The determination module 300 sets the priority of the high availability daemon service of the new management node to the lowest level in the target cluster, restarts the high availability daemon service of the new management node, and removes the stain to complete the recovery action on the target management node.

[0112] According to one embodiment of the present application, the matching module 100 is specifically used to: identify the operating system of the target management node and install the operating system for the new management node; configure the Internet Protocol address of the new management node based on the Internet Protocol address of the target management node; generate a key pair on the new management node, and send the public key in the key pair to the target cluster to which the target management node belongs, and configure the new management node to communicate with other nodes in the target cluster without secrets; install the same containerized platform component as the target cluster version for the new management node, and configure the container service corresponding to the containerized platform component to start up the new management node.

[0113] According to one embodiment of the present application, the matching module 100 is specifically used to: execute a preset communication test instruction through any node of the target cluster based on the Internet Protocol address, and determine whether the Internet Protocol address of the target management node is occupied; if the Internet Protocol address of the target management node is occupied, release the Internet Protocol address, and after the Internet Protocol address of the target management node is released, configure the Internet Protocol address of the new management node so that the Internet Protocol address of the new management node is the same as the Internet Protocol address of the target management node.

[0114] According to one embodiment of the present application, the processing module 200 is specifically used to: add the new management node to the node list of the target cluster; when initializing the preset container orchestration platform component, use the preset management tool to set a taint for the new management node, and the taint is configured to prohibit the creation of multiple copies of containers on the new management node.

[0115] According to one embodiment of the present application, the processing module 200 is specifically used to: determine any healthy node from the target cluster, and use any healthy node to detect the status of the distributed key-value storage system in the target cluster to obtain a detection result; determine the faulty node based on the detection result, and remove the faulty node from the target cluster; copy at least one certificate from the target management node to the new management node, and obtain the latest distributed key-value storage system data snapshot from the healthy node of the target cluster, and restore the latest distributed key-value storage system data snapshot on the new management node, and start the distributed key-value storage system service of the new management node.

[0116] According to one embodiment of the present application, the processing module 200 is also used to: determine whether the Internet Protocol address of the new management node is consistent with the Internet Protocol address of the target management node; if the Internet Protocol address of the new management node is consistent with the Internet Protocol address of the target management node, restart the distributed key-value storage system service of the new management node.

[0117] According to one embodiment of the present application, the processing module 200 is further used to: restart the distributed key-value storage system service of at least some nodes in the target cluster if the Internet Protocol address of the new management node is inconsistent with the Internet Protocol address of the target management node.

[0118] According to one embodiment of the present application, the determination module 300 is specifically used to: determine the priority set of at least some management nodes in the target cluster; determine the initial priority of the high-availability daemon service of the new management node based on the priority set; determine a new priority set of at least some management nodes in the target cluster, and based on the new priority set, verify the initial priority to obtain a priority that passes the verification; and use the priority that passes the verification as the lowest level in the target cluster.

[0119] According to one embodiment of the present application, the determination module 300 is specifically used to: collect configuration files of high-availability daemon services of at least some management nodes in the target cluster; based on the configuration files, identify the priority parameter values ​​corresponding to the Internet Protocol addresses of the management nodes; and obtain the priority set of at least some management nodes in the target cluster according to the priority parameter values ​​corresponding to the Internet Protocol addresses of the management nodes.

[0120] According to one embodiment of the present application, the determination module 300 is specifically used to: determine a baseline minimum value of the priority configuration parameter based on a priority set; and determine the initial priority of the high availability daemon service of the new management node based on the difference between the baseline minimum value and a preset threshold.

[0121] According to one embodiment of the present application, the determination module 300 is specifically used to: determine whether the initial priority is lower than any priority in the new priority set; if the initial priority is lower than any priority in the new priority set, the initial priority is used as the priority that passes the verification.

[0122] According to one embodiment of the present application, the determination module 300 is specifically used to: if the initial priority is not less than any priority in the new priority set, then based on a preset correction strategy, iteratively correct the initial priority until the corrected priority is less than any priority in the new priority set; and use the corrected priority as the priority that has passed the verification.

[0123] According to the management node downtime recovery device proposed in the embodiments of the present application, when a target management node is down, a new node is configured based on the target management node, a taint is set for the new node via the container orchestration platform, the failed node in the cluster is removed, and the distributed key-value storage service is restarted. The high-availability daemon process priority of the new node is set to the lowest in the cluster, the service is restarted, the taint is removed, and recovery is completed. This solves the problems of the prior art in that the recovery process is complex and time-consuming, and affects the availability of running services and platform pages, and achieves rapid and stable recovery of management nodes.

[0124] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include: A memory 901 , a processor 902 , and a computer program stored in the memory 901 and executable on the processor 902 .

[0125] When the processor 902 executes the program, the method for recovering from a downtime of a management node provided in the above embodiment is implemented.

[0126] Furthermore, the electronic device further includes: The communication interface 903 is used for communication between the memory 901 and the processor 902 .

[0127] The memory 901 is used to store computer programs that can be run on the processor 902 .

[0128] The memory 901 may include a high-speed RAM (Random Access Memory) memory, and may also include a non-volatile memory, such as at least one disk memory.

[0129] If the memory 901, processor 902, and communication interface 903 are implemented independently, the communication interface 903, memory 901, and processor 902 can be connected to each other via a bus and communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 9 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0130] Optionally, in a specific implementation, if the memory 901, the processor 902 and the communication interface 903 are integrated on a chip, the memory 901, the processor 902 and the communication interface 903 can communicate with each other through an internal interface.

[0131] The processor 902 may be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application.

[0132] An embodiment of the present application also provides a non-volatile computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned method for recovering from a downtime of a management node.

[0133] An embodiment of the present application also provides a computer program product, including a computer program, which implements the above-mentioned method for recovering from a management node failure when the computer program is executed by a processor.

[0134] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0135] The above is a detailed introduction to the downtime recovery method for a management node provided by this application. This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only intended to help understand the method and core ideas of this application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of this application, several improvements and modifications can be made to this application, and these improvements and modifications also fall within the scope of protection of the claims of this application.

Claims

1. A method for recovering a management node from downtime, characterized in that: The following steps are involved: When the target management node is in a down state, configuring a new management node according to the configuration information of the target management node; Setting a taint for the new management node through a preset container orchestration platform component, removing the faulty node in the target cluster to which the target management node belongs, and restarting the distributed key-value storage system service of the target cluster; The priority of the high availability daemon service of the new management node is set to the lowest level in the target cluster, and the high availability daemon service of the new management node is restarted, and the stain is removed to complete the recovery action on the target management node.

2. The method for recovering a management node from downtime according to claim 1, wherein: Configuring a new management node according to the configuration information of the target management node includes: Identifying the operating system of the target management node and installing the operating system for the new management node; Configuring an Internet Protocol address of the new management node based on the Internet Protocol address of the target management node; Generate a key pair on the new management node, send the public key in the key pair to the target cluster to which the target management node belongs, and configure secret-free communication between the new management node and other nodes in the target cluster; Install a containerized platform component with the same version as the target cluster for the new management node, and configure the new management node to automatically start the container service corresponding to the containerized platform component when it is powered on.

3. The method for recovering a management node from downtime according to claim 2, wherein: The configuring the Internet Protocol address of the new management node based on the Internet Protocol address of the target management node includes: Based on the Internet Protocol address, executing a preset communication test instruction through any node of the target cluster, and determining whether the Internet Protocol address of the target management node is in an occupied state; If the Internet Protocol address of the target management node is in the occupied state, the Internet Protocol address is released, and after the Internet Protocol address of the target management node is released, the Internet Protocol address of the new management node is configured so that the Internet Protocol address of the new management node is the same as the Internet Protocol address of the target management node.

4. The method for recovering a management node from downtime according to claim 1, wherein: The setting of a taint for the new management node by using a preset container orchestration platform component includes: Adding the new management node to the node list of the target cluster; When initializing the preset container orchestration platform component, a preset management tool is used to set a taint for the new management node, where the taint is configured to prohibit creation of multiple copies of containers on the new management node.

5. The method for recovering a management node from downtime according to claim 1, wherein: The removing of the faulty node in the target cluster to which the target management node belongs and restarting the distributed key-value storage system service of the target cluster includes: Determine any healthy node from the target cluster, and use the any healthy node to detect the state of the distributed key-value storage system in the target cluster to obtain a detection result; Determine a faulty node according to the detection result, and remove the faulty node from the target cluster; Copy at least one certificate from the target management node to the new management node, obtain the latest distributed key-value storage system data snapshot from the healthy node of the target cluster, restore the latest distributed key-value storage system data snapshot on the new management node, and start the distributed key-value storage system service of the new management node.

6. The method for recovering a management node from downtime according to claim 5, wherein: When starting the distributed key-value storage system service of the new management node, the method further includes: Determining whether the Internet Protocol address of the new management node is consistent with the Internet Protocol address of the target management node; If the Internet Protocol address of the new management node is consistent with the Internet Protocol address of the target management node, restart the distributed key-value storage system service of the new management node.

7. The method for recovering a management node from downtime according to claim 6, wherein: After determining whether the Internet Protocol address of the new management node is consistent with the Internet Protocol address of the target management node, the method further includes: If the Internet Protocol address of the new management node is inconsistent with the Internet Protocol address of the target management node, the distributed key-value storage system service of at least some nodes in the target cluster is restarted.

8. The method for recovering a management node from downtime according to claim 1, wherein: Setting the priority of the high availability daemon service of the new management node to the lowest level in the target cluster includes: Determining a priority set of at least some management nodes in the target cluster; Determining an initial priority of the high availability daemon service of the new management node based on the priority set; Determine a new priority set for at least some management nodes in the target cluster, and verify the initial priorities based on the new priority set to obtain verified priorities; The priority of the verified one is set as the lowest level in the target cluster.

9. The method for recovering a management node from downtime according to claim 8, wherein: Determining a priority set of at least some management nodes in the target cluster includes: Collecting configuration files of high availability daemon services of at least some management nodes in the target cluster; Based on the configuration file, identifying a priority parameter value corresponding to an Internet Protocol address of a management node; A priority set of at least some management nodes in the target cluster is obtained according to the priority parameter value corresponding to the Internet Protocol address of the management node.

10. The method for recovering a management node from downtime according to claim 8, wherein: Determining the initial priority of the high availability daemon service of the new management node based on the priority set includes: Determining a baseline minimum value of a priority configuration parameter based on the priority set; An initial priority of the high-availability daemon service of the new management node is determined according to a difference between the benchmark minimum value and a preset threshold.

11. The method for recovering a management node from downtime according to claim 8, wherein: The verifying the initial priority based on the new priority set to obtain a verified priority includes: Determining whether the initial priority is less than any priority in the new priority set; If the initial priority is lower than any priority in the new priority set, the initial priority is used as the priority that passes the verification.

12. The method for recovering a management node from downtime according to claim 11, wherein: Determining whether the initial priority is less than any priority in the new priority set includes: If the initial priority is not less than any priority in the new priority set, iteratively correcting the initial priority based on a preset correction strategy until the corrected priority is less than any priority in the new priority set; The corrected priority is used as the priority that passes the verification.

13. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for recovering a management node from downtime according to any one of claims 1 to 12.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the method for recovering from a downtime of a management node according to any one of claims 1 to 12.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for recovering a management node from downtime according to any one of claims 1 to 12 is implemented.

Images