Method for recovery from downtime of management node, electronic device and storage medium

Abstract

The application relates to the field of artificial intelligence, in particular to a method for recovering a down management node, an electronic device and a storage medium. The method comprises the following steps: in the case that a target management node is in a down state, configuring a new management node according to configuration information of the target management node; setting a stain for the new management node through a preset container orchestration platform component, removing a fault node in a target cluster to which the target management node belongs, and restarting a distributed key-value storage system service of the target cluster; setting a priority of a high-availability daemon service of the new management node to the lowest level in the target cluster, restarting the high-availability daemon service of the new management node, and removing the stain, so as to complete a recovery action of the target management node. Thus, the problems that the recovery process is complex and time-consuming and the availability of a running service and a platform page is affected in the prior art are solved, and quick and stable recovery of the management node is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, in particular to a method for recovering from a management node failure, an electronic device and a storage medium. BACKGROUND

[0002] In the field of artificial intelligence, large-scale artificial intelligence development platforms are usually built on Kubernetes (k8s) high-availability clusters to achieve efficient management and scheduling of large-scale computing resources. As the core of the k8s cluster, the management node is deployed with key components such as the control plane components (such as the API Server (Application Programming Interface Server)), the Etcd database, the Docker engine, the high-availability components (such as Keepalived), and bears the core functions of resource scheduling, task orchestration, and state management. The stable operation of the management node is a prerequisite for the normal operation of the platform, and its failure may cause the control plane of the cluster to fail, thereby affecting the entire platform operation. Therefore, the high availability and rapid recovery capability of the management node are crucial to the artificial intelligence platform cluster.

[0003] In related technologies, the commonly used methods for recovering from a management node failure on the Internet mainly include the following two methods: periodic backup of the management node; and self-healing capability based on container orchestration platform components.

[0004] However, the backup recovery effect in related technologies depends on the completeness and timeliness of the backup; and the self-healing capability based on container orchestration platform components has high requirements for the initial configuration of the cluster, and requires automatic deployment and migration mechanisms to be configured in advance. This results in problems such as loss of backup data, long time consumption, and impact on the availability of running services and platform pages, which need to be solved urgently. SUMMARY

[0005] The present application provides a method for recovering from a management node failure, an electronic device and a storage medium to solve the problems of complex and time-consuming recovery process and impact on the availability of running services and platform pages in the prior art, and to achieve rapid and stable recovery of the management node.

[0006] To achieve the above-mentioned purpose, the first aspect of the present application provides a method for recovering from a management node failure, the method comprising the following steps:

[0007] In the case that the target management node is in a failure state, configuring a new management node according to the configuration information of the target management node;

[0008] setting a taint for the new management node through a preset container orchestration platform component, removing a faulty node in a target cluster to which the target management node belongs, and restarting a distributed key-value storage system service of the target cluster;

[0009] setting a priority of a high-availability daemon service of the new management node to the lowest level in the target cluster, restarting the high-availability daemon service of the new management node, and removing the taint to complete the recovery action of the target management node.

[0010] According to the management node recovery method provided in the embodiments of the present application, in the case that the target management node is in a down state, a new node is configured according to the target management node, a taint is set for the new node through a container orchestration platform, a faulty node in a cluster is removed and a distributed key-value storage service is restarted, a priority of a high-availability daemon service of the new node is set to the lowest level in the cluster, the service is restarted and the taint is removed, and the recovery is completed. Thus, the problems of complex recovery process, time-consuming and influence on the availability of a running service and platform page in the prior art are solved, and the fast and stable recovery of the management node is achieved.

[0011] To achieve the above object, the second aspect of the present application provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the management node recovery method as described in the above embodiments.

[0012] To achieve the above object, the third aspect of the present application provides a computer readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the management node recovery method as described in the above embodiments.

[0013] To achieve the above object, the fourth aspect of the present application provides a computer program product, wherein the computer program is executed by a processor to implement the management node recovery method as described in the above embodiments.

[0014] Additional aspects and advantages of the present application will be made apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort.

[0016] Figure 1A flowchart of a method for recovery from a management node outage according to an embodiment of the present application;

[0017] Figure 2 A flowchart of a method for configuration of a new management node according to an embodiment of the present application;

[0018] Figure 3 A flowchart of a method for configuration of a new management node according to an embodiment of the present application;

[0019] Figure 4 A flowchart of a method for configuration of a new management node according to an embodiment of the present application;

[0020] Figure 5 A flowchart of a method for recovery from a management node outage according to an embodiment of the present application;

[0021] Figure 6 A schematic diagram of an artificial intelligence training platform according to an embodiment of the present application;

[0022] Figure 7 A flowchart of a method for recovery from a management node outage according to an embodiment of the present application;

[0023] Figure 8 A block schematic diagram of a device for recovery from a management node outage according to an embodiment of the present application;

[0024] Figure 9 A schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, any other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0026] It should be noted that, in the description of the present application, the terms “comprise”, “contain” or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. The terms “first”, “second” and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0027] For those skilled in the art of the technical field, the present application will be further described in detail below with the help of the drawings and specific embodiments.

[0028] The skilled in the art can understand that, for the artificial intelligence platform cluster, the normal operation of the management node is the premise to ensure the stable work of the platform. Once the management node is down, it may cause the control plane of the cluster to fail, and further affect the normal operation of the whole platform. For example, the developer cannot submit a new training task, the running training task may be interrupted, the inference service cannot respond to external requests, the platform page cannot be normally accessed, etc. Therefore, the high availability and rapid recovery capability of the management node are crucial for the artificial intelligence platform cluster.

[0029] In the process of recovering the abnormal management node of the k8s cluster, there are two key challenges:

[0030] On the one hand, how to ensure that the containerized services such as development environment, training task, inference service, etc. running in the cluster are not affected during the recovery process. During the recovery process, improper operation may cause container restart, service interruption, and thus affect the ongoing artificial intelligence training and inference work, causing unnecessary loss.

[0031] On the other hand, how to ensure that the platform page can be normally used, and all other normal node services do not restart. The platform page is the main interface for users to interact with the platform, and its availability directly affects the user experience. At the same time, if the services on other normal nodes restart unnecessarily, it may cause the service state to be unstable, affecting the running efficiency of the whole cluster.

[0032] In the prior art, the commonly used management node recovery methods on the Internet mainly include the following two kinds: backup recovery and k8s-based self-healing capability.

[0033] Backup recovery refers to periodically backing up the management node, and when the management node is down, using the backup data to recover the management node. The backup content includes operating system image, configuration file, database data, etc. The defect is that the effect of backup recovery depends on the completeness and timeliness of the backup. If the backup is not timely, some important data may be lost. Moreover, the recovery process needs to import the backup data into the new node, which takes a long time, especially when the data volume is large. In addition, during the recovery process, there may be configuration conflicts and other problems, causing the service to fail to start normally.

[0034] Self-healing capability based on k8s: Some k8s clusters use their self-healing capability to automatically redeploy control plane components on other nodes when a management node is detected to be down. The defect is that this method requires high initial configuration requirements for the cluster, and needs to configure the automatic deployment and migration mechanism in advance. Moreover, during the automatic recovery process, problems such as improper deployment order of components, resource competition, etc. may occur, resulting in recovery failure or unstable services. In addition, for some components that rely on local storage (such as Etcd), automatic migration is difficult, and data inconsistency is likely to occur.

[0035] As can be seen, the current management node down recovery method has significant technical bottlenecks, either the recovery process is complex and time-consuming, or it is easy to affect the availability of running services and platform pages.

[0036] Below, with reference to the accompanying drawings, a management node down recovery method, an electronic device and a storage medium according to an embodiment of the present application are described.

[0037] Figure 1 is a flowchart of a management node down recovery method according to an embodiment of the present application.

[0038] For example, as shown in Figure 1 , the management node down recovery method includes the following steps:

[0039] In step S101, in the case that the target management node is in a down state, a new management node is configured according to the configuration information of the target management node.

[0040] Wherein, the target management node in a down state means that the node originally responsible for the core management function of the system, due to hardware failure, software exception, network interruption, etc., cannot continue to perform the preset management responsibilities and enters the failure state. Configuration information refers to the structured parameter and setting set used to define the identity attribute, management responsibility, running rule and coordination logic of the target management node with other nodes in the system when it is running normally.

[0041] Optionally, in some embodiments, configuring the new management node according to the configuration information of the target management node includes: identifying the operating system of the target management node, and installing the operating system for the new management node; configuring the Internet Protocol address of the new management node based on the Internet Protocol address of the target management node; generating a key pair on the new management node, and sending the public key in the key pair to the target cluster to which the target management node belongs, and configuring the new management node and other nodes in the target cluster to communicate without encryption; installing the containerized platform components of the same version as the target cluster for the new management node, and configuring the new management node to start the container service corresponding to the containerized platform components automatically.

[0042] The operating system refers to a core system software for managing computer hardware and software resources, and is a basic platform for computer system operation. The Internet Protocol Address (IP address) refers to a digital label allocated to a device in a network for identifying the position of the device in the network, and is a core identification for data communication between devices. The key pair refers to a pair of key combinations generated by a cryptography algorithm, associated with each other and complementary in function, including a public key and a private key. The public key is a part of the key pair with public property, and the private key is a part of the key pair with private property. The target cluster refers to a collection of multiple nodes originally managed and coordinated by a target management node. The container platform component refers to a series of core software modules and tool sets constituting a container platform, which collectively support the whole life cycle operations such as creation, deployment, running, management and scheduling of containers. The container service refers to a background service process or program generated by the container platform component runtime for supporting the container life cycle management.

[0043] Specifically, the embodiment of the present application configures the new management node based on the configuration information of the target management node. A new management node needs to be prepared, and it is ensured that the hardware configuration of the new node meets the cluster requirements, and the same version of the operating system as the original management node is installed to build a compatible underlying running environment. The network address of the new management node is configured based on the Internet Protocol Address of the target management node, so as to ensure that it can be accurately identified and normally communicated in the target cluster network. A key pair (such as an ssh key pair) is generated in the new management node, and the public key in the key pair is sent to all other nodes (including management nodes and computing nodes) in the target cluster to which the target management node belongs, so as to realize the password-free login between the new node and other nodes in the cluster, and to ensure the security of node interaction through the asymmetric encryption mechanism. The container platform component (such as Docker software) consistent with the version of the target cluster is installed for the new management node, and the container service corresponding to the container platform component is configured to start automatically at boot, so that the new management node has the core function of participating in the cluster container management. Through the above configuration, the new management node can realize seamless docking with the target cluster based on the configuration information of the target management node, and take over the management functions of the original management node.

[0044] Furthermore, in some embodiments, configuring the Internet Protocol address of a new management node based on the Internet Protocol address of the target management node includes: executing a preset communication test command through any node of the target cluster based on the Internet Protocol address, and determining whether the Internet Protocol address of the target management node is in a occupied state; if the Internet Protocol address of the target management node is in a occupied state, releasing the Internet Protocol address, and after the Internet Protocol address of the target management node is released, configuring the Internet Protocol address of the new management node so that the Internet Protocol address of the new management node is the same as the Internet Protocol address of the target management node.

[0045] The preset communication test commands can be user-defined, obtained through a limited number of experiments, or derived from a limited number of computer simulations. Occupancy status refers to the state in a network environment where a specific Internet Protocol (IP) address is registered, bound, or actually used for data transmission by a network device through network protocols. Releasing an IP address refers to the operation of releasing the binding relationship between the IP address and the original occupying device when the target management node's IP address is detected to be in an occupied state, by executing preset network management commands or triggering network protocol interactions, thus changing the address from an occupied state to an unoccupied state.

[0046] Specifically, when configuring the Internet Protocol (IP) address of a new management node based on the IP address of the target management node, a preset communication test command, such as "ping 10.0.0.10" (where 10.0.0.10 is the IP address of the target management node), is executed on any node in the target cluster, using the target management node's IP address as a reference. This determines whether the target management node's IP address is in use: if the result indicates that the target management node's IP address is in use, the cause needs to be investigated and a release operation performed. After release, the new management node's IP address is configured to match the target management node's IP address. If the result indicates that the IP address is not in use, it means that the IP address is not occupied and can be used. This ensures that the new management node reuses the original IP address, maintaining cluster configuration consistency; simultaneously, it avoids IP address conflicts and ensures normal access for the new node.

[0047] To facilitate those skilled in the art to further understand the configuration method of the new management node in the embodiments of this application, the following is combined with... Figure 2 Further explanation is needed.

[0048] like Figure 2 As shown, Figure 2 This is a flowchart of a configuration method for a new management node according to an embodiment of this application. The configuration method for the new management node includes the following steps:

[0049] S201, prepare a new management node, install an operating system.

[0050] S202, execute a ping command on a normal node to check whether an IP is occupied, if yes, execute step S204; otherwise, execute step S203.

[0051] S203, the IP address is available, execute step S205.

[0052] S204, after investigating the reason for IP occupation and releasing, execute step S205.

[0053] S205, configure the IP address of the new node to be consistent with the IP address of the original down node.

[0054] S206, distribute the public key to other normal nodes, and configure password-free communication.

[0055] S207, install docker consistent with the original cluster version, and set up self-starting at boot.

[0056] S208, the basic configuration is completed.

[0057] In step S102, the preset container orchestration platform components are used to set a stain for the new management node, and remove the fault node in the target cluster to which the target management node belongs, and restart the distributed key-value storage system service of the target cluster.

[0058] The container orchestration platform components refer to the core functional modules that constitute the container orchestration system. These components work together to be responsible for the whole life cycle management of containerized applications in the cluster, such as deployment, scheduling, expansion, node management and state maintenance. The stain refers to a repulsive attribute applied to the node in the container orchestration platform, which is used to limit the container group from being scheduled to the node. The fault node refers to a node that cannot normally participate in cluster work or provide services due to hardware, software, network and other problems. The distributed key-value storage system service refers to a service that runs on a cluster composed of multiple nodes and is specifically used to store and manage key-value pair data.

[0059] Specifically, based on the preset container orchestration platform components, the stain configuration operation is performed on the new management node to limit unnecessary container groups from being scheduled to it, so as to ensure the running stability of the new management node in the initialization stage; at the same time, for the fault node identified by the health check mechanism in the target cluster, the removal instruction is issued through the container orchestration platform components to remove it from the node list of the target cluster, so as to avoid the continuous occupation of resources or interference with the normal scheduling process by the fault node; the container orchestration platform components are used to initiate a restart operation to the distributed key-value storage system service of the target cluster, so that the service reloads and updates the information of the nodes in the cluster after the change.

[0060] Optionally, in some embodiments, the preset container orchestration platform component is used to set the taint for the new management node, including: adding the new management node to the node list of the target cluster; and using the preset management tool to set the taint for the new management node when initializing the preset container orchestration platform component, the taint being configured to prohibit the creation of multiple copies of containers on the new management node.

[0061] The preset management tool refers to a tool configured in advance for performing node attribute configuration in a cluster deployment or node management process. The multiple copies of containers refer to multiple identical container instances created based on the same container image.

[0062] Specifically, a configuration file of a container orchestration component (such as Kubernetes) is edited, and the information of the new node is added to the node list of the cluster. When initializing the preset container orchestration platform component by using a preset management tool (such as kubeadm), the preset management tool (such as kubectl) is used to set the taint for the new management node, and the creation of multiple copies of containers on the new management node is prohibited. The specific command is as follows:

[0063] kubeadm init --config=kubeadm-config.yaml

[0064] kubectl taint node ${recover_master_name} inais-recovery=:NoSchedule

[0065] The ${recover_master_name} is the name of the new management node. By setting the taint, the automatic creation of pod containers (the smallest deployment and management unit in a Kubernetes cluster) on the new node by multiple copies of microservices before the new node is completely repaired is avoided, and problems such as page unavailability are avoided.

[0066] To make the taint configuration method of the embodiments of the present application more understandable to those skilled in the art, the taint configuration method of the embodiments of the present application is further described below with reference to the accompanying drawings. Figure 3

[0067] As shown in Figure 3 , a flowchart of a taint configuration method according to an embodiment of the present application is shown, and the taint configuration method includes the following steps: Figure 3 S301, edit the k8s configuration file.

[0068] S302, execute kubeadm init to initialize the new node.

[0069]

[0070] ​​S303 adds a taint to a new node, preventing the creation of multi-replica containers.

[0071] S304, Kubernetes expansion complete.

[0072] Optionally, in some embodiments, removing a faulty node in the target cluster to which the target management node belongs and restarting the distributed key-value storage system service of the target cluster includes: determining any healthy node from the target cluster, and using any healthy node to detect the status of the distributed key-value storage system in the target cluster to obtain a detection result; determining a faulty node based on the detection result and removing the faulty node from the target cluster; copying at least one certificate from the target management node to the new management node, obtaining the latest distributed key-value storage system data snapshot from the healthy node of the target cluster, restoring the latest distributed key-value storage system data snapshot on the new management node, and starting the distributed key-value storage system service of the new management node.

[0073] In this context, a healthy node refers to a node in the cluster that is in a normal state, whose core components are running stably, can communicate normally with the cluster control plane, and has the ability to execute preset tasks. A certificate is a digital credential file used for authentication and encrypted communication, issued by a certificate authority built into the cluster, and contains key information such as the public key, certificate holder's identity information, issuing authority information, and validity period. A distributed key-value storage system data snapshot refers to a complete and consistent backup of all key-value pairs and related metadata in the distributed key-value storage system at a specific point in time.

[0074] Specifically, a healthy node in normal condition is selected from the target cluster. Then, using this healthy node as the operating platform, the running status of the distributed key-value storage system (such as an etcd cluster) in the target cluster is detected to identify faulty nodes. This detection is performed using management tools (such as etcdctl), with the specific commands as follows:

[0075] ETCDCTL_API=3 etcdctl --endpoints=https: / / [health node IP]:2379\

[0076] --cacert= / etc / kubernetes / pki / etcd / ca.crt \

[0077] --cert= / etc / kubernetes / pki / etcd / healthcheck-client.crt \

[0078] --key= / etc / kubernetes / pki / etcd / healthcheck-client.key \

[0079] endpoint status --write-out=table

[0080] After determining the faulty node according to the detection result, the following command is executed on the normal management node to remove the faulty node from the target cluster:

[0081] ETCDCTL_API=3 etcdctl --endpoints=https: / / [healthy node IP]:2379 \

[0082] --cacert= / etc / kubernetes / pki / etcd / ca.crt \

[0083] --cert= / etc / kubernetes / pki / etcd / healthcheck-client.crt \

[0084] --key= / etc / kubernetes / pki / etcd / healthcheck-client.key \

[0085] member remove [faulty node ID]

[0086] Copy at least one certificate from the target management node to the new management node to ensure secure access, and obtain the latest data snapshot of the distributed key-value storage system from the healthy node of the target cluster, restore the snapshot on the new management node to ensure data consistency, then restore the snapshot on the new node, and start the distributed key-value storage system service of the new node to ensure normal access of the new node, and improve the recovery reliability and cluster stability.

[0087] In step S103, the priority of the high availability daemon service of the new management node is set to the lowest level in the target cluster, the high availability daemon service of the new management node is restarted, and the taint is removed, to complete the recovery action of the target management node.

[0088] The high availability daemon service refers to a key process that runs continuously in the background of the management node to ensure the high availability of the core service of the cluster. The priority refers to a parameter used to define the priority level of a node or service in resource allocation, fault handling, role election, etc. The lowest level refers to the lowest priority level of a node or service in the priority parameter sequence.

[0089] Specifically, a network failover component (such as Keepalived) needs to be installed and configured to ensure that its configuration is consistent with the original cluster. The priority of the high-availability daemon service of the new management node is set to the lowest level in the target cluster to avoid interfering with the cluster high-availability mechanism when the new node is not stable; restarting the high-availability daemon service makes the priority configuration take effect; at the same time, the new management node is removed from the taint, allowing workloads to be scheduled to this node, thereby completing the recovery of the target management node.

[0090] For example, execute the command:

[0091] kubectl taint node ${recover_master_name} inais-recovery-

[0092] After the new node is fully recovered, the taint is removed to allow the multi-replica container to restart, ensuring that the running development environment, training tasks, inference services, and other containerized services are not affected. Multi-replica microservices automatically create pod containers on the new node, and the entire process of replacing the down node with the new node is completed.

[0093] Optionally, in some embodiments, setting the priority of the high-availability daemon service of the new management node to the lowest level in the target cluster includes: determining a priority set of at least part of the management nodes in the target cluster; determining an initial priority of the high-availability daemon service of the new management node based on the priority set; determining a new priority set of at least part of the management nodes in the target cluster, and verifying the initial priority based on the new priority set to obtain a verified priority; and setting the verified priority as the lowest level in the target cluster.

[0094] Specifically, for example, when the Keepalived service is restarted, the virtual Internet Protocol address will be preferentially drifted to the node with the maximum value of the priority parameter (a numerical parameter used to define node priority) in the service configuration. Therefore, in order to avoid the virtual Internet Protocol address drifting to the new management node and causing the service to be unavailable, the priority parameter priority in the keepalived.conf configuration of the recovery node needs to be set to the minimum value among all management nodes. The implementation method includes: determining a priority set of at least part of the management nodes in the target cluster, and setting an initial priority of the high-availability daemon service (such as Keepalived) of the new management node based on the priority set; verifying the initial priority based on a new priority set of part of the management nodes in the cluster to obtain a verified priority, and setting the verified priority as the minimum value among all management nodes and writing it into the keepalived.conf configuration of the new management node.

[0095] Optionally, in some embodiments, the determining of the priority set of the at least part of the management nodes in the target cluster comprises: collecting a configuration file of a high availability daemon service of the at least part of the management nodes in the target cluster; identifying, based on the configuration file, a priority parameter value corresponding to an Internet Protocol address of each management node; and obtaining the priority set of the at least part of the management nodes in the target cluster according to the priority parameter value corresponding to the Internet Protocol address of each management node.

[0096] Specifically, the configuration file (i.e., keepalived.conf) of the high availability daemon service (e.g., Keepalived) of the at least part of the management nodes (preferably, all the management nodes) in the target cluster is collected in batches through an automated script; based on the configuration file, a virtual Internet Protocol address related priority parameter value corresponding to the Internet Protocol address of each management node is identified (assuming that the parameter value is in the default range of [1, 255]); and the priority set P = [p1, p2, …, pn] of the at least part of the management nodes in the target cluster is obtained according to the priority parameter value corresponding to the Internet Protocol address of each management node, where n is the number of the management nodes. Thus, the priority parameter value corresponding to the Internet Protocol address of each management node is identified through the configuration file, which greatly improves the efficiency of the priority pulling and facilitates the determination of the priority set.

[0097] Further, in some embodiments, the determining of the initial priority of the high availability daemon service of the new management node based on the priority set comprises: determining a reference minimum value of a priority configuration parameter based on the priority set; and determining the initial priority of the high availability daemon service of the new management node according to the difference between the reference minimum value and a preset threshold.

[0098] The reference minimum value refers to the minimum priority parameter value determined from the collected priority set of the management nodes, i.e., the element with the smallest value in the set. The preset threshold can be a threshold preset by a user, a threshold obtained through a limited number of experiments, or a threshold obtained through a limited number of computer simulations.

[0099] Specifically, based on the priority set of the management nodes of the target cluster, the minimum value min_p = min(P) is calculated and determined as the reference minimum value of the priority configuration parameter; in order to ensure that the priority of the recovery node is greater than 1 than that of other nodes and provide a buffer space, a safety redundancy value δ (δ is a positive integer, and is recommended to be 2, and the threshold can be modified) is set to avoid priority critical conflicts (e.g., the priority of other nodes suddenly decreases to min_p-1); and the initial priority of the high availability daemon service of the new management node is determined according to the difference between the reference minimum value and the threshold, and the calculation formula is:

[0100] .

[0101] Thus, by setting a security redundancy value (i.e. a preset threshold value), the priority critical conflict is avoided, and it is effectively ensured that the value is much smaller than any future possible new value of the rest of the nodes, effectively avoiding the problem that the virtual Internet Protocol address drifts to the new management node to cause service unavailability.

[0102] Further, in some embodiments, based on the new priority set, the initial priority is checked to obtain a checked priority, including: judging whether the initial priority is less than any priority in the new priority set; if the initial priority is less than any priority in the new priority set, the initial priority is taken as the checked priority.

[0103] Specifically, taking Keepalived as an example, considering that the Keepalived priority value of the cluster in the actual scene may change dynamically, that is, there may be a delay between the collected data set P and the actual configuration, such as the priority of a certain node being 100 when collecting and being reduced to 98 when verifying, resulting in an inaccurate initial priority set, therefore, a new priority set P is collected again, a checking module is added, and it is checked whether the initial priority priority_temp satisfies being less than all node priorities, that is, priority_temp < pn for any pn ∈ P is true. If it is true, the initial priority is taken as the checked priority. Thus, by judging whether the initial priority is less than any priority in the new priority set, the problem that the virtual Internet Protocol address drifts to the new management node to cause service unavailability is effectively avoided.

[0104] Further, in some embodiments, in the judgment of whether the initial priority is less than any priority in the new priority set, including: if the initial priority is not less than any priority in the new priority set, the initial priority is iteratively corrected based on a preset correction strategy until the corrected priority is less than any priority in the new priority set; the corrected priority is taken as the checked priority.

[0105] The preset correction strategy refers to a set of explicit and executable rules or methods when the initial priority of the new management node does not satisfy the condition of being less than all priorities in the new priority set. Iterative correction refers to a loop process of repeatedly adjusting and checking the priority according to the preset correction strategy.

[0106] Specifically, taking Keepalived as an example, if the initial priority is not less than any priority in the new priority set (i.e., pn ≤ priority_temp), then based on a preset correction strategy (e.g., priority_temp = priority_temp - 1), iterative correction is performed, and verification is repeated until the condition is met. The corrected priority is used as the priority that passes the verification. The final verified priority value is updated in the service configuration of the recovery node, and the high availability daemon service is restarted. Thus, through iterative correction, the priority value is kept to a minimum, effectively preventing the problem of service unavailability caused by the virtual Internet Protocol address drifting to the new management node.

[0107] To facilitate those skilled in the art to further understand the priority configuration method of the embodiments of this application, the following is combined with... Figure 4 Further explanation is needed.

[0108] like Figure 4 As shown, Figure 4 This is a flowchart of a priority configuration method according to an embodiment of this application, the priority configuration method including the following steps:

[0109] S401 collects cluster priority data.

[0110] S402, set the baseline minimum value min_p and the safety redundancy value.

[0111] S403, Set the initial priority of the new management node.

[0112] S404, collect priority data again.

[0113] S405, check if pn≤priority_temp exists. If it does, proceed to step S406; otherwise, proceed to step S407.

[0114] S406, iteratively correct the global minimum value, then return to step S403.

[0115] S407, update Keepalived priority configuration and start the service.

[0116] Therefore, based on the configuration of the target management node, a new node is created, taints are added to the new node through the container orchestration platform, the faulty node in the cluster is removed and the distributed key-value storage service is restarted, the high availability daemon of the new node is set to the lowest priority in the cluster, the service is restarted and the taints are removed, thus completing the recovery. This solves the problems of complex and time-consuming recovery processes and the impact on the availability of running services and platform pages in existing technologies, achieving fast and stable recovery of the management node.

[0117] Furthermore, to ensure that the Internet Protocol address of the new node is consistent with the Internet Protocol address of the original failed node, a check is added during startup to determine whether the Internet Protocol address of the new node is the same as the original address.

[0118] Optionally, in some embodiments, when starting the distributed key-value storage system service of the new management node, the method further includes: determining whether the Internet Protocol address of the new management node is consistent with the Internet Protocol address of the target management node; if the Internet Protocol address of the new management node is consistent with the Internet Protocol address of the target management node, then restarting the distributed key-value storage system service of the new management node.

[0119] Optionally, in some embodiments, after determining whether the Internet Protocol address of the new management node is consistent with the Internet Protocol address of the target management node, the method further includes: if the Internet Protocol address of the new management node is inconsistent with the Internet Protocol address of the target management node, restarting the distributed key-value storage system service of at least some nodes in the target cluster.

[0120] Specifically, if the Internet Protocol address of the new node is the same as the original Internet Protocol address, then only the distributed key-value storage system service of the new management node needs to be restarted; if the Internet Protocol address of the new node is different from the original Internet Protocol address, then the distributed key-value storage system service on all nodes of the cluster needs to be restarted. The purpose is to refresh the environment variable information required by the distributed key-value storage system service of the cluster and ensure that the new Internet Protocol address information is updated in the distributed key-value storage system database.

[0121] To facilitate those skilled in the art to further understand the distributed key-value storage system service recovery method of the embodiments of this application, the following is combined with... Figure 5 Further explanation is needed.

[0122] like Figure 5 As shown, Figure 5 This is a flowchart of a distributed key-value storage system service recovery method according to an embodiment of this application. The distributed key-value storage system service recovery method includes the following steps:

[0123] S501, check the etcd cluster status and obtain the faulty node ID.

[0124] S502, removes the faulty node from the etcd cluster.

[0125] S503, install the etcd service and copy the certificate on the new node.

[0126] S504 creates a snapshot from a healthy node and replicates it to a new node.

[0127] S505, determine whether the new node IP is the same as the original node. If yes, proceed to step S506; otherwise, proceed to step S507.

[0128] S506, restart the etcd service on the recovery node (i.e., the new management node).

[0129] S507, restart the etcd service on all nodes.

[0130] S508, etcd service recovery complete.

[0131] This ensures that the distributed key-value storage system service of the normal management node does not restart during the management node replacement process, thus guaranteeing the stability of the distributed key-value storage system cluster.

[0132] Furthermore, to facilitate those skilled in the art to further understand the management node crash recovery method of the embodiments of this application, the following detailed description is provided in conjunction with the artificial intelligence training platform involved in the management node crash recovery method of the embodiments of this application.

[0133] like Figure 6 As shown, Figure 6 This is a schematic diagram of an artificial intelligence training platform provided according to an embodiment of this application. In a high-availability cluster of an artificial intelligence development platform, the basic components of the management node and compute nodes are as follows: Management node: typically deploys components such as (API Server, Scheduler, Controller Manager), etcd database, Docker container engine, Keepalived (used to implement virtual IP migration and ensure high availability), MariaDB database (stores platform business data), and Harbor (container image repository); Compute node: deploys node services (for communication with the management node), development environment (for developers to write and debug code), training tasks (running the artificial intelligence model training process), inference service (providing model inference capabilities), and Docker components (for running containers), etc.

[0134] In the process of replacing the target management node, the application embodiment proposes a series of ordered steps and targeted measures to ensure the stability of the cluster and the availability of services during the recovery process. Specifically, it includes: environment preparation and basic configuration: prepare a new node consistent with the original node information, then configure the cluster between the non-encrypted communication and install the basic software such as docker; Recover k8s control plane components: when the node joins the k8s cluster, set the taint to prevent multiple containers of the cluster business module from being created on the recovery node, avoiding traffic entering the management node that has not been fully recovered, causing service unavailable; Recover etcd service: avoid etcd service restart on normal nodes during etcd service recovery, causing service interruption; Recover Keepalived service: through the Keepalived priority configuration process scheme, ensure that the priority of Keepalived service configuration on the recovery node is the lowest, avoid the virtual IP drift to the recovery node when the virtual IP drifts, causing the page unavailable; Finally, remove the taint of the recovery node to restart the multiple copies of the business module microservices, and finally complete the recovery of all services on the management node.

[0135] Further, as shown in Figure 7 , Figure 7 a flowchart of a management node downtime recovery method according to an embodiment of the application is provided, the management node downtime recovery method comprising the following steps:

[0136] S701, environment preparation and basic configuration.

[0137] S702, recover k8s control plane components.

[0138] S703, recover etcd service.

[0139] S704, recover Keepalived service.

[0140] S705, remove taint and start all microservices on the recovery node.

[0141] Thus, it can ensure that the platform can be used normally during the recovery process of the management node, and ensure that the running development environment, training task, inference service and other containerized services are not affected, and the stability of the cluster is improved.

[0142] According to the management node failure recovery method provided in the embodiment of the present application, in the case that the target management node is in a failure state, a new node is configured according to the target management node, the new node is set with a stain through a container orchestration platform, a fault node of a cluster is removed and a distributed key-value storage service is restarted, the priority of a high-availability daemon process of the new node is set to the lowest in the cluster, the service is restarted and the stain is removed, and the recovery is completed. Thus, the problems of complex recovery process, time consumption and influence on the availability of a running service and platform page in the prior art are solved, and the fast and stable recovery of the management node is achieved.

[0143] Secondly, the management node failure recovery device according to one embodiment of the present application is described with reference to the accompanying drawings.

[0144] Figure 8 is a block schematic diagram of an edge multi-source flow aggregation transmission device according to one embodiment of the present application.

[0145] As shown in Figure 8 , the edge multi-source flow aggregation transmission device 10 comprises a matching module 100, a processing module 200 and a determination module 300.

[0146] The matching module 100 configures a new management node according to the configuration information of the target management node in the case that the target management node is in a failure state.

[0147] The processing module 200 sets a stain for the new management node through a preset container orchestration platform component, removes a fault node in a target cluster to which the target management node belongs, and restarts a distributed key-value storage system service of the target cluster.

[0148] The determination module 300 sets the priority of a high-availability daemon process service of the new management node to the lowest in the target cluster, restarts the high-availability daemon process service of the new management node, and removes the stain, so as to complete the recovery action of the target management node.

[0149] According to one embodiment of the present application, the matching module 100 is specifically configured to: identify the operating system of the target management node, and install the operating system for the new management node; configure the Internet protocol address of the new management node based on the Internet protocol address of the target management node; generate a key pair in the new management node, and send the public key in the key pair to the target cluster to which the target management node belongs, and configure the new management node to perform keyless communication with other nodes in the target cluster; install the containerized platform component with the same version as the target cluster for the new management node, and configure the new management node to start the containerized platform component corresponding to the container service automatically.

[0150] According to an embodiment of the present application, the matching module 100 is specifically configured to: based on the Internet protocol address, execute a preset communication test instruction through any node of the target cluster, and determine whether the Internet protocol address of the target management node is in an occupied state; if the Internet protocol address of the target management node is in the occupied state, release the Internet protocol address, and after the Internet protocol address of the target management node is released, configure the Internet protocol address of the new management node, so that the Internet protocol address of the new management node is the same as the Internet protocol address of the target management node.

[0151] According to an embodiment of the present application, the processing module 200 is specifically configured to: add the new management node to a node list of the target cluster; and when initializing a preset container orchestration platform component, set a taint for the new management node by using a preset management tool, the taint being configured to prohibit creating multiple copies of containers on the new management node.

[0152] According to an embodiment of the present application, the processing module 200 is specifically configured to: determine any healthy node from the target cluster, and detect a state of a distributed key-value storage system in the target cluster by using the any healthy node to obtain a detection result; determine a faulty node according to the detection result, and remove the faulty node from the target cluster; copy at least one certificate from the target management node to the new management node, and obtain a latest data snapshot of the distributed key-value storage system from the healthy node of the target cluster, and restore the latest data snapshot of the distributed key-value storage system in the new management node, and start a distributed key-value storage system service of the new management node.

[0153] According to an embodiment of the present application, the processing module 200 is further configured to: determine whether the Internet protocol address of the new management node is the same as the Internet protocol address of the target management node; if the Internet protocol address of the new management node is the same as the Internet protocol address of the target management node, restart the distributed key-value storage system service of the new management node.

[0154] According to an embodiment of the present application, the processing module 200 is further configured to: if the Internet protocol address of the new management node is not the same as the Internet protocol address of the target management node, restart the distributed key-value storage system service of at least part of the nodes in the target cluster.

[0155] According to an embodiment of the present application, the determining module 300 is specifically configured to: determine a priority set of at least part of the management nodes in the target cluster; determine an initial priority of a high-availability daemon service of the new management node based on the priority set; determine a new priority set of at least part of the management nodes in the target cluster, and verify the initial priority based on the new priority set to obtain a verified priority; and take the verified priority as the lowest priority in the target cluster.

[0156] According to an embodiment of the present application, the determining module 300 is specifically configured to: collect a configuration file of the high-availability daemon service of at least part of the management nodes in the target cluster; identify, based on the configuration file, a priority parameter value corresponding to an Internet Protocol address of the management node; and obtain a priority set of at least part of the management nodes in the target cluster according to the priority parameter value corresponding to the Internet Protocol address of the management node.

[0157] According to an embodiment of the present application, the determining module 300 is specifically configured to: determine a reference minimum value of the priority configuration parameter based on the priority set; and determine an initial priority of the high-availability daemon service of the new management node according to a difference between the reference minimum value and a preset threshold.

[0158] According to an embodiment of the present application, the determining module 300 is specifically configured to: determine whether the initial priority is less than any priority in the new priority set; and if the initial priority is less than any priority in the new priority set, take the initial priority as the priority that passes the verification.

[0159] According to an embodiment of the present application, the determining module 300 is specifically configured to: if the initial priority is not less than any priority in the new priority set, iteratively correct the initial priority based on a preset correction strategy until the corrected priority is less than any priority in the new priority set; and take the corrected priority as the priority that passes the verification.

[0160] According to the management node downtime recovery apparatus provided by the embodiment of the present application, in the case that the target management node is in a downtime state, a new node is configured according to the target management node, the new node is given a taint through a container orchestration platform, a cluster fault node is removed and a distributed key-value storage service is restarted, the high-availability daemon priority of the new node is set to the lowest in the cluster, the service is restarted and the taint is removed, and the recovery is completed. Thus, the problems of complex recovery process, time consumption and influence on the availability of the running service and platform page in the prior art are solved, and the fast and stable recovery of the management node is achieved.

[0161] Figure 9 The electronic device provided by the embodiment of the present application is shown in a structural schematic diagram. The electronic device can include:

[0162] The memory 901, the processor 902, and the computer program stored in the memory 901 and executable on the processor 902.

[0163] The processor 902 implements the management node downtime recovery method provided in the above embodiments when executing the program.

[0164] Further, the electronic device further includes:

[0165] The communication interface 903 is configured to communicate between the memory 901 and the processor 902.

[0166] The memory 901 is configured to store a computer program executable in the processor 902.

[0167] The memory 901 can include a high-speed RAM (Random Access Memory) memory, and can further include a non-volatile memory, for example, at least one disk memory.

[0168] If the memory 901, the processor 902 and the communication interface 903 are independently implemented, the communication interface 903, the memory 901 and the processor 902 can be connected through a bus and communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, Figure 9 Only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus.

[0169] Optionally, in a specific implementation, if the memory 901, the processor 902 and the communication interface 903 are integrated on a chip, the memory 901, the processor 902 and the communication interface 903 can communicate with each other through an internal interface.

[0170] The processor 902 can be a CPU (Central Processing Unit) or an ASIC (Application Specific Integrated Circuit) or one or more integrated circuits configured to implement the embodiments of the present application.

[0171] The embodiments of the present application also provide a non-volatile computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the method for recovering a down management node as above.

[0172] The embodiments of the present application also provide a computer program product, which includes a computer program, and the computer program is executed by a processor to implement the method for recovering a down management node as above.

[0173] Those skilled in the art will further realize that the mere concepts, teachings, and embodiments described herein are merely meant to provide an enabling description of the applications and are not intended to limit the scope of the applications. Therefore, embodiments or examples described herein are not meant to be limiting, but merely to aid in the understanding of the overall more complete range of variations that can be implemented in accordance with the concepts, teachings, and embodiments described herein. It will be apparent to those having ordinary skill in the art that various changes and modifications can be made therein without departing from the spirit and scope of the applications. Thus, the various aspects of the applications can be embodied in many different forms, and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments were chosen and described so that this disclosure is thorough and complete, and fully conveys the scope of the applications to others skilled in the art. As such, the applications are not limited to the specific embodiments described herein, but cover all modifications that can fall within the scope of the applications.

[0174] The method for recovering a down node of a management node is described in detail above. The principles and implementation manners of the present application are described by applying specific examples. The above description of the embodiments is only applicable to helping understand the method and core idea of the present application. It should be pointed out that, for those skilled in the art, some improvements and modifications can be made to the present application without departing from the principles of the present application. These improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A method for recovering a management node from failure, characterized in that, Includes the following steps: If the target management node is down, configure a new management node according to the configuration information of the target management node; The new management node is tainted by a preset container orchestration platform component, and the faulty node in the target cluster to which the target management node belongs is removed, and the distributed key-value storage system service of the target cluster is restarted. Set the priority of the high availability daemon service of the new management node to the lowest level in the target cluster, restart the high availability daemon service of the new management node, and remove the taint to complete the recovery action of the target management node; In this context, taint refers to an exclusionary attribute imposed on a node in a container orchestration platform to restrict container groups from being scheduled to that node. The step of setting taints for the new management node using a preset container orchestration platform component includes: adding the new management node to the node list of the target cluster; and setting taints for the new management node using a preset management tool when initializing the preset container orchestration platform component, wherein the taints are configured to prohibit the creation of multi-replica containers on the new management node. The step of removing the faulty node from the target cluster to which the target management node belongs and restarting the distributed key-value storage system service of the target cluster includes: determining any healthy node from the target cluster, and using the healthy node to detect the status of the distributed key-value storage system in the target cluster to obtain a detection result; determining the faulty node based on the detection result and removing the faulty node from the target cluster; copying at least one certificate from the target management node to the new management node, obtaining the latest distributed key-value storage system data snapshot from the healthy node of the target cluster, restoring the latest distributed key-value storage system data snapshot on the new management node, and starting the distributed key-value storage system service of the new management node.

2. The method for recovering a management node from a crash according to claim 1, characterized in that, The step of configuring a new management node based on the configuration information of the target management node includes: Identify the operating system of the target management node and install the operating system on the new management node; Configure the Internet Protocol address of the new management node based on the Internet Protocol address of the target management node; A key pair is generated on the new management node, and the public key in the key pair is sent to the target cluster to which the target management node belongs. Passwordless communication is configured between the new management node and other nodes in the target cluster. Install the same containerized platform component as the target cluster version on the new management node, and configure the new management node to automatically start the container service corresponding to the containerized platform component upon startup.

3. The method for recovering a management node from a crash as described in claim 2, characterized in that, The step of configuring the Internet Protocol address of the new management node based on the Internet Protocol address of the target management node includes: Based on the Internet Protocol address, a preset communication test command is executed through any node of the target cluster, and it is determined whether the Internet Protocol address of the target management node is in use. If the Internet Protocol address of the target management node is in the occupied state, the Internet Protocol address is released, and after the Internet Protocol address of the target management node is released, the Internet Protocol address of the new management node is configured so that the Internet Protocol address of the new management node is the same as the Internet Protocol address of the target management node.

4. The method for recovering a management node from a crash according to claim 1, characterized in that, When starting the distributed key-value storage system service of the new management node, the following is also included: Determine whether the Internet Protocol address of the new management node is consistent with the Internet Protocol address of the target management node; If the Internet Protocol address of the new management node is the same as that of the target management node, then the distributed key-value storage system service of the new management node is restarted.

5. The method for recovering a management node from a crash according to claim 4, characterized in that, After determining whether the Internet Protocol address of the new management node is consistent with the Internet Protocol address of the target management node, the method further includes: If the Internet Protocol address of the new management node is inconsistent with the Internet Protocol address of the target management node, then the distributed key-value storage system service of at least some nodes in the target cluster shall be restarted.

6. The method for recovering a management node from a crash according to claim 1, characterized in that, Setting the priority of the high availability daemon service of the new management node to the lowest level in the target cluster includes: Determine the priority set of at least a portion of the management nodes in the target cluster; Based on the priority set, the initial priority of the high availability daemon service of the new management node is determined; A new priority set is determined for at least some of the management nodes in the target cluster, and the initial priority is verified based on the new priority set to obtain the priority that passes the verification. The priority of those that pass the verification will be the lowest level in the target cluster.

7. The method for recovering a management node from a crash as described in claim 6, characterized in that, Determining the priority set of at least a portion of the management nodes in the target cluster includes: Collect the configuration files of the high availability daemon service of at least some of the management nodes in the target cluster; Based on the configuration file, identify the priority parameter value corresponding to the Internet Protocol address of the management node; The priority set of at least some management nodes in the target cluster is obtained based on the priority parameter value corresponding to the Internet Protocol address of the management node.

8. The method for recovering a management node from a crash as described in claim 6, characterized in that, The process of determining the initial priority of the high availability daemon service of the new management node based on the priority set includes: Based on the priority set, determine the baseline minimum value of the priority configuration parameter; The initial priority of the high availability daemon service of the new management node is determined based on the difference between the minimum baseline value and the preset threshold.

9. The method for recovering a management node from a crash according to claim 6, characterized in that, The step of verifying the initial priority based on the new priority set to obtain the priority that passes the verification includes: Determine whether the initial priority is less than any priority in the new priority set; If the initial priority is less than any priority in the new priority set, then the initial priority is used as the priority for passing the verification.

10. The method for recovering a management node from a crash according to claim 9, characterized in that, Determining whether the initial priority is less than any priority in the new priority set includes: If the initial priority is not less than any priority in the new priority set, then the initial priority is iteratively corrected based on a preset correction strategy until the corrected priority is less than any priority in the new priority set. The corrected priority will be used as the priority for passing the verification.

11. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the management node crash recovery method as described in any one of claims 1-10.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the downtime recovery method for the management node as described in any one of claims 1-10.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the management node crash recovery method as described in any one of claims 1-10.

Images