A data center network element fault scheduling method and electronic equipment

Abstract

This invention provides a data center network element fault scheduling method and electronic device. The data center cluster includes availability zones, and representative network nodes are pre-selected in each availability zone. All representative network nodes in all availability zones are equipped with an arbitration module, a scheduling module, and a network configuration storage cluster. The method includes: a main arbitration module sending network node probe requests to all arbitration modules, enabling the arbitration modules to probe all network nodes according to the probe requests; receiving the probe results from all arbitration modules and determining whether a faulty network node exists based on a joint arbitration mechanism; if a faulty network node exists, updating the network configuration storage cluster based on the faulty network node information, so that the main scheduling module can schedule network elements running on the faulty network node to non-faulty network nodes within the data center cluster based on the updated faulty network node information in the network configuration storage cluster. This method offers fast detection, high real-time performance, and faster fault scheduling response.

Description

Technical Field

[0001] This invention relates to the field of data exchange network technology, and in particular to a data center network element fault scheduling method and an electronic device. Background Technology

[0002] Network elements serve as crucial bridges for north-south communication between a user's VPC (Virtual Private Cloud) and the outside world. Network elements are typically designed to run on specific network nodes. If a network node fails, the network element will cease functioning properly, and the data center will be unable to provide services. When a network node fails or a network partition becomes isolated, the problematic network node should be isolated immediately, and the network elements running on it should be relocated to other functioning network nodes to continue providing network services.

[0003] Traditional network devices run Bidirectional Forwarding Detection (BFD) to quickly detect communication failures between devices, enabling timely interventions such as switching routes to backup devices to ensure continuous service operation. BFD is a point-to-point fault detection method, where network devices establish BFD with both uplink and downlink devices. However, data centers typically run virtualized networks such as Open vSwitch (OVS). In this case, the uplink devices of network elements are physical switches, which can run BFD, but the downlink devices are virtualized network devices. The next hop for forwarding downlink traffic is the virtual switch gateway, but gateways are distributed and cannot establish BFD like physical devices. One approach is to fix the virtual switch gateway to a specific host node, but this introduces new single points of failure and traffic forwarding bottlenecks, making it impractical.

[0004] Essentially, the physical carriers of the virtual devices connected to a network element are all the computing nodes. Therefore, all computing nodes need to run a BFD-like probe with the network element. This can lead to a large amount of unnecessary probe traffic when there are many network elements. Furthermore, each computing node will have its own decision, while management often wants to know which network element is providing services (such as understanding the load of each network node), which will reduce control capabilities.

[0005] Existing solutions include: network nodes running VRRP (Virtual Router Redundancy Protocol). In VRRP election, the node that becomes the leader takes over all network elements, while other nodes become backup nodes and do not provide external services. The problems with this approach are: VRRP operates on a primary / backup model, meaning only one network node is active at a time, resulting in significant resource waste; furthermore, VRRP typically requires participating nodes to be located in the same Layer 2 network, which is difficult to provide for network nodes distributed across multiple data centers, as multi-availability zone data centers often operate in multiple locations. Therefore, VRRP is unsuitable for scenarios with multiple availability zones and large-scale clusters, and is more suitable for small-scale, single-availability zone scenarios.

[0006] Existing solutions involve using a fixed monitoring service, either external or internal to the cluster, to probe the availability of network nodes and then schedule network elements accordingly. The problems with this approach are: it heavily relies on the physical link between the monitoring service and the network nodes, often leading to false alarms and instability in network element services; and the availability of the monitoring and scheduling services themselves is not guaranteed. Furthermore, if these services are deployed externally to the cluster, the probes become unreliable due to long links; if deployed in an availability zone within the cluster, service is unavailable if that availability zone fails.

[0007] In summary, we need a reliable, cluster-autonomous method for network element fault scheduling that is applicable to large-scale, multi-availability zone data centers. Summary of the Invention

[0008] This invention provides a data center network element fault scheduling method, apparatus, electronic device, and computer-readable storage medium, applicable to network element fault scheduling in large-scale, multi-availability zone data center clusters.

[0009] This invention discloses a data center network element fault scheduling method. The data center cluster includes one or more availability zones. Representative network nodes are pre-selected in each availability zone. An arbitration module, a scheduling module, and a network configuration storage cluster are deployed in the representative network nodes of all availability zones. The method includes:

[0010] The main arbitration module sends a network node probe request to all arbitration modules, so that all arbitration modules can probe all network nodes according to the network node probe request; the main arbitration module is determined by all arbitration modules through competition.

[0011] The main arbitration module receives the detection results from all arbitration modules and determines whether there are faulty network nodes based on the joint arbitration mechanism;

[0012] If the faulty network node information exists, the network configuration storage cluster is updated based on the faulty network node information, so that the main scheduling module can schedule the network elements running on the faulty network node to the non-faulty network node in the data center cluster based on the updated faulty network node information in the network configuration storage cluster.

[0013] Optionally, before the main arbitration module sends a network node probe request to all arbitration modules, the method further includes:

[0014] All arbitration modules within the network nodes elect a master arbitration module using a distributed lock mechanism.

[0015] And / or,

[0016] All scheduling modules within the network nodes elect a master scheduling module through a distributed lock mechanism.

[0017] Optionally, the main arbitration module sends a network node probe request to all arbitration modules, including:

[0018] The main arbitration module sends a network element information list retrieval request to the network configuration storage cluster and receives the list information returned by the network configuration storage cluster based on the network element information list retrieval request;

[0019] The main arbitration module sends a network node probe request to all arbitration modules based on the list information. The network node probe request includes information for probing each network node in the list information.

[0020] Optionally, the main arbitration module receives the probe results from all arbitration modules and determines whether a faulty network node exists based on the joint arbitration mechanism, including:

[0021] The main arbitration module summarizes the response message information of each network node in the detection results. The response message is the message in which the network node responds to the detection message sent by the arbitration module within a specified time.

[0022] Determine whether a majority of the arbitration modules have detected a network node as unavailable.

[0023] Optionally, the network configuration storage cluster is a distributed key-value database, Etcd.

[0024] This invention also discloses a data center network element fault scheduling method. The data center cluster includes one or more availability zones. Representative network nodes are pre-selected in each availability zone. An arbitration module, a scheduling module, and a network configuration storage cluster are deployed in the representative network nodes of all availability zones. The method includes:

[0025] If the main scheduling module detects a faulty network node in the network configuration storage cluster, it will send a request to the network configuration storage cluster to obtain the configuration information of the faulty network node.

[0026] The main scheduling module receives the network element configuration information of the faulty network node returned by the network configuration storage cluster based on the configuration information.

[0027] The main scheduling module schedules network elements running on faulty network nodes to non-faulty network nodes based on network element configuration information and information of non-faulty network nodes in the data center cluster.

[0028] The faulty network node is an unavailable network node detected by the interaction between the main arbitration module and all arbitration modules, and is updated in the network configuration storage cluster.

[0029] Optionally, the main scheduling module, based on network element configuration information and information about non-faulty network nodes in the data center cluster, schedules network elements running on faulty network nodes to non-faulty network nodes, including:

[0030] The main scheduling module schedules network elements running on faulty network nodes to higher-priority non-faulty network nodes based on network element configuration information and the priority of non-faulty network nodes.

[0031] Optionally, the method further includes:

[0032] The main scheduling module adjusts the priority of network nodes and updates the network configuration storage cluster based on the frequency and duration of network node failures.

[0033] Optionally, if a shadow network element exists on a non-faulty network node, the main scheduling module, based on the network element configuration information and the information of the non-faulty network nodes in the data center cluster, schedules the network elements running on the faulty network node to the non-faulty network node, including:

[0034] The main scheduling module schedules network elements running on faulty network nodes to shadow network elements on non-faulty network nodes based on network element configuration information.

[0035] The shadow network element has the same network element configuration as the faulty network node and is in a ready but not providing service state.

[0036] This invention also discloses an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0037] The memory is used to store computer programs;

[0038] When the processor executes a program stored in the memory, it implements the method described in the embodiments of the present invention.

[0039] This invention also discloses one or more computer-readable media storing instructions that, when executed by one or more processors, cause the processors to perform the methods described in this invention.

[0040] The embodiments of the present invention have the following advantages:

[0041] The method in this embodiment of the invention interacts with the network configuration storage cluster through the main arbitration module to identify unavailable network nodes. This allows the main scheduling module to reschedule network elements running on unavailable network nodes to other normally available network nodes to continue providing network services. This achieves reliable, autonomous, and rapid network element failure scheduling. In particular, the entire link module maintains high availability during the detection process, with no split-brain scenarios.

[0042] The method in this embodiment of the invention interacts with the network configuration storage cluster through the main scheduling module to schedule network elements running on unavailable network nodes to other normally available network nodes so as to continue to provide network services. It realizes that no external node intervention is required during the cluster autonomy process and enables rapid network element fault scheduling.

[0043] Furthermore, in this embodiment of the invention, the scheduling priority of network nodes can be dynamically adjusted according to the network node failure status, thereby enabling dynamic scheduling. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the availability zone in a data center cluster provided in an embodiment of the present invention;

[0045] Figure 2 This is a flowchart of the steps of a data center network element fault scheduling method provided in an embodiment of the present invention;

[0046] Figure 3 This is a flowchart of the steps of a data center network element fault scheduling method provided in an embodiment of the present invention;

[0047] Figure 4 This is a signaling diagram of a data center network element fault scheduling method provided in an embodiment of the present invention;

[0048] Figure 5 This is a structural block diagram of a data center network element fault scheduling device provided in an embodiment of the present invention;

[0049] Figure 6 This is a block diagram of an electronic device provided in an embodiment of the present invention;

[0050] Figure 7This is a schematic diagram of a computer-readable medium provided in an embodiment of the present invention. Detailed Implementation

[0051] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0052] Reference Figure 1 This diagram illustrates the architecture of a data center cluster provided in an embodiment of the present invention. The data center cluster in this embodiment includes multiple network nodes, which can be physical servers. Each network node is configured with multiple network elements, meaning that multiple network elements run on the network node. In this embodiment, the data center cluster can be pre-divided into multiple availability zones. For example, the data center cluster may include more than one availability zone. Representative network nodes are pre-selected within each availability zone, and all representative network nodes in all availability zones deploy an arbitration module, a scheduling module, and a network configuration storage cluster.

[0053] In this embodiment, by dividing the data center cluster into multiple availability zones, even if one availability zone fails, the other availability zones can still operate normally, ensuring business continuity.

[0054] Meanwhile, load balancing technology can be used to distribute access pressure between different availability zones, optimizing resource utilization efficiency. When the load on a certain availability zone is too high, some traffic can be automatically or manually redirected to other availability zones to avoid single point of overload.

[0055] In this embodiment, the number of network nodes in each availability zone can be greater than one. The availability zone selects network nodes as representative network nodes proportionally based on the network node scale, such as one or two. This embodiment does not limit this selection; it is based on actual needs. Figure 1 The diagram shows the selection of two representative network nodes (or simply representative nodes) for each availability zone.

[0056] In the specific implementation process, servers with strong computing power are prioritized as representative network nodes to ensure that they can efficiently handle various tasks, especially those with high computing resource requirements. At the same time, considering that network nodes require multiple key components (arbitration module, scheduling module, and network configuration storage cluster), selecting servers with sufficient memory can reduce the risk of service termination due to insufficient memory. In addition, representative network nodes often need to communicate frequently with other nodes, so network bandwidth is an important consideration. Selecting servers with high network bandwidth can improve data transmission speed and reduce latency.

[0057] In this embodiment, each representative node deploys an arbitration module, a scheduling module, and a network configuration storage cluster. It is understood that in this embodiment, if a representative node fails, another representative node can be manually switched over. The arbitration module, scheduling module, and network configuration storage cluster deployed within each representative node can be directly deployed during cluster planning.

[0058] Selecting a representative node for each availability zone to centrally deploy critical components (arbitration module, scheduling module, and network configuration storage cluster) helps simplify management and maintenance. These components are essential for the stable operation of the entire data center, so their high availability and rapid response capabilities are particularly important.

[0059] If a representative node fails, the system should have the capability to automatically detect and quickly switch to a backup representative node to minimize service interruptions. Furthermore, regular fault recovery drills are essential to ensure stable operation in the event of a real failure; therefore, high availability and rapid response capabilities are particularly important.

[0060] Reference Figure 2 The diagram illustrates a flowchart of a data center network element fault scheduling method provided in an embodiment of the present invention, which may specifically include the following steps:

[0061] 201. The main arbitration module sends a network node probe request to all arbitration modules, so that all arbitration modules can probe all network nodes according to the network node probe request.

[0062] In this embodiment, the main arbitration module is determined through competition among all arbitration modules.

[0063] For example, all the arbitration modules representing the network nodes elect a master arbitration module through a distributed lock.

[0064] In this embodiment, common distributed lock implementation methods include Zookeeper and Etcd. These tools provide strongly consistent distributed coordination services, ensuring a fair and efficient election of a master arbitrator module among multiple nodes.

[0065] Each arbitration module periodically sends a heartbeat to the distributed lock service to maintain the lock's validity. If an arbitration module loses its heartbeat, the lock is released, and other arbitration modules can then attempt to acquire it.

[0066] At the same time, a reasonable timeout period is set to ensure that the lock can be released in time if a certain arbitration module is unresponsive for a long time, thus preventing deadlock.

[0067] Step 201 includes: the main arbitration module sending a network element information list retrieval request to the network configuration storage cluster, and receiving the list information returned by the network configuration storage cluster based on the network element information list retrieval request;

[0068] The main arbitration module sends a network node probe request to all arbitration modules based on the list information. The network node probe request includes information for probing each network node in the list information.

[0069] In this embodiment, the network configuration storage cluster stores the configuration information of all network elements running on the network nodes, the basic information of the network nodes, and the distribution information of the network elements.

[0070] Basic information of network element nodes: such as IP address, MAC address, hostname, operating system version, etc.

[0071] Network element configuration information: Configuration parameters of the network elements running on each network node, such as port number, service type, resource limits, etc.

[0072] Network distribution information: which network elements operate on which network nodes, and their distribution;

[0073] The network configuration storage cluster also stores health status information, specifically including the current health status of network nodes and network elements, such as CPU utilization, memory usage, and network bandwidth.

[0074] In the specific implementation process, the main arbitration module sends a request to the network configuration storage cluster to obtain a list of network element information. This request usually contains some necessary parameters, such as the identity of the requester and the request timestamp, to ensure the legality and timeliness of the request.

[0075] The network configuration storage cluster returns the generated list of network element information to the main arbitration module. The list information may include:

[0076] Network Node ID: A unique identifier for each network node;

[0077] Network element ID: A unique identifier for each network element;

[0078] Network element type: The type of network element, such as Web service, database service, etc.;

[0079] Network element configuration: The specific configuration parameters of the network element;

[0080] Network node status: The current status of a network node, such as online, offline, or under maintenance.

[0081] Furthermore, after receiving the list of network element information, the main arbitration module parses the information and extracts detailed information for each network node, including IP address, port number, etc.

[0082] The main arbitration module constructs network node probe requests based on the parsed network node information. Each probe request contains the following information:

[0083] Target network node ID, probe parameters, and probe commands;

[0084] The main arbitration module sends the constructed probe request to all arbitration modules. Each arbitration module will then probe the specified network node based on the received request.

[0085] 202. The main arbitration module receives the detection results from all arbitration modules and determines whether there is a faulty network node according to the joint arbitration mechanism. If there is, proceed to step 203 below; otherwise, continue to proceed to step 201 above according to the detection cycle.

[0086] For example, the main arbitration module summarizes the response message information of each network node in the detection results. The response message is the message that the network node replies to the detection message sent by the arbitration module within a specified time. It determines whether more than half of the arbitration modules have detected that a certain network node is unusable. If so, it determines that there is a faulty network node and obtains the information of the faulty network node.

[0087] Specifically, if more than half of the arbitration modules report a network node as unavailable, then that network node is considered faulty. The detection result of each arbitration module is equivalent to a vote; if more than half of the votes result in "unavailable," then the network node is considered a faulty node.

[0088] In some cases, different weights can be assigned to different arbitration modules to reflect their reliability or importance, and the final judgment can be based on weighted voting.

[0089] 203. If it exists, update the network configuration storage cluster according to the faulty network node information, so that the main scheduling module can schedule the network elements running on the faulty network node to the non-faulty network node in the data center cluster according to the faulty network node information in the updated network configuration storage cluster.

[0090] In addition, in this embodiment, all scheduling modules within the network nodes elect a master scheduling module through a distributed lock mechanism.

[0091] Furthermore, the main scheduling module selects a suitable target node to schedule the faulty network element based on the current network node status and load.

[0092] Among these considerations, load balancing is necessary, which means selecting nodes with lower loads to avoid scheduling multiple faulty network elements to the same node, which could lead to overload of that node. In addition, it is also necessary to ensure that the resource configuration of the target node (such as CPU, memory, and network bandwidth) can meet the needs of the faulty network elements.

[0093] In this embodiment, the network configuration storage cluster can be a distributed key-value database Etcd, which can ensure data consistency and high availability. Therefore, the main arbitration module and the main scheduling module mentioned above can be distributed in different representative nodes or located in the same representative node.

[0094] This embodiment's method interacts with the network configuration storage cluster through the main arbitration module to quickly identify faulty network nodes in the data center cluster. During implementation, multiple network nodes are simultaneously probed and jointly arbitrated, ensuring high reliability. This method requires no external node intervention and maintains high availability across the entire module during the probing process, preventing split-brain scenarios.

[0095] Reference Figure 3 The diagram illustrates a flowchart of a data center network element fault scheduling method provided in an embodiment of the present invention, which may specifically include the following steps:

[0096] 301. If the main scheduling module detects a faulty network node in the network configuration storage cluster, it will send a request to the network configuration storage cluster to obtain the configuration information of the faulty network node.

[0097] 302. The main scheduling module receives the network element configuration information of the faulty network node returned by the network configuration storage cluster based on the configuration information.

[0098] 303. The main scheduling module schedules network elements running on faulty network nodes to non-faulty network nodes based on the network element configuration information and the information of non-faulty network nodes in the data center cluster.

[0099] The faulty network node is an unavailable network node detected by the interaction between the main arbitration module and all arbitration modules, and is updated in the network configuration storage cluster.

[0100] In this embodiment, the main scheduling module can adjust the priority of network nodes and update them in the network configuration storage cluster based on the frequency and duration of network node failures.

[0101] Furthermore, in step 303 above, the main scheduling module can schedule network elements running on faulty network nodes to high-priority non-faulty network nodes based on network element configuration information and the priority of non-faulty network nodes.

[0102] In one alternative implementation, if a shadow network element exists on a non-faulty network node, step 303 above may include: the main scheduling module scheduling the network element running on the faulty network node to the shadow network element of the non-faulty network node according to the network element configuration information.

[0103] In this embodiment, the shadow network element can have the same network element configuration as the faulty network node and be in a ready-but-not-providing state. In this embodiment, not every network node in the data center cluster has a shadow network element; deployment is based on the data center cluster planning.

[0104] The method in this embodiment of the invention interacts with the network configuration storage cluster through a main scheduling module. After the main arbitration module determines that a network node is faulty, the main scheduling module can schedule network elements running on the faulty network node to other normal network nodes to continue providing network services. This achieves reliable, autonomous, and fast network element fault scheduling. The above method offers fast scheduling without the need for external node intervention, thus realizing the autonomy of the data center cluster.

[0105] The main scheduling module in this embodiment can dynamically adjust the priority of network nodes, and thus, in the scheduling of faulty network nodes, it can dynamically adjust according to the priority of normal network nodes, which is highly reliable.

[0106] Reference Figure 4 The diagram illustrates a flowchart of a data center network element fault scheduling method provided in an embodiment of the present invention, which may specifically include the following steps:

[0107] 401. The main arbitration module sends a request to the network configuration storage cluster to obtain the network element information list;

[0108] 402. The network configuration storage cluster retrieves the requested list information based on the network element information list;

[0109] 403. The main arbitration module sends a network node probe request to all arbitration modules;

[0110] 404. All arbitration modules send probe messages to each network node in the data center cluster according to the probe network node request;

[0111] 405. Each network node returns a response based on the received probe messages;

[0112] 406. Each arbitration module summarizes the responses received from the network nodes and sends its respective probe results to the main arbitration module;

[0113] 407. The main arbitration module integrates the detection results of all arbitration modules, combines the availability of arbitration network nodes, and determines whether there are unavailable network nodes as faulty network nodes or determines the availability status of network nodes.

[0114] 408. The main arbitration module updates the information of unavailable network nodes or the availability status of network nodes to the network configuration storage cluster.

[0115] 409. The main scheduling module monitors the availability of nodes in the network configuration storage cluster, makes the faulty network node unavailable, and obtains the network element distribution information, basic information of the network node, and network element related configurations from the network configuration storage cluster.

[0116] 410. The main scheduling module obtains the network element distribution information, basic network node information, and network element related configurations, etc., from the network configuration storage cluster returned by the faulty network node.

[0117] 411. The main scheduling module selects a suitable, normally functioning network node for the network element on the faulty network node.

[0118] 412. The main scheduling module schedules network elements to normal network nodes and updates information such as network element distribution and network node scheduling priority to the network configuration storage cluster.

[0119] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0120] Reference Figure 5 This diagram illustrates a structural block diagram of a data center network element fault scheduling device provided in an embodiment of the present invention. The data center network element fault scheduling device in this embodiment is located in a network node and is set up when the data center cluster is deployed. Each data center network element fault scheduling device may specifically include the following modules: an arbitration module, a scheduling module, and a network configuration storage cluster.

[0121] When deploying a data center cluster, each availability zone selects network nodes as representative nodes proportionally based on its size. Generally, the number of network nodes in each availability zone within the data center cluster is roughly equal, ensuring that each availability zone has an equal number of representative nodes, thus giving each availability zone equal decision-making power. The representative node deployment includes the aforementioned arbitration module, scheduling module, and data center network element fault scheduling device for the network configuration storage cluster.

[0122] In this embodiment, the network configuration storage cluster can be composed of Etcd deployed on representative nodes. Etcd is a highly available distributed key-value database, and the network configuration storage cluster composed of Etcd provides high-availability storage and distributed locking functionality in the interaction of the above methods.

[0123] In practical use, all arbitration modules within all representative nodes compete to acquire the distributed lock and elect a master. The module that acquires the lock becomes the master arbitration module. Similarly, all scheduling modules compete to acquire the distributed lock and elect a master; the module that acquires the lock becomes the master scheduling module. Arbitration / scheduling modules that fail to acquire the lock will continuously retry to become the master arbitration / scheduling module. The distributed lock has a lease time, which the master arbitration / scheduling module periodically updates. If the master arbitration / scheduling module fails and fails to renew the lease within the specified time, the distributed lock will be automatically released, and other modules will compete for the lock to elect a new master arbitration / scheduling module.

[0124] During the initial planning and deployment phase, the network nodes planned for the data center cluster serve as the nodes for running network elements. Network node information is stored in the network configuration storage cluster. Adding or deleting network nodes requires updating this information in the network configuration storage cluster. Network element information, network element scheduling and distribution information, and network element operating configurations are stored in the network configuration storage cluster.

[0125] The main arbitration module periodically sends network node probe requests to all arbitration modules. After receiving the request, the arbitration module will probe the network nodes and return the probe results to the main arbitration module.

[0126] The main arbitration module will aggregate the detection results and execute a joint ruling. If more than half of the arbitration modules detect that a network node is unavailable, the network node will be determined to be unavailable and the unavailable status will be updated to the network configuration storage cluster.

[0127] In other words, the main arbitration module in this embodiment will periodically organize joint arbitration. The specific process includes: the main arbitration module obtaining network node list information from the network configuration storage cluster and initiating network node probe requests to all arbitration modules. After receiving the request, the arbitration module will send probe packets to the network nodes specified in the request to check the liveness of the network nodes. If no reply packet is received from the network node within a specified time, the network node is considered unavailable; if a reply packet is received, the network node is considered available, and then the arbitration module returns the probe results to the main arbitration module.

[0128] The main arbitration module executes a joint arbitration decision based on the aggregated detection results. If more than half of the arbitration modules detect that a network node is unavailable, the network node is deemed unavailable, and its status is updated in the network configuration storage cluster. If more than half of the arbitration modules detect that a network node is available, the network node is deemed available, and its status is updated in the network configuration storage cluster. If the number of aggregated detection results is less than half the number of arbitration modules, the joint arbitration is invalid.

[0129] By combining the detection results of multiple arbitration modules distributed across various availability zones for joint arbitration, the false alarm rate is lower and the reliability is higher; synchronous detection ensures that the arbitration result is output directly within one arbitration cycle, resulting in fast detection, high real-time performance, and faster fault scheduling response.

[0130] After the main scheduling module detects that a network node in the network configuration storage cluster is unavailable, it will schedule the network elements running on it to other healthy network nodes, thus completing the fault scheduling process.

[0131] The main scheduling module performs fault scheduling. The process is as follows: when the main scheduling module detects changes in network node information in the network configuration storage cluster, if it finds that a network node is unavailable, it will query the network element scheduling distribution information in the network configuration storage cluster, obtain the network elements associated with the network node, obtain the network element running configuration, select suitable available network nodes, schedule these network element instances to the specified network nodes, update the network element scheduling distribution information to the network configuration storage cluster, and complete the fault scheduling process.

[0132] The main scheduling module will dynamically adjust the network element scheduling priority of network nodes based on the number of network node failures and their duration. Network nodes with frequent failures will be isolated and prohibited from network element scheduling.

[0133] The main scheduling module periodically assigns scheduling priorities to network nodes. Network nodes with higher scheduling priorities have a greater probability of being selected during network element scheduling. If a network node fails, its scheduling priority will be lowered; if no failure occurs within a threshold time, its scheduling priority will be raised. Network nodes with low scheduling priorities, even if they are available, will be excluded from scheduling. Network node scheduling priority information is stored in the network configuration storage cluster.

[0134] Specifically, the scheduling priority information of network nodes is stored in the network configuration storage cluster, and this information includes:

[0135] Network node ID, current scheduling priority, number of failures, duration of failure, and time of last failure.

[0136] For example, the main scheduling module periodically evaluates network nodes, including:

[0137] Retrieve historical records: Retrieve historical fault records for each network node from the network configuration storage cluster;

[0138] Priority Calculation: Calculate the scheduling priority of each network node based on the number of failures and their duration;

[0139] The calculation and adjustment of scheduling priorities specifically includes:

[0140] Lower the priority:

[0141] The number of failures exceeds the threshold: For example, if a network node experiences 3 failures in the past 24 hours, its scheduling priority is reduced from 10 to 5;

[0142] Fault duration exceeds threshold: For example, if a network node's fault duration exceeds 1 hour, its scheduling priority is reduced from 8 to 4.

[0143] Increase priority:

[0144] Fault-free: If a network node has not experienced a failure in the past 7 days, its scheduling priority is increased from 5 to 10;

[0145] Good recovery: If a network node fails in the previous evaluation period but performs well in subsequent periods, its scheduling priority can be gradually increased from 4 to 8.

[0146] When scheduling network elements, the main scheduling module prioritizes network nodes with high scheduling priority; even if some network nodes are available, they will be excluded from the scheduling scope if their scheduling priority is lower than the set threshold.

[0147] In practice, the main scheduling module can dynamically adjust the evaluation cycle and priority adjustment rules to adapt to different business needs and network conditions. For network nodes that frequently fail, the main scheduling module can isolate them to prevent them from affecting other nodes and services. Simultaneously, an alarm mechanism is set up to promptly notify operations and maintenance personnel to handle any changes in the priority of a network node.

[0148] By employing the methods described above, we can ensure high availability and performance optimization of data center clusters, thereby guaranteeing business continuity and data security.

[0149] In one alternative implementation, for fault-sensitive network elements, the main scheduling module can deploy shadow network elements on different network nodes. These shadow network elements have the same configuration as the original network element but are in a ready state and do not provide external services. Once the original network element fails, the shadow network element is immediately put into use, making fault scheduling faster.

[0150] In other embodiments, if availability zone-level fault tolerance is required, the cluster contains at least three availability zones to ensure that when one availability zone fails completely, the number of modules in the remaining availability zones is still more than half, in order to meet the operational requirements of leader election and joint adjudication.

[0151] Network nodes using the above-mentioned devices can achieve multi-point synchronous detection, joint arbitration, high reliability, and fast scheduling in data center clusters.

[0152] The data center cluster in this embodiment can handle fault scheduling on its own without the intervention of an external platform. Furthermore, this data center cluster has high fault tolerance, tolerating faults from the node level to the availability zone level, and is highly adaptable, suitable for both small-scale clusters and large-scale clusters composed of multiple availability zones.

[0153] Furthermore, the data center cluster features high availability across all modules, eliminating the risk of split-brain scenarios. Modules within the cluster act as primary and backup units for each other, and the Raft protocol within the Etcd storage cluster, backed by the network configuration, further prevents split-brain issues. Moreover, the data center cluster implements intelligent scheduling, dynamically adjusting scheduling priorities based on network node failures to avoid service quality impacts caused by frequent unavailability of unstable nodes.

[0154] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0155] In addition, this embodiment of the invention also provides a data center cluster, which is divided into multiple availability zones, each availability zone including multiple network nodes, and at least one network node including a data center network element fault scheduling device as described in the above embodiment.

[0156] In addition, embodiments of the present invention also provide an electronic device, such as... Figure 6 As shown, it includes a processor 1301, a communication interface 1302, a memory 1303, and a communication bus 1304. The processor 1301, the communication interface 1302, and the memory 1303 communicate with each other through the communication bus 1304.

[0157] Memory 1303 is used to store computer programs;

[0158] When processor 1301 executes a program stored in memory 1303, it performs the following steps:

[0159] The main arbitration module sends a network node probe request to all arbitration modules, so that all arbitration modules can probe all network nodes according to the network node probe request; the main arbitration module is determined by all arbitration modules through competition.

[0160] The main arbitration module receives the detection results from all arbitration modules and determines whether there are faulty network nodes based on the joint arbitration mechanism;

[0161] If the faulty network node information exists, the network configuration storage cluster is updated based on the faulty network node information, so that the main scheduling module can schedule the network elements running on the faulty network node to the non-faulty network node in the data center cluster based on the updated faulty network node information in the network configuration storage cluster.

[0162] or,

[0163] If the main scheduling module detects a faulty network node in the network configuration storage cluster, it will send a request to the network configuration storage cluster to obtain the configuration information of the faulty network node.

[0164] The main scheduling module receives the network element configuration information of the faulty network node returned by the network configuration storage cluster based on the configuration information.

[0165] The main scheduling module schedules network elements running on faulty network nodes to non-faulty network nodes based on network element configuration information and information of non-faulty network nodes in the data center cluster.

[0166] The faulty network node is an unavailable network node detected by the interaction between the main arbitration module and all arbitration modules, and is updated in the network configuration storage cluster.

[0167] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0168] The communication interface is used for communication between the aforementioned terminal and other devices.

[0169] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0170] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0171] like Figure 7 As shown, in another embodiment of the present invention, a computer-readable storage medium 1401 is also provided, which stores instructions that, when executed on a computer, cause the computer to perform a data center network element fault scheduling method as described in the above embodiment.

[0172] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute a data center network element fault scheduling method described in the above embodiments.

[0173] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0174] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0175] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0176] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A data center network element fault scheduling method, characterized in that, The data center cluster comprises one or more availability zones. Each availability zone has pre-selected representative network nodes. All representative network nodes in all availability zones are equipped with an arbitration module, a scheduling module, and a network configuration storage cluster. The method includes: The main arbitration module sends a network node probe request to all arbitration modules, so that all arbitration modules can probe all network nodes according to the network node probe request; the main arbitration module is determined by all arbitration modules through competition. The main arbitration module receives the detection results from all arbitration modules and determines whether there are faulty network nodes based on the joint arbitration mechanism; If the faulty network node information exists, the network configuration storage cluster is updated based on the faulty network node information, so that the main scheduling module can schedule the network elements running on the faulty network node to the non-faulty network node in the data center cluster based on the updated faulty network node information in the network configuration storage cluster.

2. The method according to claim 1, characterized in that, Before the main arbitration module sends a network node probe request to all arbitration modules, the method further includes: All arbitration modules within the network nodes elect a master arbitration module using a distributed lock mechanism. And / or, All scheduling modules within the network nodes elect a master scheduling module through a distributed lock mechanism.

3. The method according to claim 1, characterized in that, The main arbitration module sends network node probe requests to all arbitration modules, including: The main arbitration module sends a network element information list retrieval request to the network configuration storage cluster and receives the list information returned by the network configuration storage cluster based on the network element information list retrieval request; The main arbitration module sends a network node probe request to all arbitration modules based on the list information. The network node probe request includes information for probing each network node in the list information.

4. The method according to claim 1, characterized in that, The main arbitration module receives the probe results from all arbitration modules and determines whether there are faulty network nodes based on the joint arbitration mechanism, including: The main arbitration module summarizes the response message information of each network node in the detection results. The response message is the message in which the network node responds to the detection message sent by the arbitration module within a specified time. Determine whether a majority of the arbitration modules have detected a network node as unavailable.

5. The method according to claim 1, characterized in that, The network configuration storage cluster is a distributed key-value database, Etcd.

6. A data center network element fault scheduling method, characterized in that, The data center cluster comprises one or more availability zones. Each availability zone has pre-selected representative network nodes. All representative network nodes in all availability zones are equipped with an arbitration module, a scheduling module, and a network configuration storage cluster. The method includes: If the main scheduling module detects a faulty network node in the network configuration storage cluster, it will send a request to the network configuration storage cluster to obtain the configuration information of the faulty network node. The main scheduling module receives the network element configuration information of the faulty network node returned by the network configuration storage cluster based on the configuration information. The main scheduling module schedules network elements running on faulty network nodes to non-faulty network nodes based on network element configuration information and information of non-faulty network nodes in the data center cluster. The faulty network node is an unavailable network node detected by the interaction between the main arbitration module and all arbitration modules, and is updated in the network configuration storage cluster.

7. The method according to claim 6, characterized in that, The main scheduling module, based on network element configuration information and information about non-faulty network nodes in the data center cluster, schedules network elements running on faulty network nodes to non-faulty network nodes, including: The main scheduling module schedules network elements running on faulty network nodes to higher-priority non-faulty network nodes based on network element configuration information and the priority of non-faulty network nodes.

8. The method according to claim 6, characterized in that, The method further includes: The main scheduling module adjusts the priority of network nodes and updates the network configuration storage cluster based on the frequency and duration of network node failures.

9. The method according to claim 6, characterized in that, If a shadow network element exists on a non-faulty network node, the main scheduling module, based on the network element configuration information and the information of the non-faulty network nodes in the data center cluster, schedules the network elements running on the faulty network node to the non-faulty network node, including: The main scheduling module schedules network elements running on faulty network nodes to shadow network elements on non-faulty network nodes based on network element configuration information. The shadow network element has the same network element configuration as the faulty network node and is in a ready but not providing service state.

10. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; The memory is used to store computer programs; When the processor executes a program stored in the memory, it implements the method as described in any one of claims 1-9.

Images