Data center network element fault scheduling method and electronic equipment

Abstract

The embodiment of the invention provides a data center network element fault scheduling method and electronic equipment, a data center cluster comprises available areas, representative network nodes are pre-selected in each available area, and arbitration modules, scheduling modules and network configuration storage clusters are deployed in the representative network nodes of all the available areas. The method comprises the following steps: a main arbitration module sends a network node detection request to all arbitration modules, so that the arbitration modules detect all network nodes according to the network node detection request; receiving detection results fed back by all the arbitration modules, and determining whether a fault network node exists or not according to a joint arbitration mechanism; and if the fault network node information exists, updating a network configuration storage cluster according to the fault network node information, so that a main scheduling module schedules a network element running on the fault network node to a non-fault network node in a data center cluster according to the fault network node information in the updated network configuration storage cluster. The method is rapid in detection, high in real-time performance and faster in fault scheduling response.

Description

Technical Field

[0001] The present invention relates to the technical field of data exchange networks, and in particular to a data center network element fault scheduling method and an electronic device. Background Art

[0002] Network elements are important bridges for north-south communication between user VPC (Virtual Private Cloud) and the outside world. Network elements are generally planned to run on specific network nodes. Once a network node fails, the network element will not work properly and the data center will not be able to provide services to the outside world. When a network node fails or the network is isolated by partition, the problematic network node should be isolated immediately, and the network elements running on it should be dispatched to other normal network nodes to continue to provide network services.

[0003] Traditional network devices will run BFD (Bidirectional Forwarding Detection) detection to quickly detect communication faults between devices so that timely measures can be taken, such as switching routes to backup devices to ensure continuous business operation. BFD is a point-to-point fault detection method, such as network devices establishing BFD with upstream and downstream devices respectively. Data centers generally run virtualized networks such as OVS (Open vSwitch). In this case, the upstream device of the network element is a physical switch, which can run BFD, but the downstream device of the network element is a virtualized network device. The next hop for the network element to forward downstream traffic is the virtual switch gateway, but the gateway is distributed and cannot establish BFD like a physical device. One method is to fix the virtual switch gateway to a certain host node, but this will introduce new single points of failure and traffic forwarding bottlenecks, which is really undesirable.

[0004] In essence, the physical carriers of the downstream virtual devices of the network element are all computing nodes, so all computing nodes can be required to run a BFD-like detection with the network element. This will lead to a large amount of unnecessary detection traffic when there are many network elements. Moreover, each computing node will have its own decision result, and management often hopes to clearly identify which network element is providing the service (for example, understanding the load situation of each network node), and the control capability will be reduced.

[0005] The existing solutions are: network nodes run VRRP (Virtual Router Redundancy Protocol), the node that becomes the leader in the VRRP election will take over all network elements, and other nodes will become backup nodes and will not provide services to the outside world. The problem with this solution is that VRRP is a master-slave working model, and only one network node is working at the same time, which results in a large waste of resources; furthermore, VRRP generally requires that the nodes participating in the election are located in the same Layer 2 network, and multi-availability zone data centers often run in multiple computer rooms, making it difficult to provide a Layer 2 network environment for network nodes distributed in multiple computer rooms. Therefore, VRRP is not suitable for scenarios with multiple availability zones and large-scale clusters, but is more suitable for scenarios with small scale and single availability zones.

[0006] The existing solutions include: using a fixed monitoring service outside or inside the cluster to detect the availability of network nodes and then dispatching network elements. The problem with this solution is that it relies heavily on the monitoring service for the physical link status between network nodes, which often leads to misjudgment and jitter of network element services; the availability of the monitoring service and scheduling service itself is not guaranteed. At the same time, if these services are deployed outside the cluster, detection will become unreliable due to long links; if they are deployed in a certain availability zone of the cluster, once the availability zone fails, the service cannot be provided.

[0007] In summary, we need a reliable, cluster-autonomous network element fault scheduling method that is suitable for large-scale, multi-availability zone data centers. Summary of the invention

[0008] The embodiments of the present invention provide a data center network element fault scheduling method, device, electronic device and computer-readable storage medium, which are suitable for autonomous network element fault scheduling of large-scale, multi-availability zone data center clusters.

[0009] The embodiment of the present invention discloses a method for scheduling network element failures in a data center. The data center cluster includes more than one availability zone, and a representative network node is pre-selected in each availability zone. An arbitration module, a scheduling module, and a network configuration storage cluster are deployed in the representative network nodes of all availability zones. The method includes:

[0010] The main arbitration module sends a network node detection request to all arbitration modules, so that all arbitration modules detect all network nodes according to the network node detection request; the main arbitration module is the main arbitration module determined by all arbitration modules through competition;

[0011] The main arbitration module receives the detection results fed back by all arbitration modules and determines whether there is a faulty network node according to the joint arbitration mechanism;

[0012] If so, the network configuration storage cluster is updated according to the faulty network node information, so that the main scheduling module schedules the network elements running on the faulty network node to the non-faulty network nodes in the data center cluster according to the updated faulty network node information in the network configuration storage cluster.

[0013] Optionally, before the main arbitration module sends the network node detection request to all arbitration modules, the method further includes:

[0014] All arbitration modules in the representative network nodes elect the main arbitration module through distributed locking;

[0015] and / or,

[0016] All scheduling modules in the representative network nodes elect the main scheduling module through distributed locking.

[0017] Optionally, the main arbitration module sends a network node detection request to all arbitration modules, including:

[0018] The main arbitration module sends a network element information list acquisition request to the network configuration storage cluster, and receives the list information returned by the network configuration storage cluster according to the network element information list acquisition request;

[0019] The main arbitration module sends a network node detection request to all arbitration modules according to the list information, and the network node detection request includes: information for detecting each network node in the list information.

[0020] Optionally, the main arbitration module receives the detection results fed back by all arbitration modules and determines whether there is a faulty network node according to the joint arbitration mechanism, including:

[0021] The main arbitration module summarizes the reply message information of each network node in the detection result, and the reply message is a message that the network node replies to the detection message sent by the arbitration module within a specified time;

[0022] Determine whether a majority arbitration module detects that a certain network node is unavailable due to a failure.

[0023] Optionally, the network configuration storage cluster is a distributed key-value database Etcd.

[0024] The embodiment of the present invention also discloses a data center network element fault scheduling method, wherein the data center cluster includes more than one availability zone, a representative network node is pre-selected in each availability zone, and an arbitration module, a scheduling module and a network configuration storage cluster are deployed in the representative network nodes of all availability zones, and the method includes:

[0025] When the main scheduling module detects that there is a faulty network node in the network configuration storage cluster, it sends a configuration information acquisition request of the faulty network node to the network configuration storage cluster;

[0026] The main scheduling module receives the network element configuration information of the faulty network node returned by the network configuration storage cluster according to the configuration information acquisition request;

[0027] The main scheduling module schedules the network elements running on the faulty network nodes to the non-faulty network nodes according to the network element configuration information and the information of the non-faulty network nodes in the data center cluster;

[0028] The faulty network node is an unavailable network node detected by the interaction between the main arbitration module and all arbitration modules, and is updated in the network configuration storage cluster.

[0029] Optionally, the main scheduling module schedules the network elements running on the faulty network nodes to the non-faulty network nodes according to the network element configuration information and the information of the non-faulty network nodes in the data center cluster, including:

[0030] The main scheduling module schedules the network elements running on the faulty network nodes to the non-faulty network nodes with high priorities according to the network element configuration information and the priorities of the non-faulty network nodes.

[0031] Optionally, the method further comprises:

[0032] The main scheduling module adjusts the priority of the network node according to the frequency and duration of network node failures and updates it to the network configuration storage cluster.

[0033] Optionally, if a shadow network element exists in the non-faulty network node, the main scheduling module schedules the network element running on the faulty network node to the non-faulty network node according to the network element configuration information and the information of the non-faulty network nodes in the data center cluster, including:

[0034] The main scheduling module schedules the network elements running on the faulty network node to the shadow network elements of the non-faulty network node according to the network element configuration information;

[0035] The shadow network element has the same configuration as the network element of the failed network node and is in a ready but non-service state.

[0036] The embodiment of the present invention further discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0037] The memory is used to store computer programs;

[0038] The processor is used to implement the method described in the embodiment of the present invention when executing the program stored in the memory.

[0039] The embodiment of the present invention further discloses one or more computer-readable media on which instructions are stored. When executed by one or more processors, the processors are enabled to execute the method described in the embodiment of the present invention.

[0040] The embodiments of the present invention include the following advantages:

[0041] The method of the embodiment of the present invention interacts with the network configuration storage cluster through the main arbitration module to determine the unavailable network node, so that the main scheduling module can schedule the network elements running on the unavailable network node to other normally available network nodes to continue to provide network services, which realizes reliable, autonomous, and fast network element fault scheduling. In particular, during the detection process, the full link module is highly available without brain split.

[0042] The method of the embodiment of the present invention interacts with the network configuration storage cluster through the main scheduling module, and schedules the network elements running on the unavailable network nodes to other normally available network nodes to continue to provide network services. It realizes the cluster autonomy process without the need for external node intervention and rapid network element fault scheduling.

[0043] Furthermore, in the embodiments of the present invention, the scheduling priority of the network node can be dynamically adjusted according to the failure status of the network node, thereby performing dynamic scheduling. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a schematic diagram of an available zone in a data center cluster provided in an embodiment of the present invention;

[0045] Figure 2 It is a flowchart of the steps of a data center network element fault scheduling method provided in an embodiment of the present invention;

[0046] Figure 3 It is a flowchart of the steps of a data center network element fault scheduling method provided in an embodiment of the present invention;

[0047] Figure 4 It is a signaling diagram of a data center network element fault scheduling method provided in an embodiment of the present invention;

[0048] Figure 5 It is a structural block diagram of a data center network element fault scheduling device provided in an embodiment of the present invention;

[0049] Figure 6 is a block diagram of an electronic device provided in an embodiment of the present invention;

[0050] Figure 7is a schematic diagram of a computer-readable medium provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0052] Reference Figure 1 , shows an architecture diagram of a data center cluster provided in an embodiment of the present invention. The data center cluster in this embodiment includes multiple network nodes, which may be physical servers. Multiple network elements are configured in each network node, that is, multiple network elements run on the network node. In this embodiment, the data center cluster may be pre-divided into multiple availability zones. For example, the data center cluster may include more than one availability zone, and a representative network node is pre-selected in each availability zone. The arbitration module, the scheduling module, and the network configuration storage cluster are deployed in the representative network nodes of all availability zones.

[0053] In this embodiment, by dividing the data center cluster into multiple availability zones, even if a failure occurs in one availability zone, other availability zones can still maintain normal operation, thereby ensuring business continuity.

[0054] At the same time, load balancing technology can be used to share access pressure between different availability zones to optimize resource utilization efficiency. When the load of a certain availability zone is too high, part of the traffic can be automatically or manually directed to other availability zones to avoid single point overload.

[0055] In this embodiment, the number of network nodes in each available zone may be greater than one, and the available zone selects network nodes as representative network nodes in proportion to the scale of the network nodes. For example, one or two nodes may be selected, which is not limited in this embodiment and is selected according to actual needs. Figure 1 It is shown that two representative network nodes (referred to as representative nodes) are selected for each available zone.

[0056] In the specific implementation process, servers with strong computing power are given priority as representative network nodes to ensure that they can efficiently handle various tasks, especially tasks that require high computing resources. At the same time, considering that network nodes require multiple key components (arbitration module, scheduling module, and network configuration storage cluster), selecting servers with sufficient memory can reduce the risk of service termination due to insufficient memory. In addition, representative network nodes often need to communicate frequently with other nodes, so network bandwidth is an important consideration. Selecting servers with high network bandwidth can increase data transmission speed and reduce latency.

[0057] In this embodiment, an arbitration module, a scheduling module, and a network configuration storage cluster are deployed in each representative node. It is understandable that in this embodiment, after a representative node fails, other representative nodes can be manually switched, and the arbitration module, scheduling module, and network configuration storage cluster deployed in each representative node can be directly deployed and implemented during cluster planning.

[0058] Each availability zone selects representative nodes to centrally deploy key components (arbitration module, scheduling module, and network configuration storage cluster), which helps simplify management and maintenance. These components are critical to the stable operation of the entire data center, so their high availability and rapid response capabilities are particularly important.

[0059] If a representative node fails, the system has the ability to automatically detect and quickly switch to a backup representative node to reduce service interruptions. In addition, regular fault recovery drills should be conducted to ensure stable operation in the event of a real failure, so high availability and rapid response capabilities are particularly important.

[0060] Reference Figure 2 , shows a flow chart of the steps of a data center network element fault scheduling method provided in an embodiment of the present invention, which may specifically include the following steps:

[0061] 201. The main arbitration module sends a network node detection request to all arbitration modules, so that all arbitration modules detect all network nodes according to the network node detection request.

[0062] The main arbitration module in this embodiment is a main arbitration module determined by all arbitration modules through competition.

[0063] For example, all arbitration modules representing network nodes elect a master arbitration module through a distributed lock method.

[0064] In this embodiment, common distributed lock implementation methods include Zookeeper and Etcd. These tools provide a strong consistency distributed coordination service, which can ensure that the main arbitration module is elected fairly and efficiently among multiple nodes.

[0065] Each arbitration module periodically sends heartbeats to the distributed lock service to maintain the validity of the lock. If an arbitration module loses its heartbeat, the lock will be released and other arbitration modules can try to acquire the lock.

[0066] At the same time, set a reasonable timeout period to ensure that the lock can be released in time when a certain arbitration module is unresponsive for a long time to prevent deadlock.

[0067] Step 201 includes: the main arbitration module sends a network element information list acquisition request to the network configuration storage cluster, and receives list information returned by the network configuration storage cluster according to the network element information list acquisition request;

[0068] The main arbitration module sends a network node detection request to all arbitration modules according to the list information, and the network node detection request includes: information for detecting each network node in the list information.

[0069] In this embodiment, the network configuration storage cluster stores configuration information of network elements running on all network nodes, basic information of network nodes, network element distribution information, etc.

[0070] Basic information of network element nodes: such as IP address, MAC address, host name, operating system version, etc.

[0071] NE configuration information: configuration parameters of NEs running on each network node, such as port number, service type, resource limit, etc.

[0072] Network distribution information: which network elements run on which network nodes, and their distribution;

[0073] The network configuration storage cluster also stores health status information, including the current health status of network nodes and network elements, such as CPU utilization, memory usage, network bandwidth, etc.

[0074] In the specific implementation process, the main arbitration module sends a network element information list acquisition request to the network configuration storage cluster. This request usually contains some necessary parameters, such as the requester identity, request timestamp, etc., to ensure the legitimacy and timeliness of the request.

[0075] The network configuration storage cluster returns the generated network element information list to the main arbitration module. The list information may include:

[0076] Network node ID: an ID that uniquely identifies each network node;

[0077] NE ID: uniquely identifies each NE ID;

[0078] Network element type: the type of network element, such as Web service, database service, etc.

[0079] NE configuration: specific configuration parameters of NE;

[0080] Network node status: the current status of the network node, such as online, offline, maintenance, etc.

[0081] Furthermore, after receiving the network element information list, the main arbitration module parses the information therein and extracts detailed information of each network node, including IP address, port number and the like.

[0082] The main arbitration module constructs a network node detection request based on the parsed network node information. Each detection request contains the following information:

[0083] Target network node ID, detection parameters and detection instructions;

[0084] The main arbitration module sends the constructed detection request to all arbitration modules. Each arbitration module will detect the specified network node according to the received request.

[0085] 202. The main arbitration module receives the detection results fed back by all arbitration modules, and determines whether there is a faulty network node according to the joint arbitration mechanism; if so, execute the following step 203; otherwise, continue to execute the above step 201 according to the detection cycle.

[0086] For example, the main arbitration module summarizes the reply message information of each network node in the detection results, where the reply message is the message that the network node replies to the detection message sent by the arbitration module within a specified time; determines whether more than half of the arbitration modules detect that a certain network node has an unavailable fault, and if so, determines that there is a faulty network node and obtains the information of the faulty network node.

[0087] Specifically, if more than half of the arbitration modules report that a network node is unavailable, the network node is considered to be faulty. The detection result of each arbitration module is equivalent to a vote. If more than half of the votes are "unavailable", the network node is considered to be a faulty node.

[0088] In some cases, different weights can be assigned to different arbitration modules to reflect their reliability or importance, and the final judgment result can be based on weighted voting.

[0089] 203. If so, update the network configuration storage cluster according to the faulty network node information, so that the main scheduling module schedules the network elements running on the faulty network node to the non-faulty network nodes in the data center cluster according to the updated faulty network node information in the network configuration storage cluster.

[0090] In addition, in this embodiment, all scheduling modules in the representative network nodes elect a main scheduling module through a distributed lock method.

[0091] Furthermore, the main scheduling module selects a suitable target node to schedule the faulty network element according to the current network node status and load conditions;

[0092] The load situation needs to be balanced, that is, nodes with lower loads are selected to avoid scheduling multiple faulty network elements to the same node, causing the node to be overloaded. In addition, it is also necessary to ensure that the resource configuration of the target node (such as CPU, memory, and network bandwidth) can meet the needs of the faulty network element.

[0093] The network configuration storage cluster in this embodiment can be a distributed key-value database Etcd, which can ensure data consistency and high availability; therefore, the above-mentioned main arbitration module and main scheduling module can be distributed in different representative nodes or located in the same representative node.

[0094] The method of this embodiment interacts with the network configuration storage cluster through the main arbitration module to quickly identify the faulty network nodes in the data center cluster. In the implementation process, multiple network nodes are detected synchronously and jointly arbitrated, with high reliability. In the above method, no external node intervention is required, and the full-link module is highly available during the detection process without brain split.

[0095] Reference Figure 3 , shows a flow chart of the steps of a data center network element fault scheduling method provided in an embodiment of the present invention, which may specifically include the following steps:

[0096] 301. When the main scheduling module detects that there is a faulty network node in the network configuration storage cluster, it sends a configuration information acquisition request of the faulty network node to the network configuration storage cluster;

[0097] 302. The main scheduling module receives the network element configuration information of the faulty network node returned by the network configuration storage cluster according to the configuration information acquisition request;

[0098] 303. The main scheduling module schedules the network elements running on the faulty network nodes to the non-faulty network nodes according to the network element configuration information and the information of the non-faulty network nodes in the data center cluster;

[0099] The faulty network node is an unavailable network node detected by the interaction between the main arbitration module and all arbitration modules, and is updated in the network configuration storage cluster.

[0100] The main scheduling module in this embodiment can adjust the priority of the network node according to the frequency and duration of the failure of the network node and update it to the network configuration storage cluster.

[0101] Furthermore, in the above step 303, the main scheduling module may schedule the network elements running on the faulty network node to the non-faulty network node with a higher priority according to the network element configuration information and the priority of the non-faulty network node.

[0102] In an optional implementation, if a shadow network element exists in a non-faulty network node, the above step 303 may include: the main scheduling module schedules the network element running on the faulty network node to the shadow network element of the non-faulty network node according to the network element configuration information.

[0103] The shadow network element in this embodiment can have the same configuration as the network element of the failed network node and is in a ready but unserviced state. In this embodiment, not every network node in the data center cluster has a shadow network element, which is deployed according to the planning of the data center cluster.

[0104] The method of the embodiment of the present invention interacts with the network configuration storage cluster through the main scheduling module. After the main arbitration module determines that the network node is faulty, the main scheduling module can schedule the network elements running on the faulty network node to other normal network nodes to continue to provide network services, which realizes reliable, autonomous, and fast network element fault scheduling. The above method is fast in scheduling and does not require external node intervention, thus realizing the autonomy of the data center cluster.

[0105] The main scheduling module of this embodiment can dynamically adjust the priorities of network nodes, and further can implement dynamic adjustment according to the priorities of normal network nodes in the scheduling of faulty network nodes, thus achieving high reliability.

[0106] Reference Figure 4 , shows a flow chart of the steps of a data center network element fault scheduling method provided in an embodiment of the present invention, which may specifically include the following steps:

[0107] 401. The main arbitration module sends a request for obtaining a network element information list to the network configuration storage cluster;

[0108] 402. The network configuration storage cluster returns list information according to the network element information list acquisition request;

[0109] 403. The main arbitration module sends a network node detection request to all arbitration modules;

[0110] 404. All arbitration modules send a detection message to each network node in the data center cluster according to the detection network node request;

[0111] 405. Each network node returns a response according to the received detection message;

[0112] 406. Each arbitration module summarizes the responses received from the network nodes and sends its own detection results to the main arbitration module;

[0113] 407. The main arbitration module integrates the detection results of all arbitration modules and the availability of arbitration network nodes to determine that there is an unavailable network node as a faulty network node or to determine the availability of the network node.

[0114] 408. The main arbitration module updates the information of the unavailable network nodes or the availability of the network nodes to the network configuration storage cluster.

[0115] 409. The main scheduling module monitors the availability of nodes in the network configuration storage cluster, changes the faulty network node to unavailable, and obtains the network element distribution information, basic network node information, and network element related configurations running on the faulty network node from the network configuration storage cluster;

[0116] 410. The main scheduling module obtains the network element distribution information, basic information of the network node, and related configuration of the network element running on the faulty network node returned by the network configuration storage cluster;

[0117] 411. The main scheduling module selects a suitable normally operating network node for the network element on the faulty network node.

[0118] 412. The main scheduling module schedules the network elements to normal network nodes, and updates information such as network element distribution and network node scheduling priority to the network configuration storage cluster.

[0119] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0120] Reference Figure 5 , showing a structural block diagram of a data center network element fault scheduling device provided in an embodiment of the present invention. The data center network element fault scheduling device of this embodiment is located in a network node, which is set when a data center cluster is deployed, and each data center network element fault scheduling device may specifically include the following modules: an arbitration module, a scheduling module and a network configuration storage cluster.

[0121] When deploying a data center cluster, each availability zone selects network nodes as representative nodes in proportion to their size. Generally speaking, the number of network nodes in the availability zones within the data center cluster is comparable, so that each availability zone is selected with an equal number of representative nodes, thereby allowing each availability zone to have the same arbitration power. The representative node deployment includes the above-mentioned arbitration module, scheduling module, and data center network element fault scheduling device of the network configuration storage cluster.

[0122] The network configuration storage cluster in this embodiment can be formed by deploying Etcd on the representative node. Etcd is a highly available distributed key-value database. The network configuration storage cluster formed by Etcd provides high-availability storage and distributed lock functions in the interaction of the above method.

[0123] In the specific use process, all arbitration modules in all representative nodes compete to obtain the distributed lock to elect the master, and the module that obtains the lock becomes the main arbitration module; all scheduling modules compete to obtain the distributed lock to elect the master, and the module that obtains the lock becomes the main scheduling module. The arbitration module / scheduling module that has not obtained the lock will continue to retry to obtain the distributed lock to become the main arbitration module / main scheduling module. The distributed lock has a lease time, and the main arbitration module / main scheduling module will periodically update the lease time of the distributed lock. If the main arbitration module / main scheduling module fails and the lease time is not renewed, the distributed lock will be automatically released, and other modules will compete for the distributed lock to elect a new main arbitration module / main scheduling module.

[0124] In the initial planning and deployment phase, the network nodes planned in the data center cluster are used as nodes for running network elements. The network node information is stored in the network configuration storage cluster. The addition and deletion of network nodes need to be updated in the network configuration storage cluster. Network element information, network element scheduling distribution information, and network element operation configuration are stored in the network configuration storage cluster.

[0125] The main arbitration module will periodically send network node detection requests to all arbitration modules. After receiving the request, the arbitration module will detect the network nodes and return the detection results to the main arbitration module.

[0126] The main arbitration module will summarize the detection results and perform joint arbitration. If more than half of the arbitration modules detect that a network node is unavailable, the network node will be determined to be unavailable and the unavailable status will be updated to the network configuration storage cluster;

[0127] That is to say, the main arbitration module in this embodiment will periodically organize joint arbitration. The specific process includes: the main arbitration module obtains the network node list information from the network configuration storage cluster and initiates a network node detection request to all arbitration modules. After receiving the request, the arbitration module will send a detection message to the network node specified in the request to detect the survival of the network node. If no reply message is received from the network node within the specified time, the network node is considered unavailable; if a reply message is received, the network node is considered available, and then the arbitration module returns the detection result to the main arbitration module.

[0128] The main arbitration module aggregates the detection results and performs joint arbitration. If more than half of the arbitration modules detect that a network node is unavailable, the network node is determined to be unavailable and the network node status is updated to the network configuration storage cluster; if more than half of the arbitration modules detect that a network node is available, the network node is determined to be available and the network node status is updated to the network configuration storage cluster. If the number of aggregated detection results is less than half of the number of arbitration modules, the joint arbitration is invalid.

[0129] The detection results of multiple arbitration modules distributed in various availability zones are jointly arbitrated, which has a lower false alarm rate and higher reliability. With synchronous detection, the arbitration results are directly output within one arbitration cycle, with fast detection, high real-time performance, and faster fault scheduling response.

[0130] After the main scheduling module detects that a network node in the network configuration storage cluster is unavailable, it will schedule the network elements running on it to other healthy network nodes to complete the fault scheduling process;

[0131] The main scheduling module will perform fault scheduling, and the process is as follows: the main scheduling module monitors changes in network node information in the network configuration storage cluster. If a network node is found to be unavailable, it will query the network element scheduling distribution information in the network configuration storage cluster, obtain the network element associated with the network node, obtain the network element operation configuration, select appropriate available network nodes, and then schedule these network element instances to the specified network node, update the network element scheduling distribution information to the network configuration storage cluster, and complete the fault scheduling process.

[0132] The main scheduling module will dynamically adjust the network element scheduling priority of the network node according to the number of network node failures and duration. Network nodes with frequent failures will be isolated and network element scheduling will be prohibited;

[0133] The main scheduling module periodically assigns scheduling priorities to network nodes. Network nodes with high scheduling priorities have a greater probability of being selected during network element scheduling. If a network node fails, the scheduling priority will be lowered. If no failure occurs within the threshold time, the scheduling priority will be raised. Network nodes with low scheduling priorities will be excluded from scheduling even if they are available. Network node scheduling priority information is stored in the network configuration storage cluster.

[0134] Specifically, the scheduling priority information of the network nodes is stored in the network configuration storage cluster, which includes:

[0135] Network node ID, current scheduling priority, number of failures, duration of failures, and last failure time.

[0136] For example, the main scheduling module periodically evaluates network nodes including:

[0137] Get historical records: Get historical fault records of each network node from the network configuration storage cluster;

[0138] Calculate and adjust priority: Calculate the scheduling priority of each network node based on the number of failures and duration;

[0139] The calculation and adjustment scheduling priority specifically includes:

[0140] Lower priority:

[0141] The number of failures exceeds the threshold: For example, if a network node has experienced three failures in the past 24 hours, its scheduling priority is reduced from 10 to 5;

[0142] Fault duration exceeds the threshold: For example, if the fault duration of a network node exceeds 1 hour, its scheduling priority is reduced from 8 to 4.

[0143] Increase priority:

[0144] No failure: If a network node has not experienced any failure in the past 7 days, its scheduling priority is increased from 5 to 10;

[0145] Good recovery: If a network node fails in the previous evaluation cycle but performs well in subsequent cycles, its scheduling priority can be gradually increased from 4 to 8.

[0146] When scheduling network elements, the main scheduling module gives priority to network nodes with high scheduling priorities; even if some network nodes are available, they will be excluded from the scheduling range if their scheduling priorities are lower than the set threshold.

[0147] In the specific implementation process, the main scheduling module can dynamically adjust the evaluation cycle and priority adjustment rules to adapt to different business needs and network conditions. For network nodes that frequently fail, the main scheduling module can isolate them to prevent them from affecting other nodes and businesses. At the same time, an alarm mechanism is set up to promptly notify the operation and maintenance personnel to handle when the priority of a network node changes.

[0148] Through the above method, the high availability and performance optimization of the data center cluster can be ensured, and business continuity and data security can be guaranteed.

[0149] In an optional implementation, for fault-sensitive network elements, the main scheduling module can deploy shadow network elements for network elements in different network nodes. The shadow network elements have the same configuration as the network elements, but are in a ready state and do not provide services to the outside world. Once the original network element fails, the shadow network element is immediately put into use, making fault scheduling faster.

[0150] In other embodiments, if zone-level fault tolerance is required, the cluster includes at least three zones to ensure that when one zone fails completely, the number of modules in the remaining zones still exceeds half to meet the operational requirements of leader election and joint arbitration.

[0151] The network nodes using the above device can realize multi-point synchronous detection, joint arbitration, high reliability and fast scheduling in the data center cluster.

[0152] The data center cluster of this embodiment can complete fault scheduling by itself without the intervention of the external platform of the cluster. In addition, the data center cluster has a high fault tolerance, can tolerate node-level to availability zone-level faults, and has strong adaptability, which can adapt to both small-scale clusters and large-scale clusters consisting of multiple availability zones.

[0153] Furthermore, the full-link module of the data center cluster is highly available, without brain splits. The modules between network nodes in the cluster are mutually active and standby, and the Raft protocol inside the Etcd of the back-end network configuration storage cluster is used to avoid brain splits. In addition, the data center cluster implements intelligent scheduling, dynamically adjusts scheduling priorities according to network node failures, and avoids frequent unavailability of unstable nodes, which affects service quality.

[0154] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0155] In addition, an embodiment of the present invention further provides a data center cluster, which is divided into multiple availability zones, each availability zone includes multiple network nodes, and at least one network node includes a data center network element fault scheduling device described in the above embodiment.

[0156] In addition, an embodiment of the present invention further provides an electronic device, such as Figure 6 As shown, it includes a processor 1301, a communication interface 1302, a memory 1303 and a communication bus 1304, wherein the processor 1301, the communication interface 1302, and the memory 1303 communicate with each other through the communication bus 1304.

[0157] Memory 1303, used for storing computer programs;

[0158] The processor 1301 is used to implement the following steps when executing the program stored in the memory 1303:

[0159] The main arbitration module sends a network node detection request to all arbitration modules, so that all arbitration modules detect all network nodes according to the network node detection request; the main arbitration module is the main arbitration module determined by all arbitration modules through competition;

[0160] The main arbitration module receives the detection results fed back by all arbitration modules and determines whether there is a faulty network node according to the joint arbitration mechanism;

[0161] If so, the network configuration storage cluster is updated according to the faulty network node information, so that the main scheduling module schedules the network elements running on the faulty network node to the non-faulty network nodes in the data center cluster according to the updated faulty network node information in the network configuration storage cluster.

[0162] or,

[0163] When the main scheduling module detects that there is a faulty network node in the network configuration storage cluster, it sends a configuration information acquisition request of the faulty network node to the network configuration storage cluster;

[0164] The main scheduling module receives the network element configuration information of the faulty network node returned by the network configuration storage cluster according to the configuration information acquisition request;

[0165] The main scheduling module schedules the network elements running on the faulty network nodes to the non-faulty network nodes according to the network element configuration information and the information of the non-faulty network nodes in the data center cluster;

[0166] The faulty network node is an unavailable network node detected by the interaction between the main arbitration module and all arbitration modules, and is updated in the network configuration storage cluster.

[0167] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0168] The communication interface is used for communication between the above terminal and other devices.

[0169] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0170] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0171] like Figure 7 As shown, in another embodiment provided by the present invention, a computer-readable storage medium 1401 is also provided, in which instructions are stored. When the computer-readable storage medium 1401 is run on a computer, the computer executes a data center network element fault scheduling method described in the above embodiment.

[0172] In another embodiment provided by the present invention, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute a data center network element fault scheduling method described in the above embodiment.

[0173] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0174] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0175] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0176] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A data center network element fault scheduling method, characterized in that: The data center cluster includes more than one availability zone, a representative network node is pre-selected in each availability zone, and an arbitration module, a scheduling module and a network configuration storage cluster are deployed in the representative network nodes of all availability zones. The method includes: The main arbitration module sends a network node detection request to all arbitration modules, so that all arbitration modules detect all network nodes according to the network node detection request; the main arbitration module is the main arbitration module determined by all arbitration modules through competition; The main arbitration module receives the detection results fed back by all arbitration modules and determines whether there is a faulty network node according to the joint arbitration mechanism; If so, the network configuration storage cluster is updated according to the faulty network node information, so that the main scheduling module schedules the network elements running on the faulty network node to the non-faulty network nodes in the data center cluster according to the updated faulty network node information in the network configuration storage cluster.

2. The method according to claim 1, characterized in that Before the main arbitration module sends the network node detection request to all arbitration modules, the method further includes: All arbitration modules in the representative network nodes elect the main arbitration module through distributed locking; and / or, All scheduling modules in the representative network nodes elect the main scheduling module through distributed locking.

3. The method according to claim 1, characterized in that The main arbitration module sends a network node detection request to all arbitration modules, including: The main arbitration module sends a network element information list acquisition request to the network configuration storage cluster, and receives the list information returned by the network configuration storage cluster according to the network element information list acquisition request; The main arbitration module sends a network node detection request to all arbitration modules according to the list information, and the network node detection request includes: information for detecting each network node in the list information.

4. The method according to claim 1, characterized in that: The main arbitration module receives the detection results fed back by all arbitration modules and determines whether there is a faulty network node based on the joint arbitration mechanism, including: The main arbitration module summarizes the reply message information of each network node in the detection result, and the reply message is a message that the network node replies to the detection message sent by the arbitration module within a specified time; Determine whether a majority arbitration module detects that a certain network node is unavailable due to a failure.

5. The method according to claim 1, characterized in that The network configuration storage cluster is a distributed key-value database Etcd.

6. A data center network element fault scheduling method, characterized in that: The data center cluster includes more than one availability zone, a representative network node is pre-selected in each availability zone, and an arbitration module, a scheduling module and a network configuration storage cluster are deployed in the representative network nodes of all availability zones. The method includes: When the main scheduling module detects that there is a faulty network node in the network configuration storage cluster, it sends a configuration information acquisition request of the faulty network node to the network configuration storage cluster; The main scheduling module receives the network element configuration information of the faulty network node returned by the network configuration storage cluster according to the configuration information acquisition request; The main scheduling module schedules the network elements running on the faulty network nodes to the non-faulty network nodes according to the network element configuration information and the information of the non-faulty network nodes in the data center cluster; The faulty network node is an unavailable network node detected by the interaction between the main arbitration module and all arbitration modules, and is updated in the network configuration storage cluster.

7. The method according to claim 6, characterized in that The main scheduling module schedules the network elements running on the faulty network nodes to the non-faulty network nodes according to the network element configuration information and the information of the non-faulty network nodes in the data center cluster, including: The main scheduling module schedules the network elements running on the faulty network nodes to the non-faulty network nodes with high priorities according to the network element configuration information and the priorities of the non-faulty network nodes.

8. The method according to claim 6, characterized in that The method further comprises: The main scheduling module adjusts the priority of the network node according to the frequency and duration of network node failures and updates it to the network configuration storage cluster.

9. The method according to claim 6, characterized in that If there is a shadow NE on the non-faulty network node, the main scheduling module schedules the NE running on the faulty network node to the non-faulty network node according to the NE configuration information and the information of the non-faulty network nodes in the data center cluster, including: The main scheduling module schedules the network elements running on the faulty network node to the shadow network elements of the non-faulty network node according to the network element configuration information; The shadow network element has the same configuration as the network element of the failed network node and is in a ready but non-service state.

10. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the method according to any one of claims 1 to 9 when executing the program stored in the memory.

Images