Systems and methods for debugging, optimizing, analyzing, or restoring network devices in real-time networks.

Abstract

An exemplary method is disclosed that facilitates the on-demand creation of exemplary instrumented network devices in cloud infrastructure, remote servers, evaluation platforms, or custom test servers, forming a stack between the instrumented network devices (serving as debug network devices) and target network devices. A switching operation is then performed to switch from the control plane of the target network device to the control plane of the debug network device, while the data plane of the target network device continues to operate. Once the switch is complete, the control plane or the instrumentation (e.g., hardware or software) of the debug network device facilitates the debugging, optimization, profiling, and / or recovery of the physical network device, even in live networks.

Description

[0001] Related applications

[0002] This PCT application claims priority and benefit to U.S. Patent Application No. 17 / 332,248, filed May 27, 2021, entitled "System and Method to Debug, Optimize, Profile, or Recover Network Device in Live Network," the entire contents of which are incorporated herein by reference. Technical Field

[0003] Various embodiments of the present invention relate to networking devices, and more specifically, to hardware and software architectures and components that provide debugging network devices and environments during handover operations. Background Technology

[0004] With the ever-increasing number of features, products, SKUs, and configurations / templates, testing and validating the interoperability of every combination becomes impractical. Therefore, problems and vulnerabilities in networks and network devices (within hardware, software, and middleware) are inevitable. More severe issues can lead to network operational disruptions. Network equipment manufacturers often expend significant technical and financial resources to resolve these problems.

[0005] Meanwhile, network equipment manufacturers often design and configure release images optimized for performance and cost to run on production network equipment. Release images may include an operating system for a given network device, as well as instructions (e.g., command lines) for routing protocols and debugging capabilities. However, in production configurations, network devices are configured to generate minimal and often essential profiling / debugging information. While some issues may be anticipated, it is impractical to anticipate and include a large number of debugging features (e.g., counters, debug or trace logs) to debug every potential problem. Doing so can severely impact performance and consume the limited system resources in a given production network device (e.g., a switch). For example, network equipment manufacturers may include debugging elements in the most common code paths. While this approach works well for reverse engineering or performing root cause analysis on most functional and logic-related vulnerabilities / issues, some functional and timing-related issues (e.g., in ASICs, in hardware, in firmware, in hardware-software interactions, etc.) are more difficult to troubleshoot and determine their root causes. Many problems ultimately require troubleshooting in a lab environment with an instrumented image of the equivalent network setup and operation to gather more data for further troubleshooting. In such cases, it can take weeks or months to troubleshoot certain issues and determine their root cause, and network equipment manufacturers may expend significant resources in their Technical Assistance Centers (TACs) and field engineers to resolve these problems.

[0006] Furthermore, the design trend of embedded systems utilizing open-source software, vendor IP, and off-the-shelf modules exacerbates interoperability issues, as these elements may not have been designed, tested, or configured to work together. Additionally, troubleshooting can be extremely difficult because customer network conditions, environments, and configurations are hard to reproduce, or even impossible to reproduce under identical conditions. When Technical Assistance Center (TAC) staff and field engineers are authorized to troubleshoot issues at a customer's site, they often first use the manufacturer-provided debugging capabilities of production switches, or in some cases, implement instrumented imaging on the problematic node(s). Network administrators (network owners) often hesitate to debug via instrumented imaging because it disrupts business operations and frequently reserve maintenance windows to gather more data when instrumented imaging is possible on live network devices. If the problem is intermittent or occurs during certain network conditions (e.g., peak hours), this time window may not capture the problem. Attached Figure Description

[0007] The embodiments described herein can be better understood by referring to the following description taken in conjunction with the accompanying drawings, wherein like reference numerals denote like or functionally similar elements, in which:

[0008] Figure 1 An illustration shows a debug network device according to an illustrative embodiment, which is created on demand to form a stack with target network devices located in the network.

[0009] Figure 2 This is an illustration showing an exemplary method for establishing and debugging a network device according to an illustrative embodiment.

[0010] Figure 3A , Figure 3B and Figure 3C Illustrations are shown illustrating exemplary methods for performing debugging or profiling operations using a debugging network device according to various illustrative embodiments.

[0011] Figure 4A These are illustrations of debugging network devices configured in a cloud or remote server according to various illustrative embodiments.

[0012] Figure 4B This is an illustration of a debugging network device configured in an evaluation switch according to various illustrative embodiments.

[0013] Figure 4C This is an illustration of a debug network device configured in a debug / evaluation server according to various illustrative embodiments.

[0014] Figure 5 An example operation of a control plane data plane transmission module according to various illustrative embodiments is shown, which is used by an instrumented control plane to update the data plane of a target network device and / or receive updates from the data plane of the target network device.

[0015] Figure 6 A method for establishing and debugging a network device using stateful switching operations is illustrated according to an illustrative embodiment.

[0016] Figure 7 An exemplary timing diagram is shown for a stateful switching operation between a target network device and a test network device according to an illustrative embodiment.

[0017] Figure 8 A system configured to perform debugging or profiling operations in a cloud server using virtualized stateful switching operations, according to an illustrative embodiment, is shown.

[0018] Figure 9 An exemplary sequence is shown for configuring an exemplary network device and using the device to perform debugging or profiling operations according to an illustrative embodiment. Detailed Implementation

[0019] Overview

[0020] Various aspects of the invention are set forth in the independent claims, and preferred features are set forth in the dependent claims. A feature of one aspect may be applied to each aspect alone or in combination with other aspects.

[0021] An exemplary method is disclosed that facilitates the on-demand creation of exemplary instrumented network devices in cloud infrastructure (or remote servers, evaluation platforms, or custom test servers) and the formation of a stack (e.g., using a stacking mechanism) between the instrumented network devices (as debug network devices) and target network devices (e.g., physical network switches or switching infrastructure devices including basic debug capabilities). The control plane of the target network device is then switched to the control plane of the debug network device via a switching operation (e.g., SSO), while the data plane of the target network device continues to operate in a hitless or near-hitless manner by updating control plane and data plane transfer operations that update data transferred from the control plane of the debug network device to the data plane of the target network device. Once the switch is made, the instrumented control plane or the instruments (e.g., hardware or software) of the debug network device facilitate the debugging, optimization, profiling, and / or recovery of the physical network device, even in live networks. In some embodiments, the control plane of the target network device can be recovered with or without restarting the target network device. The target network device can be a standalone, non-redundant physical system, such as a stackable non-redundant switch. The example methods are not necessarily limited to physical network devices; they can also be implemented on non-physical network devices, such as software-based switches.

[0022] A stack consisting of (i) cloud servers or instrumented network devices or servers and (ii) target network devices will operate and function in a manner equivalent to a high-availability (HA) system. Instrumented network devices provide additional temporary hardware and software resources for debugging and profiling physical network devices (e.g., switches). Similarly, debug stacks can be implemented in test or laboratory environments to provide a more robust test platform for debugging or profiling network devices under design or testing. Compared to traditional stacks, control plane and data plane transport operations facilitate control plane updates via the instrumented control plane of the debug network devices to maintain the operation of the target network devices, particularly the data plane of the target network.

[0023] The control plane of a network device can be instrumented or coupled to instruments on the network device to perform control plane functions against the data plane of the physical network device and allow for instrument evaluation of these control plane functions, as well as data plane and network functions. Instrumentation of the control plane, and in some embodiments, the use of instrumented hardware, can additionally provide debugging and profiling information for data plane hardware, firmware, and middleware.

[0024] Debugging and profiling operations can be performed while the target network device continues operating in a near-uninterrupted manner. Debugging or profiling data can be used to adjust the configuration of the control plane, data plane, or network to resolve issues in the production network. Once debugging and / or profiling are complete, exemplary methods and systems facilitate the return of control plane operations to the control plane of the target network device.

[0025] The term "switchover" as used herein (which may be used interchangeably with the terms "stateful switchover" or "SSO") generally refers to a manually or automatically triggered switchover operation from one network device to a redundant or standby network device (e.g., due to a failure of the control plane process), and is similarly used in this embodiment for the control plane between the debugging network device and the target network device. However, instead of switching the data plane of the target network device to the data plane of the debugging network device, the data plane of the target network device is maintained. To this end, during the switchover operation, the control plane operation of the target network device switches from active mode to standby mode, while the control plane of the debugging network device switches from standby mode to active mode, wherein the data plane operation continues to run on the target network device. The switchover operation preferably relies on conventional and / or existing switchover mechanisms and procedures to perform various handshake operations and control plane state synchronization. The switchover mechanism is coupled with other operations described herein, including virtualization mechanisms, cloud infrastructure, and / or control plane and data plane transport operations, etc. In some embodiments, the control plane state between the debugging network device and the target network device can be synchronized using the synchronization operations used in high availability (HA) operations (e.g., uninterrupted service software upgrade (ISSU) operations and / or stateful switch (SSO) operations).

[0026] The term "control plane data plane transport" operation (also referred to herein as "control plane data plane interface transport" operation) refers to the execution of a virtual transport layer (e.g., implemented by the control plane data plane interface transport module described herein, also referred to herein as "virtual PCI" or "VPCI module") at the respective edges of the data plane and control plane in each of the target network device and the test network device to transport bus transactions, such as bus transactions associated with control plane updates and data plane updates, between the data plane of the target network device and the instrumented control plane of the test network device via a communication link (e.g., a network tunnel).

[0027] Switching operations and control plane and data plane transfer operations can be selectively and temporarily used to provide control plane operation to the data plane of a target network device undergoing debugging or profiling of its control plane image and state (including network state). This allows the target network device to maintain uninterrupted or near-uninterrupted operation of its data plane even when its control plane is unavailable (e.g., during a reboot). The term "selectively" as used herein refers to the selective use of additional hardware (virtualized or non-virtualized) as a proxy for the control plane of the active target network device. The term "temporarily" as used herein refers to the limited duration of the use of an instrumented control plane, although in some embodiments it may be expected that the exemplary systems and methods described herein can be used continuously, for example, for monitoring the operation of a target network device.

[0028] The term "data plane" (and data-plane) generally includes one or more data plane processors and one or more data plane resources configured to route packets from one port of a physical network device to another. A data plane processor (also referred to herein as a data plane device) may include processing units involved in packet switching and / or routing within a physical network device, such as a network processor (NPU), a switching ASIC (Application-Specific Integrated Circuit), a switching FPGA (Field-Programmable Gate Array), a CPLD (Complex Programmable Logic Device), etc. Examples of data plane resources may include, but are not limited to: one or more MAC address tables, one or more FIB tables, one or more ACL tables, and any other tables maintained or used by the data plane processor; register contents; content-addressable memory (CAM) contents; tri-state content-addressable memory (TCAM) contents; bi-state content-addressable memory (BCAM) contents; and memory contents (e.g., non-persistent, volatile, etc.).

[0029] The term "control plane" (and control-plane) generally refers to a set of functions and associated control packets or traffic involving configuration and management, protocol state machines, switch states, etc., and is often implemented in the host processor of a switch. Examples of such traffic may include: Spanning Tree Protocol (STP), Hot Standby Router Protocol (HSRP), and control packets sent to or from network devices (e.g., switches), or application layer protocols (e.g., Secure Shell (SSH) and Simple Network Management Protocol (SNMP)) typically handled by the host processor. As used herein, the term "host processor" is used interchangeably with the term "host CPU" and generally refers to the core of a microprocessor or microcontroller (e.g., with a RISC or CISC architecture) of a network device or device of concern (e.g., a physical network device (e.g., a switch)) configured to execute computer instructions within the framework of an operating system in the networked device.

[0030] In some embodiments, the debug network device is instantiated in a cloud or remote infrastructure (e.g., a cloud server or a private server). In some embodiments, state switching operations are performed using a local computing device or a second physical network device with more computing resources or equipped with instrumentation compared to the physical network device, rather than the cloud or remote infrastructure. In some embodiments, the local computing device or the second physical network has processing resources comparable to (or even less than) those of the target network device, but only executes a portion of the system image or application that runs on the target network device, in order to provide those relatively idle resources for instrumentation. In fact, in each of these embodiments, the debug network device does not require its own data plane component, but only the computing power to manage the instrumentation control plane for the target network device. In some embodiments, the debug network device can be implemented across multiple cores or processing units, for example, using hyper-threading capabilities.

[0031] In some embodiments, the debugging network device is configured with computer-readable instructions associated with a system image (e.g., including an operating system and routing applications) that are equivalent to the computer-readable instructions of the physical network device, but are instrumented via external software, using debugging or profiling operations, to be executed concurrently with the system image or control plane application. In such embodiments, modularity also ensures that stateful switching operations are performed on any system image as generated through current development processes and workflows, without requiring customization or modification of the system image for this purpose. In some embodiments, the software image and control plane may, for example, run on a virtual machine (VM) or virtual machine based on virtualization or containerization technologies (which are used interchangeably herein).

[0032] The term "stackable switch" (including "stackable non-redundant switch") refers to a network switch that is fully functional when operating as a standalone device, but can also be configured to operate collaboratively with one or more other network switches as a group, where the group can be configured to exhibit the characteristics of a single switch. In fact, the stack may have a common set of one or more IP addresses for remote management of the stack as a whole. Stackable switches and similar categories of devices can provide redundancy to individual switches during handover operations. Non-stackable switches also refer to switches configured to operate as standalone devices. Stackable and non-stackable switches can be configured with modules to perform the stateful handover operations described herein.

[0033] On the other hand, an exemplary method (and associated system) is configured for a network device to establish a debug cloud or remote infrastructure. The method (system approach) includes: instantiating a debug network device at a remote or cloud computing device (e.g., a cloud computing platform, an evaluation test platform for the target switch, a custom computing server) using an operational image (e.g., a non-redundant or redundant switch) of the target network device (e.g., a non-instrumented or instrumented production image), wherein the debug network device is configured by software instructions or hardware to perform one or more debug procedures not performed on the target network device, and wherein the target network device includes a first control plane and a data plane for receiving and forwarding network traffic in the network; and joining the debug network device and the target network device in a stack configuration to synchronize the state of the first control plane of the target network device to a second control plane of the debug network device, wherein the first control plane of the target network device is initially executed in an active stack configuration, and wherein the second control plane of the debug network device is initially executed in a standby stack configuration. The process is executed concurrently with the second control plane of the network device to evaluate at least one of the following: i) the second control plane, ii) the hardware or firmware configuration of the target network device, iii) the network (e.g., to analyze the hardware, firmware, and / or timing characteristics of a real-time or non-real-time network device, to analyze the protocol operation of the network or the timing of the network device).

[0034] In some embodiments, the method includes: using control plane and data plane transmission operations to update the data plane of a target network device by debugging the second control plane of the network device.

[0035] In some embodiments, one or more procedures for debugging network devices are used to recover a target network device from an error or invalid state associated with the data plane and / or control plane (e.g., to recover and restore a problematic switch in a live network).

[0036] In some embodiments, the one or more debugging processes are associated with at least one of the following analysis tools: network, ASIC, custom, system analysis tools such as Valgrind and Callgrind, call graph analyzer, memory analyzer, and cache profiler.

[0037] In some embodiments, the step of instantiating the debugging network device is performed dynamically (e.g., automatically or manually debugging the target network device), and the method further includes: deleting the second control plane of the debugging network device (e.g., at a remote or cloud computing device) after evaluating and restoring the target network device.

[0038] In some embodiments, the one or more debugging processes are used to profil the target network device or network to optimize the data plane configuration and / or control plane configuration of the target network device (e.g., ASIC configuration, hardware configuration, firmware configuration, hardware-software interoperability configuration).

[0039] In some embodiments, the one or more debugging processes are used to profil the target network device or network to optimize network operation, network policy, or network configuration.

[0040] In some embodiments, when the second control plane of the target network device is debugged by one or more debugging procedures performed at the debugging network device, the second control plane in the active stack configuration provides near-uninterrupted operation for the target network device.

[0041] In some embodiments, the operational image includes a non-instrumented production image.

[0042] In some embodiments, the target network device includes a non-redundant switch.

[0043] In some embodiments, the target network device includes a redundant switch.

[0044] In some embodiments, the join operation is performed using a stacking technique.

[0045] In some embodiments, the switching operation is performed by at least one of the following: high availability (HA) operation, uninterrupted service software upgrade (ISSU) operation, or stateful switch (SSO) operation.

[0046] In some embodiments, the debugging network device further includes an instrumented data plane, which includes a data plane component and additional instruments for accessing the data plane component, wherein the one or more debugging processes or additional instruments are configured to send one or more debugging packets to the data plane component, and wherein the instruments are configured to provide logging and / or analysis for the one or more debugging packets.

[0047] In some embodiments, the remote or cloud computing device includes at least one of the following: a cloud server, a remote server, an evaluation platform for the target network device, and a custom computing server that includes one or more debugging or evaluation subsystems (e.g., FPGA card, GPU card, RTL emulator, hardware accelerator, PCIe analyzer).

[0048] In some embodiments, the step of instantiating and debugging a network device includes: retrieving an operational image of the target network device from an image server; and using the retrieved operational image (e.g., where the operational image may be i) the same operational image as the target switch, or ii) an instrumented operating system equivalent to the target switch) to coordinate a virtualized environment (e.g., a container or virtualization) with the operating system and environment.

[0049] In some embodiments, the method further includes: establishing a tunnel connection or a direct connection between the debugging network device and the target network device, wherein the tunnel connection or direct connection serves as a network connection for the debugging network device to update the data plane of the target network device.

[0050] On the other hand, a system is disclosed (e.g., a non-redundant switch, a controller (e.g., DNAC, SDN controller, etc.), a remote / cloud system, a remote terminal), comprising: a processor; and a memory having computer-readable instructions, wherein execution of the computer-readable instructions causes the host processor to: instantiate a debug network device at a remote or cloud computing device (e.g., a cloud computing platform, an evaluation test platform for the target switch, a custom computing server) using an operational image (e.g., a non-instrumented or instrumented operational image) of the target network device (e.g., a non-redundant switch or a redundant switch), wherein the debug network device is configured by software instructions or hardware to perform one or more debugging procedures not performed on the target network device, and wherein the target network device includes a first control plane and a data plane for receiving and forwarding network traffic in the network; and to join the debug network device and the target network device in a stack configuration (e.g., via a stacking protocol, such as stacked cables (e.g., back stacking) or stacked virtual links (SVL)) to join the first... The state of the control plane is synchronized to the second control plane of the debug network device, wherein the first control plane of the target network device is initially executed in an active stack configuration, and wherein the second control plane of the debug network device is initially executed in a standby stack configuration; and a handover operation is triggered (e.g., via a high availability (HA) operation, such as a service software upgrade without interruption (ISSU) operation or a stateful switch (SSO) operation), wherein the first control plane of the target network device switches from the active stack configuration to the standby stack configuration and disconnects without updating the data plane of the target network device, and wherein the second control plane of the debug network device switches from the standby stack configuration to the active stack configuration and connects via a network connection to update the data plane of the target network device, wherein the one or more debug processes are operable concurrently with the second control plane of the debug network device to evaluate at least one of the following: i) the second control plane (which reflects the operation of the first control plane), ii) the hardware or firmware configuration of the target network device, and iii) the network.

[0051] In some embodiments, the method includes: transmitting operations using a control plane and a data plane, and updating the data plane of a target network device via an instrumented control plane.

[0052] In some embodiments, when these instructions are executed by the processor of the network device being debugged, they also cause the processor to perform one or more debugging procedures to restore the target network device from detected error or invalid states associated with the data plane and / or control plane (e.g., ASIC configuration, hardware configuration, firmware configuration, hardware-software interoperability configuration) of the target network device (e.g., recovering and restoring a problematic switch detected in a live / production network).

[0053] In some embodiments, the system (e.g., via the host processor or second processor unit or logic circuit of the target network device) is configured to: read a set of bus interconnect transactions (e.g., originating from the data plane) from the bus interconnect of the target network device, and transmit the set of bus interconnect transactions as a set of data plane transaction messages via a network interface (e.g., via control plane and data plane transmission operations) to the debug network device, wherein the debug network device is configured to: use the set of data plane transaction messages to write (e.g., via control plane and data plane transmission operations) the set of bus interconnect transactions to the bus interconnect or the host processor of the debug network device to update the control plane state maintained by the debug network device; and based on a second set of data plane transaction messages received from the debug network device via the network interface, write (e.g., via control plane and data plane transmission operations) a second set of bus interconnect transactions to the bus interconnect of the target network device, wherein the second set of bus interconnect transactions updates a portion of a plurality of data plane association tables of the target network device.

[0054] On the other hand, a computer-readable medium is disclosed having instructions stored thereon, wherein these instructions are executed by a processor (e.g., a processor of a non-redundant switch, a controller (e.g., DNAC), a remote / cloud system, or a TAC remote device), causing the processor to: instantiate a debug network device at a remote or cloud computing device (e.g., a cloud computing platform, an evaluation test platform for the target switch, or a custom computing server) using an operational image (e.g., a non-redundant or redundant switch) of the target network device (e.g., a non-instrumented or instrumented operational image), wherein the debug network device is configured by software instructions or hardware to: execute one or more debugging procedures not executed on the target network device, and wherein the target network device includes a first control plane and a data plane for receiving and forwarding network traffic in the network; and to cause the debug network device and the target network device to join a stack configuration (via a stacking protocol, such as stacked cables (e.g., back stacking) or stacked virtual links (SVL)) to adjust the state of the first control plane of the target network device. The first control plane of the target network device is initially executed in an active stack configuration, and the second control plane of the target network device is initially executed in a standby stack configuration; and a switching operation is triggered (e.g., via a high availability (HA) operation, such as a business software upgrade without interruption (ISSU) operation or a stateful switching (SSO) operation), wherein the first control plane of the target network device switches from the active stack configuration to the standby stack configuration and disconnects without updating the data plane of the target network device, and wherein the second control plane of the target network device switches from the standby stack configuration to the active stack configuration and connects via a network connection to update the data plane of the target network device, wherein the one or more debugging processes are operably executed concurrently with the second control plane of the target network device to evaluate at least one of the following: i) the second control plane (which reflects the first control plane), ii) the hardware or firmware configuration of the target network device, and iii) the network.

[0055] On another front, the exemplary systems and methods facilitate the profiling and debugging of target systems in real-time networks without impacting the performance, throughput, and / or functionality of the target devices. This can also be done in a non-disruptive manner (for both the control plane and the data plane).

[0056] On the other hand, the exemplary systems and methods enhance the ability to debug and profil the target system (for a short or long period of time) and provide this capability on demand by instantiating instrumented images in the cloud or remote servers.

[0057] On another front, exemplary systems and methods facilitate the on-demand creation of a control plane on a debug switch (e.g., executed in the cloud or on a remote server) that can execute different software images (instrumented or other software images) that can control the target node without altering its intended behavior and functionality. In other words, the debug switch can execute different software images that are functionally equivalent and can be performed dynamically in a non-disruptive or near-non-disruptive manner.

[0058] In another aspect, exemplary systems and methods facilitate the on-demand creation of data planes and the dynamic linking and association of these data planes with debugging or profiling environments, which may include various hardware accelerators, simulators, data models, etc., that can be used to debug / profil the data plane (hardware, software, and / or middleware).

[0059] On another front, the exemplary systems and methods facilitate near-uninterrupted recovery of problematic systems in real-time networks. Recovery can be performed without restarting and without interrupting or interfering with the network.

[0060] On another front, exemplary systems and methods facilitate the operation of a debug switch (as a shadow system), in some embodiments of which the debug switch is configured to operate in a passive mode, in which the debug switch receives the same set of control plane updates, messages, and signals from the target node (which remains in the active role) while in a standby role.

[0061] In another aspect, exemplary systems and methods are configured to employ coordination mechanisms, such as coordination mechanisms via network controllers (e.g., DNAC) or other controllers or centralized control tools.

[0062] In another aspect, exemplary systems and methods are configured to provide protection against system failures during a live debug session at the target node. In such embodiments, when the active control plane fails, the standby control plane of another device is configured to switch over without manual intervention and take over operations on the data plane of the physical network device. System operation is not interrupted during the switchover. In fact, failover operations from active to standby devices can be performed via SSO / HA mechanisms, whether from the debug network device to the target network device or vice versa.

[0063] In another aspect, exemplary systems and methods are configured to execute a debug switch that operates a control plane that updates the data plane by at least one of the following: (i) connecting to the data plane in the target node, (ii) instantiating a separate data plane locally using a software model and / or RTL model running on the same machine as the control plane, and (iii) implementing a hardware data plane within the virtual debug switch itself.

[0064] Example System

[0065] Figure 1 A diagram of a debug network device 102 is shown, which is created on demand to form a stack 103 with a target network device 104 located in network 106. The debug network device 102 executes the control plane 108 (with instrumentation code or hardware) corresponding to the control plane 110 of the target network device 104 to facilitate debugging, profiling, or recovery of the target network device 104.

[0066] The stacking 103 between the debug network device 102 and the target network device 104 is formed through a stacking mechanism or stacking protocol. The stacking operation synchronizes various system states of the target network device 104 (e.g., through batch synchronization and subsequent incremental synchronization) to the debug network device 102, which is instrumented for debugging and / or profiling. When the stacking 103 is formed, the debug network device 102 is connected to the target network device via a communication link 116 (shown as network connection 116a or direct connection 116b) established between the target network device 104 (e.g., at a port on data plane 114) and port 118 of the debug network device 102.

[0067] In stack 103, target network device 104 is initially placed in active mode, while debug network device 102 is placed in standby mode. In active mode, the control plane 110 of target network device 104 serves the data plane 114 of target network device 104. Once both target network device 104 and debug network device 102 are state-synchronized, a handover operation is triggered to place the control plane 108 of debug network device 102 into active mode, which then takes over the role of serving the data plane 114 of target network device 104. It is worth noting that after the handover operation, the data plane 114 remains operational throughout the debug or profiling session—therefore, from the perspective of the network and peer network devices, the handover operation does not change the network. In this active state, debug network device 102 provides instrumented operations (e.g., through an instrumented control plane, hardware instruments, or both) to debug or profil the control plane operations, data plane operations, and network operations of the stack and its sub-components.

[0068] Examples of switching operations include high availability (HA) operations, such as uninterrupted service software upgrade (ISSU) operations or stateful switchover (SSO) operations. To enable the control plane 108 of the debugging network device 102 to serve the data plane 114, a virtual transport layer can be implemented, for example, through a control plane data plane interface transport module (also referred to herein as a "virtual PCI" or "VPCI module"), and in... Figure 1 The diagram is shown as "Control Plane Interface / Data Plane Interface Transport" (CPI / DPI Transport) 120 and 122. In some embodiments, the virtual transport layer is performed at the respective edges of the data plane and control plane in each of the physical network device 102 and the debug network device 102 to transport bus transactions 126, such as transactions associated with control plane or data plane updates, between the data plane 114 of the target network device 104 and the control plane 108 of the debug network device 102 via a communication link (116a or 116b). For a further description of the example control plane-data plane transport module, please refer to U.S. Patent Application No. 17 / 29559, filed December 2, 2020, the entire contents of which are incorporated herein by reference.

[0069] Debugging network device 102 can be different types of machines (such as...) Figure 4A , Figure 4B and Figure 4C As shown in the image, the machine executes a debug / instrumentation image (or includes instrumentation hardware) corresponding to the software version running on the target network device 104. Figure 4A In the diagram, network device 102 (shown as 102a) is a cloud or remote server. Figure 4B In this context, network device 102 (shown as 102b) is used to evaluate the switch. Figure 4C In this embodiment, debug network device 102 (shown as 102c) is a debug / evaluation server. In some embodiments, debug network device 102 includes additional instruments or simulators 124.

[0070] Debugging network device 102 is typically a high-end computer that runs the instrumented control plane corresponding to the control plane of the target network device, or is instrumented for debugging / profiling. Debugging network device 102 is configured to perform operations equivalent to the control plane 110 of the target network device 104 in the active mode of stack 103, thereby enabling the collection of profiling / debugging data in any environment, including production networks. Once the debugging session is complete, the temporary roles of debugging network device 102 and target network device 104 in stack 103 are swapped, and target network device 104 is placed in active mode in stack 103. Debugging network device 102 can then disconnect and delete the instrumented control plane 108.

[0071] In some embodiments, the debug network device 102 is implemented in a data center, public cloud, or similar environment using general-purpose or custom-designed computing hardware and software. Each product line from a given device manufacturer can utilize these general-purpose and custom-designed resources for instrumentation. Technical Assistance Center (TAC) personnel and field engineers can use one of these nodes as needed to debug the target network device and release it upon completion of debugging. In other embodiments, the debug network device 102 is implemented in a standalone computing platform (e.g., another network device or a custom computing platform) that can be brought to the field of the given network, enabling TAC personnel and field engineers to link the computing platform to the target network device via a network connection or direct connection to perform the debug / profiling operations described herein.

[0072] In some embodiments, the debugging network device 102 includes an instrument 124 consisting of debugging hardware, a test board, and a simulator, for operation in conjunction with an instrumentation control plane 108. The instrument 124 may include commercially available or custom-made debugging hardware, test boards, simulators, and software.

[0073] Establish methods for debugging network devices

[0074] Figure 2 This is an illustration of an exemplary method 200 for establishing a debug network device 102 according to an illustrative embodiment. First, the debug network device 102 is instantiated (202) to have a control plane 108. In some embodiments, the instantiation operation requires loading a system image of a target network device into the debug network device or booting that loading. In the application space of the debug network device, the loaded system image may be instrumented, or instrumentation software may be executed. In some embodiments, the debug network device 102 includes debug or profiling hardware that can be instantiated in process 202.

[0075] Method 200 then includes establishing (204) a stack 103 between the debug network device 102 and the target network device 104. The control plane 110 of the target network device is initially designated to be in an active mode of the stack 103, while the control plane 108 of the debug network device is in a standby mode. In some embodiments, a (204) connection 116 is established between the debug network device 102 and the target network device 104. In some embodiments, connection 116 (shown as 116a) is a network tunnel established between the debug network device 102 and the target network device 104 via network 106 (shown as 106a). In other embodiments, connection 116 (shown as 116b) is established as a direct communication link between the debug network device 102 and the target network device 104. The direct communication link may include, but is not limited to, a direct serial communication link that can adequately provide control plane and data plane updates, such as an Ethernet crossover cable, Ethernet, USB, FireWire, SVL link, Sonet / SDH, Frame Relay, X.25, T1 / E1, etc. Direct communication links can also refer to wireless links such as WiFi (e.g., 802.11), LTE 4G / 5G, and WiMAX.

[0076] After stack 103 is formed, bulk synchronization and / or incremental synchronization of the stacking protocol are performed to synchronize the state (e.g., control plane state) of the target network device 104 (in active mode) with that of the debug network device (in standby mode). Once the states of both the target network device 104 and the debug network device 102 are synchronized, method 200 includes, during a handover operation, switching (206) the instrumented control plane 108 of the debug network device 102 in the stack from standby mode to active mode, and switching the control plane of the target network from active mode to standby mode. During and after the handover operation (206), the data plane operation of the target network device 104 (shown via data plane 114) is maintained, and the target network device continues to actively route any traffic it receives to the appropriate network node. Concurrent with the continuous data plane operation of the target network device 104 (performed by the data plane 114), the control plane 108 of the debugging network device 102 takes over the control plane operation of the target network device 102 in active mode and provides or supplies (e.g., via virtual transport operation via connection 116) any control plane updates (and management plane updates) to the data plane 114.

[0077] At this point, the instrumentation control plane 108 has been established and is operating in active mode to control the target network device 104. Method 200 may then include performing (210) debugging and / or profiling operations at the instrumentation control plane 108 of the debugging network device 102. While the debugging or profiling operation is in progress, as described above, the control plane 108 of the debugging network device 102 continues to provide control plane updates to the data plane 114 of the target network device 104. In some embodiments, the instrumentation control plane 108 may generate a notification to the user informing them that the control plane 108 of the debugging network device is in a debug-ready state. In some embodiments, the notification may be information from a command-line interface or a status window.

[0078] During step 210, debugging network device 102 may be triggered to run a profiler (e.g., a runtime profiler or debugger), such as a cache simulator, branch predictor, call graph analyzer, Valgrind, etc. Control plane 108 (instrumentized) may generate trace logs or execute memory analysis tools, such as a memory leak detector, memory profiler, dynamic analysis tool, memcheck (memcheck tool), etc. Debugging operations may include adding instructions or commands to execute system images or control plane applications. Debugging operations may be performed over a period of several hours, after which the activity mode is switched to control plane 110 of target network device 104. In some embodiments, debugging operations may be maintained for an extended period of time (e.g., running overnight and / or running for days or even weeks, for example, to perform profiling). Concurrently with, before, or after debugging operation 210, control plane 108 of debugging network device 102 is configured to update (208) the control plane or system state via the control plane data plane interface transfer module.

[0079] In fact, the debugging network device 102 can take over the control plane operation of the target network device 104 in a non-interrupted or near-non-interrupted manner, that is, without interrupting the switching operation of the target network device.

[0080] Methods of debugging / profiling using debugging network devices

[0081] Figure 3A and Figure 3B Illustrations are shown of exemplary methods 300 (shown as 300a, 300b, respectively) for performing debugging or profiling operations using debugging network device 102 according to various illustrative embodiments. Figure 3A An operational example of establishing and debugging a target network device 104 using a debugging network device 102 is shown according to an illustrative embodiment. Figure 3B Another debugging operation according to an illustrative embodiment is shown to enable the target network device 104 to recover without interrupting or interfering with the network.

[0082] Debugging operation example #1. exist Figure 3A and Figure 3B In the methods 300a and 300b, respectively, the methods include establishing and executing (shown via 202-208) a debug network device 102 having a control plane 108 (e.g., an instrumentation control plane) corresponding to the target network device 104 in an active mode, and a corresponding set of debug operations (e.g., as shown in reference). Figure 2 (As described). Next, different methods for debugging and profiling the target network device 104 will be shown.

[0083] exist Figure 3A In the process, after the debugging network device 102 initializes and controls the data plane 114 of the target network device 104 (steps 202-208) and the debugging operation (step 210) is performed, method 300a further includes: adjusting (302) the configuration of the control plane of the target network device using the acquired debugging / profiling data. The adjustment (302) may be a configuration of the control plane configuration, data plane, network settings, or any configurable or reprogrammable feature of the target network device (including the ASIC and various hardware of the target network device 104). In some embodiments, the adjustment (302) includes adjusting a specified boot system image of the target network device 104 as an upgrade to the control plane 108.

[0084] Method 300a then includes restarting (304) the control plane 110 of the target network device 104 and removing the stack configuration between the target network device 104 and the debug network device 102. Once loaded, the target network device 104 will resume operation after the vulnerability or misconfiguration issue is resolved. Additional debugging and profiling operations can be repeated until the expected results are achieved.

[0085] Following or concurrently with the restart operation (304), the control plane 108 of the debugging network device 102 may be disabled, and in some cases may be deleted (306). In some embodiments, during this shutdown process (306), the configuration of the control plane 108 and / or the control plane of the debugging network device 102 may be stored so that it can be retrieved later for, for example, analysis or use.

[0086] In fact, method 300a can be executed while the target network device 104 continues to operate in a real-time environment, while minimizing or eliminating interference with the network 106. Similarly, methods 200 and 300 can be executed on the target network device 104 in a controlled test or laboratory environment (e.g., during the design and / or testing of the network device).

[0087] Debugging Operation Example #2 - Recovering a faulty switch with minimal service impact.In some embodiments, in order to restore network devices with minimal service impact, the control plane 110 of the target network device 104 can be guided back to active mode, and recovery can be performed (whether or not a reboot) with no or even less service interruption to the network 106 from the target network device 104.

[0088] exist Figure 3B In the process, after adjusting the configuration of the control plane 110 of the target network device 104 (step 302), for example, regarding... Figure 3A The method 300b further includes switching the instrumented control plane 108 in the stack from active mode to standby mode via a second switching operation, and switching the control plane of the target network from standby mode to active mode. In effect, control plane operation on the target network device 104 has now been restored without any network interruption.

[0089] Prior to the second handover operation (step 308), the control plane 110 of the target network device 104 can be rebooted and reconnected to the stack in standby mode. The second handover operation (308) can then be performed to put the control plane 110 of the target network device 104 into active mode. As discussed above, the adjustment operation 302 may include adjusting the configuration of the target network device's control plane, data plane, network settings, or any configurable or reprogrammable features, including specifying a new boot system image for upgrading the control plane 108 of the target network device 104.

[0090] and Figure 3A Similarly, after switching, the instrumentation control plane 108 can be disabled and, in some cases, deleted (306). In some embodiments, the instrumentation control plane 108 and / or its configuration can be stored for later retrieval, for example, for analysis or use.

[0091] In fact, if a non-redundant switch in a customer's network is affected by an uncorrectable / unrecoverable problem (e.g., partial traffic loss, uncorrectable memory errors, hardware problems, control plane / data plane inconsistencies, etc.), and reloading is the only option, then the example method can be used in certain situations to recover the target switch with minimal impact. In some embodiments, the debug network device can be spawned in the cloud and then stacked with the target network device (the problematic switch), subsequently assuming an "active" role when the target switch is able to undergo a reboot. The debug switch can collect additional data from the target network device's data plane for analysis before the target network device is reset. In some embodiments, this operation provides near-uninterrupted control plane functionality and data plane functionality. In other embodiments, this operation provides near-uninterrupted control plane functionality and data plane functionality with brief interruptions (e.g., seconds or even sub-seconds).

[0092] This exemplary workflow, as well as other workflows described herein, can be integrated with network controllers (e.g., DNAC) or other controllers or centralized control tools, which can be used to coordinate or trigger orchestration, switching, to configure and debug network devices. In some embodiments, triggering can be automated based on predefined rules or policies.

[0093] In some embodiments, when a real-time debugging session is in progress and an engineer / TAC performs an intrusive debugging operation that causes the debugging network device to crash, the same HA / SSO mechanism can automatically trigger the control plane 110 to switch to active mode to restore normal operation in an uninterrupted manner.

[0094] Example of debugging network devices

[0095] Figure 4A , Figure 4B and Figure 4C Example debugging network devices 102 (shown as 102a, 102b and 102c respectively) according to illustrative embodiments are shown respectively.

[0096] Cloud infrastructure. Figure 4A A debug network device 102a implemented in a remote or cloud computing device (e.g., a cloud computing platform) is illustrated. In some embodiments, the remote or cloud computing device may be implemented in Amazon AWS, Microsoft Azure, Cisco Cloud Solutions, Google GCP, or any public / private cloud or local / remote network. Figure 4AIn this embodiment, the remote or cloud computing device 102a is preferably a modular, streamlined, high-end general-purpose server computer. In some embodiments, the debugging network device 102a is implemented on a high-end general-purpose computer. This computer can accommodate a powerful CPU (e.g., 64 / 128 / 256 cores) and hundreds of gigabytes of RAM, and is capable of running custom software models, RTL emulators, simulators, etc., to achieve the corresponding functionality of a network switch.

[0097] Exemplary debugging network devices are typically configured with greater computing power than the target network device, such as more cores, more memory resources, faster clock speeds, and larger caches. These resources can be aggregated from multiple distributed computing resources using scalable cloud systems.

[0098] exist Figure 4A In this embodiment, the debugging network device 102a runs an instrumentation control plane 108 (shown as 108a). The debugging network device 102a executes a system image 402 and instrumentation code 404 within the instrumentation control plane 108a. The instrumentation code 404 within the instrumentation system image 402 can be used to generate trace logs and may include command-line functionality to evaluate various subsets of modules applied to the system image or control plane. In some embodiments, the instrumentation control plane 108a includes application code 406, which executes within the network device's operating system and may include the instrumentation code 408. In other embodiments, the instrumentation control plane 108a includes debugging or profiling software 410 installed in its application space.

[0099] In some embodiments, network device 102a is debugged using general-purpose or custom-designed computing hardware and software in a data center, public cloud, private cloud, or similar environment. These general-purpose and custom-designed resources can be used for instrumentation across each product line of a given network device manufacturer.

[0100] Switch platform. Figure 4B A debug network device 102b implemented in an evaluation / verification platform is shown. The evaluation / verification platform 102b can implement a debug switch that is functionally equivalent to or similar to the target network device 104.

[0101] The debugging network device 102b may include an instrumentation control plane 108. In some embodiments, the debugging network device 102b may include an instrumentation data plane 414. Generally, the ASIC block and embedded microcontroller in the design may include additional I / O debug pins that, for security reasons, are intentionally not exposed or not included in the production switch, but are now exposed or included in the production switch for debugging purposes. The debugging network device 102b may be a custom development platform that is typically used during network device development and can be configured to enable these I / O debug pins in the ASIC block and embedded microcontroller of the debugging network device 102b. In some embodiments, the instrumentation data plane 414 includes a data plane of the debugging network device 102b that is instrumented by an external test device.

[0102] Evaluation platform 102b may include instruments 124, which include debugging hardware, line cards, test boards, hardware and / or software simulators, hardware accelerators, graphics processing units (GPUs), RTL simulators, PCIe / AXI or various analyzers, which may be installed in debugging network device 102b (see also...). Figure 4C ( ), to operate in conjunction with the instrumentation control plane 108. Instrument 124 can be connected to a separate commissioning terminal 416.

[0103] In addition, instrument hardware and systems 418, which are external test equipment (e.g., oscilloscopes, logic analyzers, EMI evaluation equipment, network test equipment, etc.), can be used to evaluate the instrumented control plane 108 or the instrumented data plane 414.

[0104] The evaluation platform 102b can be used during hardware startup and has additional functionality to assist in debugging. In some embodiments, the evaluation platform 102b includes additional debug pins in the switch board and module.

[0105] In some embodiments, exemplary methods and systems can be configured to debug data plane problems, wherein the target network device is configured to selectively perform traffic mirroring operations. For most use cases, a small number of packets can be copied to the data plane for debugging. Debugging the network device allows these packets to pass through an instrumented data plane for detailed analysis. For example, an ASIC simulation model in software can be implemented as a data plane implementation. This model can be used to provide detailed logging and analysis. Current troubleshooting sessions are often limited to offline debugging / analysis, such as collecting the ASIC status of the target network device (e.g., a switch) through multiple iterations and replaying it in a lab with the necessary packets. Instrumented control planes facilitate real-time analysis of traffic from real-time systems.

[0106] In some embodiments, the hardware data plane may be implemented in an evaluation platform 102b that operates in passive mode while the control plane state is synchronized with the target node. In this passive operation, the debug network device does not switch to active mode. Instead, the debug network device 102 is always in passive mode during troubleshooting operations. This approach and operation can be used to debug / profil processes that are SSO-aware and in hot standby mode. All control plane updates can be synchronized to the debug network device, which will have the same code flow. In embodiments where the debug network device 102 includes a data plane, the data plane will be placed in passive mode.

[0107] Customized computing servers. Figure 4C A debug network device 102 (shown as 102c) implemented in a custom computing server is illustrated. This custom computing server is designed for a specific product line of a given network device and is tightly integrated with hardware accelerators, FPGAs, hardware emulators, etc.

[0108] In some embodiments, such as Figure 4C As shown, the debugging network device 102c includes instruments in the form of development tools, such as software model 402 or simulation model 404.

[0109] A data plane simulation model 430 (e.g., a data plane simulation model of an operating system for an ASIC VTP) can be implemented in the instrumentation control plane 108. In such examples, traffic can be manipulated (e.g., via tunneling) and fed into the network ports of the data plane simulation model. This debugging operation provides potentially enhanced debugging of data plane traffic through carefully crafted event logs and trace messages from the RTL model.

[0110] In addition, such as regarding Figure 4B As described in the text and now Figure 4C As shown, the debugging network device 102c may also include various hardware development and debugging tools, such as a hardware accelerator 420, a graphics processing unit (GPU) 422, an RTL simulator 424 (e.g., a register transfer layer (RTL) description, Verilog, or HDL simulator), and a PCIe analyzer 428. Instrument 124 (shown as 124b) can be connected to a separate debugging terminal 416.

[0111] In addition, such as regarding Figure 4B As described in the text and now Figure 4C As shown, the instrumentation hardware and system 418 (shown as 418a) may include an oscilloscope, logic analyzer, EMI evaluation equipment, network test equipment, etc., as external test equipment, which can be used to evaluate the instrumentation control plane 108 (e.g., 108a) (or as shown in the image). Figure 4BThe instrumentation data plane 414 is shown. The instrumentation hardware and system may also include various hardware development and debugging tools, such as hardware accelerator 420 (shown as 420a), graphics processing unit (GPU) 422 (shown as 422a), RTL simulator 424 (e.g., register transfer layer (RTL) description, Verilog or HDL simulator) (shown as 424a), and PCIe analyzer 428 (shown as 428a).

[0112] Control plane data plane transmission module

[0113] Figure 5 Example operation of control plane data plane transmission modules 120, 122 (denoted as "VPCI") 120, 122 according to various illustrative embodiments is shown. These control plane data plane transmission modules are used by the instrumented control plane to update the data plane 114 of the target network device 104 and / or receive updates from the data plane 114 of the target network device 104. Figure 5 In this configuration, the target network device 104 and the debugging network device 102 each include control plane-data plane interface transmission modules 120 and 122, which respectively provide logical device access operations, such as bus transactions or their logical equivalents. The control plane-data plane interface transmission modules 120 and 122 are configured to transmit bus transactions between the target network device 104 (specifically, the data plane 114) and the debugging network device 102 (specifically, the instrument control plane 108).

[0114] The instrumented control plane 108 of the debugging network device 102 uses control plane-to-data plane transmission modules 120 and 122 to process control plane updates received at the data plane 114 of the target network device 104. Data plane updates determined at the instrumented control plane 108 are also pushed to the data plane 114 of the target network device 104 via control plane-to-data plane interface transmission modules 120 and 122. For example... Figure 5 As shown, the debugging network device 102 may or may not include its own data plane 501.

[0115] Example control plane update.In some embodiments, the configuration of the data plane 114 of the target device 104 is initiated by the control plane 110 of the debugging network device 102. Once a control plane update packet is received at the target network device, the control plane data plane transmission module (shown as VPCI) 122 is configured to perform read and write transactions. That is, the control plane data plane transmission module can read bus transactions 502 (e.g., control plane updates) from the data plane 114 of the target network device 104 (intended for use with its control plane 110) and provide the read transactions 502 to the network interface 504, which is connected via communication link 116 (in Figure 1 (Illustrated as 116a or 116b) The transaction 502 is sent as message 506 to debug network device 102. Debug network device 102 receives message 506 and writes bus transaction 508 to the bus interconnect 510 of debug network device 102 or its logical equivalent via the corresponding control plane-data plane interface transmission module 120 (illustrated as "VPCI" 120), thereby writing 512 to the control plane 108 of debug network device 102. At this point, the stacked control plane has been updated. An example control plane update is an OSPF update "punt" packet. The bus interconnect (e.g., 510, 514) is a bus interface such as PCI, PCIe (PCI-Fast) bus, AXI, SPI (System Packet Interface), PCI-X, PCI-Fast 16x, PCI-Fast 1x, PCIe 4.0, PCIe 5.0, PCIe 6.0, or similar. Similarly, the control plane data plane transmission module 120 can provide a data plane update (e.g., as a result of a control plane update) to the data plane 114 of the target network device 104 by acquiring the update and sending it as a message 506 via the communication link 116. The target network device 104 receives the message 506 and writes it into its bus interconnect 502 or its logical equivalent via the corresponding control plane-data plane interface transmission module 122, thereby writing it into the data plane 114.

[0116] Example data plane update. Similarly, the control plane data plane interface transmission module 120 of the debug network device 102 is configured to acquire bus transaction 520 or its equivalent from the instrumentation control plane 108. The read bus transaction 520 or its equivalent is provided to the network interface 522 of the debug network device 102 and sent as a message (similar to 506) via communication link 116 to the target network device 104, which reads the message. The control plane data plane interface transmission module 122 of the target network device 104 uses the message (similar to 506) to write the bus transaction (similar to 502) to the bus interconnect 514, and then to the data plane 114.

[0117] Message 506 (addressing a control plane or data plane update) can be of any format or follow any protocol. In some embodiments, a bus transaction is encapsulated as a payload within an encapsulated packet (which serves as a message). In some embodiments, multiple bus transactions may be encapsulated as a payload within a single encapsulated packet. In some embodiments, message 506 includes a tunnel header 514, a packet header 516, and a packet payload 518. In some embodiments, message 506 is sent using an existing stacking operation that encapsulates the packet header and packet payload together with an SVL header 520, encapsulating the connection-associated tunnel header into the resulting packet. However, message 506 can be of any format or follow any protocol. In some embodiments, a bus transaction is encapsulated as a payload within an encapsulated packet (which serves as a message). In some embodiments, multiple bus transactions may be encapsulated as a payload within a single encapsulated packet.

[0118] In some embodiments, the control plane-data plane transmission module 122 is implemented as an integrated component of the target network device 104. In other embodiments, the control plane-data plane interface transmission module 122 is implemented as core or logic circuitry in the ASIC of the target network device 104. In yet another embodiment, the control plane-data plane interface transmission module 122 is implemented in the core or logic circuitry of an auxiliary card of the target network device. For a further description of the control plane-data plane transmission module, please refer to U.S. Patent Application No. 17 / 29559, filed December 2, 2020, the entire contents of which are incorporated herein by reference.

[0119] Debugging network devices using examples of stacked protocols, stateful switching, and virtual transport layers.

[0120] Figure 6 The illustration depicts the establishment of a debug network device 102 using a stacking protocol and stateful switching operations according to an illustrative embodiment. Stacking and stateful switching operations are complementary concepts and not interchangeable. Generally, stateful switching operations do not require stacking (e.g., stateful switching operations can be performed in any system with multiple control planes, such as a modular system with dual supervisors), but are used herein to establish the debug network device 102 as an active device to control the data plane of the target network device 104.

[0121] Stacking is generally a process in which the members of a stack form a single logical entity managed by one of these entities (called the "active entity"). Other stack members are in "standby" mode, or, if there are more than two, the rest are simply "members." Stacks are formed according to stacking protocols. Examples of stacking protocols include stacked cables (e.g., back-end stacking) and stacked virtual links (SVLs). Figure 6 In this exemplary system, a remote server or cloud server 602 is orchestrated (or configured as a debug machine as described herein), and a secure tunnel 116a is established between the standby remote / cloud server 102a and the active target network device 104 to form a stack. Once a stack configuration is established between the target network device and the debug network device, a bulk synchronization operation (e.g., a bulk synchronization operation of the stack protocol) can be initiated and synchronization continues until the configuration of the active target network device is synchronized with the standby debug network device. Incremental synchronization can then be performed for any subsequent updates. A virtual transport layer, such as a control plane data plane transport operation, takes over the updates after the switchover operation occurs and is generally a separate operation from the synchronization operation of the stack protocol.

[0122] In another embodiment, according to an illustrative embodiment, virtualized high availability operation is used to establish a virtualized redundant debug switch as another form of virtualized standby switch. Therefore, the terms "virtualized redundant switch" and "virtualized standby switch" are used interchangeably in this disclosure. Virtualized high availability operation originates from high availability (HA) or similar operation. In virtualized high availability operation, as in high availability operation, network devices are connected via configurable control links and data synchronization links. The control link is used to transmit the status of the network devices. The data synchronization link is used to transmit stateful information to synchronize stateful databases for calls and media streaming. Each pair of redundant interfaces can be configured with the same unique ID number.

[0123] Stateful switching (SSO) is a mechanism that changes a state from "standby" to "active." This can be caused by a failure in the active member or by operationally forcing it into that state. Stateful switching is generally used for (e.g., Figure 6 (As shown on the left) Fault resilience is provided for the active / primary stackable switch / chassis by employing a redundant supervisory engine (shown as 606) on the same or different chassis. This redundant supervisory engine has similar or identical functionality to the primary supervisory engine and hardware (shown as 604) to take over the network operation of the primary supervisory engine (604) if the primary supervisory engine (604) fails or becomes unavailable. Here, the exemplary system and method use a stateful switching operation to put the control plane of the debug network device into active mode and control the data plane of the target network device, while putting the control plane of the target network device into standby mode. The stateful switching operation relies on redundant hardware (e.g., 606) in the standby network device to take over the operation of the active network device (e.g., 604) to continue forwarding network traffic without losing sessions when the active network device becomes unavailable. The redundant hardware as debug network device 102a is used to generate an instrumented control plane 108 to debug, optimize, profil, or recover network devices in a live network.

[0124] A virtual transport layer, such as the control plane-data plane transport modules (e.g., 120 and 122, respectively) in each network device (e.g., 102, 104), provides transport operations for control plane and data plane updates between the active target network device (e.g., 104) and the virtualized standby debug network device (e.g., 102a). When the control plane 108 of debug network device 102 is in active mode, the control plane-data plane transport operations provide any control plane and data plane updates to the data plane 110 and the instrumented control plane 108.

[0125] In other words, the data plane-control plane transmission modules 120 and 122 implement a logical data plane interface (e.g., for PCI (vPCI), AXI (vAXI), or other bus interconnects) that (i) provides a device access layer or is capable of connecting to a device access layer interface, and (ii) provides communication between the data plane driver running on the instrumentation control plane 108 of the debug network device 102 and the data plane 110 of the target network device 104. The device access layer is the interface that sits directly above the hardware. This is the lowest layer in the ASIC / hardware driver. The data plane driver in the debug network device 102 (e.g., 102a) can be mapped to the underlying data plane device (or logical data plane interface endpoint), and the control plane 108 of the debug network device 102 can view and access the entire memory mapping of the data plane device (e.g., 114). The data plane-control plane transmission modules 120 and 122 can use technologies such as GRE, VxLAN, or similar mechanisms to implement tunneling (or socketing). Data plane-control plane transport modules 120 and 122 can encapsulate a given bus transaction for transmission over a given tunnel. Raw register / memory access operations are then sent and received via data plane-control plane transport modules 120 and 122. For a further description of the virtual transport layer operation, please refer to U.S. Patent Application No. 17 / 29559, filed December 2, 2020, the entire contents of which are incorporated herein by reference.

[0126] Example of debugging network devices using virtualized standby switches

[0127] Figure 7 An exemplary timing diagram 700 is shown illustrating a stateful handover operation between a target network device 104 and a virtualized remote / cloud standby debugging network device (shown as 102a) according to an illustrative embodiment. Similar operations can be performed on other embodiments of the debugging network device 102 as described herein.

[0128] exist Figure 7In the diagram, prior to the debugging / profiling operation (shown as 706), the target network device 104 is shown (702, 702a) as receiving (708a) data packets at port 504. These data packets are routed (708b) to another port (still shown as 504) by the data plane, which includes a forwarding engine 709 (shown as including an "ASIC / switching structure" 709), using resources 711 associated with the data plane (shown as via a "routing / forwarding table" 711 in operation 708c). Additionally, in Figure 7 In this example, for control plane packets with control plane updates (e.g., simple control plane updates), target network device 104 is shown (704) receiving control plane packets at port 504. These packets are routed (710a) via forwarding engine 709 and then (710b) via bus interconnect 514 (shown as "data plane interface" 514) to control plane 110 (e.g., including a host CPU). In this example, control plane 110 then updates (710c) data plane resource 711 by writing to data plane interface 514 (710d).

[0129] During debugging or profiling operations, such as Figure 7 As shown, the operation is initiated in step 712, where a virtualized debug command is received (712) by debug controller 713. In some embodiments, debug controller 713 is an application executing on the processing core or logic circuitry at the target network device 104. In other embodiments, debug controller 713 is an application executing on the processing core or logic circuitry at an external controller. In yet another embodiment, debug controller 713 is a cloud-based application executing in a cloud infrastructure. In some embodiments, debug controller 713 bootstraps (714) a virtualized standby debug network device 102a (or other debug network devices 102b, 102c) instantiated in a remote or cloud infrastructure (e.g., a remote or cloud server) to provide a redundant and instrumented control plane 108 to control plane 110. In some embodiments, debug controller 713 bootstraps various applications that load a system image (e.g., an instrumented version of a system image), a control plane application, or execute on the target network device 104 at debug network device 102a. In other embodiments, the debug controller 713 guides the instantiation of the control plane 108 using pre-configured instrumented system images and / or instrumented applications. In another embodiment, the debug controller 713 guides the creation of a control plane compute space to which field engineers and / or TACs can manually or subsequently install various debug or profiling software. In some embodiments, where instances of virtualized standby switches are pre-instantiated in a remote or cloud infrastructure, the debug controller 713 can guide the allocation of the pre-instantiated, virtualized standby switches to the active target network device 104.

[0130] Still refer to Figure 7 Preferably, for example, a stacking protocol (e.g., SVL or stacking cable) is used to stack the active target network device 104 (as a physical switch) and the virtualized standby debug network device 102a (or other debug network devices 102b, 102c). Figure 7 The diagram shows an active target network device 104 and a virtualized standby debug network device 102a being guided to join a stacking mode, wherein the active target network device 104 is set to active mode (see 716a) and the virtualized standby debug network device 102a is set to standby mode (716b).

[0131] Then, the active target network device 104 performs a batch synchronization operation on its control plane state and a subsequent incremental synchronization (718) to synchronize with the instrumented control plane 108 of the virtualized standby debug network device 102a.

[0132] During the initialization process (720), control plane-data plane transmission modules 120 and 122 may be initialized in their respective active target network device 104 and virtualized standby debug network device 102a (shown as 722a and 722b). In some embodiments, for example, as part of the initialization process, debug controller 713 pushes instructions or equivalent instructions for control plane-data plane transmission modules 120 and 122 to active target network device 104 and virtualized standby debug network device 102a. In other embodiments, the system image for active target network device 104 and virtualized standby debug network device 102a includes instructions for the control plane-data plane transmission modules, which debug controller 713 may initialize.

[0133] Once the control plane 110 of the target network device 104 and the control plane 108 of the debugging network device 102a are synchronized to have the same control plane state, and the control plane-data plane transmission modules 120 and 122 are instantiated, the initialization process (720) ends.

[0134] As control plane-data plane transmission modules 120 and 122 perform transport bus interconnect transactions between physical and virtualized network devices (102a, 104), the control plane 110 (but not the data plane) of the active target network device 104 can be switched from active mode to standby mode via a switching operation (shown as 723a, 723b) guided by the debug controller 713, and the control plane 108 of the debug network device 102a can be switched from standby mode to active mode. When the control plane 110 of the active target network device 104 is in standby mode, the control plane can be disabled (as shown in 724) or remain headless. Simultaneously, instruments on control plane 108 or instruments on the currently active debug network device 102a can be used to debug or profile the control plane, data plane, and / or network operations of the target network device 104.

[0135] During this period, the data plane 114 of the target network device 104 continues to forward the packets it receives, and the control plane 108 of the debugging network device 102a performs any updates associated with the control plane (e.g., updates to the data plane table and resources or the control plane) via the control plane-data plane transmission modules 120, 122. In some embodiments, when in standby mode, the control plane 110 of the active target network device 104 can be rebooted and / or upgraded to a new system image.

[0136] exist Figure 7 In the process, after handover operations 723a and 723b, the data plane 114 of the target network device 104 continues to supply data plane packets received from the network. As shown, when a packet arrives at port 504 (726a), the forwarding engine 709 uses resource 711 associated with the data plane (726c) to continue routing the packet (726b) to another port 504. For control plane updates (730), the control plane 108 of the debugging network device 102a initializes the process. Figure 7An example is shown where target network device 104 receives (728a) a control plane packet (e.g., a "punt" packet for OSPF updates) at port 504, which is routed (728b) via forwarding engine 719 to the driver of bus interconnect 514. However, instead of the control plane 110 of target network device 104 reading bus interconnect 514 (shown as data plane interface 514), the control plane-data plane transport module 122 (vPCI 122) of target network device 104 reads (728c) the bus interconnect at data plane interface 514 and transmits the read transaction as a message via a network or communication link (728d) to the control plane-data plane transport module 120 of the control plane 108 of debug network device 102a. Then, the control plane-data plane interface transmission module 120 (shown as vPCI 120) writes (728e) the transaction to the bus interconnect including the data plane interface 510 of the control plane 108 of the debug network device 102a, and then the control plane 108 reads (728f) the transaction to process (728g) the stacked control plane updates.

[0137] These data and control plane packets can be received from peer network devices and enterprise-grade network controllers, such as Cisco Digital Network Architecture (DNA) controllers, OpenStack controllers, or similar controllers.

[0138] When the control plane has a data plane update, the instrumented control plane 108 writes to the bus interconnect 510 (730a). The control plane-data plane interface transmission module 120 reads (730b) the transaction and transmits it as a message via the network (730c) to the control plane-data plane transmission module 122 of the control plane 110 of the target network device 104. Then, the control plane-data plane interface transmission module 122 writes (730d) the transaction to the bus interconnect 514 (as if it were written by the local control plane), and then writes (730e) the transaction to the addressed data plane resource.

[0139] It can be observed that even when the control plane 110 of the target network device 102a is in standby mode, the data plane 114 continues to maintain the same active mode. Furthermore, when the control plane 108 of the debugging network device (e.g., 102a or 102c) is in active mode, the debugging network device itself may not have a local data plane—the role of the debugging network device's control plane 108 is to temporarily maintain uninterrupted or near-uninterrupted control plane operation and provide space for performing debugging or profiling operations.

[0140] Furthermore, the debugging and profiling 750 can be performed multiple times as needed to obtain data logs of interest or to restore the target network device 104, with a time span that could be hours, days, or months. The debugging and profiling 750 may include monitoring any aspect of the various hardware and software components of the debugging network device 102 and the target network device 104, such as as it processes received data or control plane updates or any other system operation.

[0141] In some embodiments, a second switching operation (not shown) can be performed to restore the control plane 110 of the target network device 104 to active mode without any network interruption or interference. This feature allows the target network device 104 to be restored without interrupting the network. Among other advantages, this feature can also be used for making minor updates or changes, or for maintaining continuous operation in real-time control operations.

[0142] Virtualized debugging cloud infrastructure

[0143] Figure 8 A system 800, configured to perform debugging or profiling operations in a cloud server using virtualized stateful switching operations according to an illustrative embodiment, is shown.

[0144] As referenced above Figures 1-7 The debugging or profiling operations discussed generally involve instantiating a debugging network device 102 (e.g., a virtualized debugging network switch 102a) on a cloud or local / remote server (or debugging server) that has connectivity to physical network devices (e.g., active, stackable, or non-stackable switches). The target network device and the debugging network device can then be stacked using a stacking protocol, followed by a switchover using HA (e.g., ISSU and SSO mechanisms) to configure the control plane of the debugging network device that controls the data plane of the target network device.

[0145] In some embodiments, in addition to using HA (e.g., ISSU and SSO mechanisms), debugging network devices are also configured to execute both the control plane software and data plane drivers of the physical switch, the latter including, for example, the forwarding engine driver (FED), forwarding engine SDK (Software Development Kit), and ASIC driver. Figure 8In this system image, the control plane software / instructions, as well as HA and SSO instructions and the control plane for debugging network devices, can be stored and retrieved in a stackable switch image library (shown as 802) or an upgradeable switch image library (e.g., for non-stackable switches). In some embodiments, library 802 is stored on a remote or cloud server. An example of a remote or cloud image library system is the Cisco Software Image Management (SWIM) system. In other embodiments, the library is a computer-readable medium (e.g., DVD, CD, compact flash memory, and other persistent storage devices).

[0146] Methods for setting up a virtualized debugging switch

[0147] Figure 9 An exemplary sequence is shown for configuring an exemplary network device and using the device to perform debugging or profiling operations according to an illustrative embodiment.

[0148] exist Figure 9 The illustrated process 900 includes: first, instantiating (represented as "Creating S2" 902) a debugging network device 102 (represented as a virtualized debugging switch "S2" 102a) in a cloud, local, or remote machine. This debugging network device is used to debug or profil a target network device 104 (represented as a target switch "S1" 104a). In other embodiments, other computing devices can be used to host the virtualized standby switch, including portable computing devices, custom servers, or evaluation switches, as discussed herein.

[0149] The physical target switch "S1" 104a is shown as initially operating in standalone mode (904). Figure 9 In the image, target switch 104 is shown as executing switch image version "16.12.1".

[0150] Once the debugging operation is initialized, a virtual debug switch "S2" 102a is created. The creation of the virtual debug switch "S2" 102a involves instantiating a container or virtual machine (VM) in the cloud infrastructure. The container or VM includes the corresponding operating system, drivers, and one or more control plane applications that are executed on the target switch "S1" 104a. In some embodiments, instantiation is guided by a debug controller, which can be executed on the physical switch "S1" or a remote computing device.

[0151] After the virtualized switch “S2” 102a is instantiated, a debug controller or similar device can bootstrap the target switch “S1” 104a and the debug switch “S2” 102a into a stack via a stacking operation (908), in which the debug switch “S2” 102a is initially placed in standby mode and the target switch “S1” 104a is placed in active mode (shown as “S2 joins S1 in stacking mode” 908). During the stacking process, a bulk synchronization is performed (start shown as 906 and end shown as 914). The virtualized standby debug switch “S2” 102a is shown as executing a system image that is the same as or compatible with the target switch “S1” 104a, shown as switch image version “16.12.1”. The virtualized standby debug switch “S2” 102a also executes instrumentation or control plane applications from the system image. Batch synchronization (906) synchronizes the control plane states between the virtualized debug switch "S2" 102a and the target switch "S1" 104a, so that the control plane states of the two switches "S1" 104a and "S2" 102a are identical. Incremental synchronization can also be performed.

[0152] Once the control plane states are synchronized to the same state, the debug controller triggers a switchover (SSO) operation (917), and the debug switch "S2" 102a is directed to assume the active role, while the target switch "S1" 104a assumes the standby role. Once in the active role, the debug switch "S2" 102a uses a logical data plane interface (e.g., vPCI) to operate as the control plane for the target switch "S1" 104a (922), as discussed herein, which may be initiated in this order or earlier. The control plane 108 of the debug switch "S2" 102a performs a data plane update (923) on the data plane 114 of the target switch "S1" 104a using the logical data plane interface (represented as "Virtual Transport: DP Update" 923). Debugging and / or profiling operations 924 are then performed on the control plane 108 of the debug switch "S2" 102a. Data plane 114 continues to operate in slave mode, as shown for a duration of 926, in which the data plane is controlled by control plane 108 of debug switch “S2” 102a until the debug or profiling operation (924) is completed.

[0153] exist Figure 9In the example shown, once debugging is complete, the debugging controller initiates a second switching operation (928), and the control plane 108 of the debugging switch "S2" 102a is placed in standby mode, while the control plane 110 of the target switch "S1" 104a is placed in active mode (930). The target switch "S1" 104a is now repaired and can continue to operate normally (934), while the virtualized debugging switch "S2" 102a can be deleted (932).

[0154] In some embodiments, to configure the logical data plane interface, the data plane 110 of the target switch "S1" 104a can be programmed by the control plane 110 of the target switch "S1" 104a before the control plane 110 of the target switch "S1" 104a is designated to standby mode and under the guidance of the control plane 108 of the debugging network device "S2" 102a. The programming ensures resource association between the control planes and data planes of the target switch "S1" 104a and the debugging switch "S2" 102a and ensures that the addresses of these resources remain consistent. Without programming, even if the control plane states of the target switch "S1" 102a and the debugging switch "S2" 104a are identical, the data plane states may have different table indexes or addresses in the data plane because the order of operations at the control plane may not be preserved or executed in the same sequence. To optimize programming of the data plane device (e.g., 114) for the target switch “S1” 104a, the data plane device (e.g., 114) can be programmed only for resources for which such changes are expected or have already occurred.

[0155] In another embodiment, programming may involve generating an index translation table that translates addresses between: i) the old index associated with the data plane resources of the target switch "S1" 104a, and ii) the new index associated with the data plane resources of the debug switch "S2" 102a. This translation can improve network / switch availability because programming can be performed very quickly without overwriting the data plane resources of the target switch "S1" 104a. In fact, once the mapping between the old and new indexes is generated, "read" and "write" operations can traverse the translation table, and the correct index / address can be accessed.

[0156] Discussion and additional examples

[0157] As described herein, the exemplary systems and methods have numerous practical applications. While debugging is underway, the target network device can maintain considerable throughput while being served by the instrumented control plane, even if latency performance may vary. Network protocols typically have timeouts on the order of a few seconds, so additional latency does not necessarily impact protocol operation. Route updates and MAC learning may take longer, but their impact on data plane operation is similarly limited.

[0158] In fact, the exemplary systems and methods provide, for example, the ability to create an instrumented control plane on demand in the cloud or a remote server or other platform, and to stack that instrumented control plane with a non-instrumented production image on a physical target switch. This setup is equivalent to an HA system and will act like an HA system.

[0159] Furthermore, stacking and SSO are generally used for high availability of switches. Exemplary systems and methods can utilize traditional or existing stacking and SSO in debugging operations, such as for debugging and profiling real-time systems.

[0160] Furthermore, while debugging and profiling are typically invasive and can impact performance, the exemplary systems and methods can be used in customer networks for TAC / field support, for example, to assess common problems and difficulties that are not easily reproduced in the lab, without any noticeable impact on performance or latency.

[0161] Furthermore, stacking, SSO, and CNF (Cache and Clear) operations are used in conjunction with Fast Software Upgrade (FSU / xFSU) operations to reduce outages in non-redundant systems. Additionally, exemplary systems and methods can employ these operations to recover problematic switches in a near-uninterrupted manner in live customer networks. Once the debug network device is created on demand, it can be stacked with the physical target network device. The SSO operation described herein can be used to synchronize the state of the target network device to the debug network device. At this point, a switchover is performed, and the instrumented control plane of the debug network device is set to active mode. Subsequently, control traffic intended for the control plane of the target network device can be redirected (rather than mirrored) to the debug network device, for example, via tunneling. Most use cases do not require mirroring control plane traffic, although this can be done. This is similar to the operation of any HA system.

[0162] Data plane traffic does not need to be mirrored. Since the data plane in the physical target switch retains its functionality, traffic forwarding continues to be performed through that hardware. In use cases where debugging or profiling of the data plane (e.g., NPU and ASIC logic) is desired, the network device can be configured to run its data plane (e.g., via a simulator or emulator model). For data plane debugging of the target device, a subset of the data plane traffic can be mirrored to the cloud switch. Such data plane debugging may only require a small set of packets and can provide very detailed functional traces and logs of packets passing through the ASIC (e.g., using periodically accurate simulator models such as Doppler RTL emulators or CIMA).

[0163] Furthermore, the exemplary methods and systems facilitate debugging in a passive mode, where the physical target network device is unmodified / unaffected, and the debug machine, while still in a stacked configuration with the target network device, operates in parallel with it on separate hardware. In this mode, control plane traffic can be mirrored. This operation may involve HA-aware processes that are in hot standby.

[0164] Furthermore, the on-demand creation of standby switches in remote / cloud networks can be used for many other applications, such as profiling and troubleshooting problems in real-time customer switches by generating a standby, instrumented control plane without impacting performance. It can also be used for rapid troubleshooting sessions with cost-saving purposes. For example, in some real-time control applications that must maintain a real-time network, operations need to be performed on the control and data planes without disruption. Additionally, the exemplary methods and systems facilitate the recovery of faulty systems in customer networks with near-zero traffic interruption (which would typically require a reboot). Furthermore, the exemplary methods and systems can be automated and integrated with DNAC or other network management workflows. Moreover, the exemplary methods and systems can be used to test new images / versions in customer networks before full release. In fact, if a new image fails, the HA infrastructure provides protection and performs a non-disruptive switchover of the control plane on the physical switches. Even for non-redundant systems, this is a near-non-disruptive ISSU with rollback capability.

[0165] It should be understood that the various technologies and modules described herein (including the control plane-data plane interface transmission module) can be implemented by hardware components or software components, or, where appropriate, by a combination of both. Illustrated types of hardware components that can be used include: Field-Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application-Specific Standard Products (ASSPs), Systems-on-Chip (SOCs), Complex Programmable Logic Devices (CPLDs), etc. The methods and apparatuses of this disclosure, or certain aspects or portions thereof, can be embodied in the form of program code (i.e., instructions) on a tangible medium, such as a floppy disk, CD-ROM, hard disk drive, or any other machine-readable storage medium, wherein the machine becomes an apparatus for practicing the subject matter of this disclosure when the program code is loaded into and executed by it.

[0166] In summary, an exemplary method is disclosed that facilitates the on-demand creation of exemplary instrumented network devices in cloud infrastructure, remote servers, evaluation platforms, or custom test servers, forming a stack between the instrumented network devices (serving as debug network devices) and target network devices. A switching operation is then performed to switch from the control plane of the target network device to the control plane of the debug network device, while the data plane of the target network device continues to operate. Once the switch is complete, the control plane or the instrumentation (e.g., hardware or software) of the debug network device facilitates the debugging, optimization, profiling, and / or recovery of the physical network device, even in live networks.

[0167] In addition to physical hardware, implementations of network devices can be carried out, in whole or in part, in virtualized network hardware.

[0168] While exemplary implementations may involve utilizing various aspects of this disclosure within the context of one or more independent computer systems, this subject matter is not limited thereto and can be implemented in any computing environment, such as a networked or distributed computing environment. Furthermore, various aspects of this disclosure can be implemented in or across multiple processing chips or devices, and similarly, storage can be implemented across multiple devices.

[0169] While various embodiments of this disclosure have been described above, it should be understood that these embodiments are presented by way of example only and are not intended to be limiting. It will be apparent to those skilled in the art that various changes in form and detail can be made without departing from the spirit and scope of this disclosure. Therefore, the breadth and scope of this disclosure should not be limited to any of the exemplary embodiments described above, but should be defined solely by the appended claims and their equivalents.

Claims

1. A method for establishing a debugging cloud or remote infrastructure for network devices, the method comprising: At a remote or cloud computing device, a virtualized network device is instantiated using an operational image of a target network device, wherein the virtualized network device is configured by software instructions or hardware to perform one or more debugging processes not executed on the target network device, and wherein the target network device includes a first control plane and a data plane for receiving and forwarding network traffic in the network. The virtualized network device and the target network device are added to a stack configuration to synchronize the state of the first control plane of the target network device to the second control plane of the virtualized network device, wherein the first control plane of the target network device is initially executed in an active stack configuration, and wherein the second control plane of the virtualized network device is initially executed in a standby stack configuration; and A handover operation is triggered, wherein the handover operation switches the first control plane of the target network device from the active stack configuration to the standby stack configuration, and disconnects the first control plane from updating the data plane of the target network device; wherein the handover operation switches the second control plane of the virtualized network device from the standby stack configuration to the active stack configuration; and wherein the second control plane in the active stack configuration is connected to the data plane via a network connection and updates the data plane. The one or more debugging processes are operatively executed concurrently with the second control plane of the virtualized network device to evaluate at least one of the following: i) the second control plane, ii) the hardware or firmware configuration of the target network device, and iii) the network.

2. The method according to claim 1, wherein, The one or more debugging processes of the virtualized network device are used to profil the target network device or the network to recover the target network device from an error or invalid state associated with the data plane and / or the control plane.

3. The method according to claim 2, wherein, The one or more debugging processes are associated with at least one analysis tool.

4. The method according to claim 3, wherein, The step of instantiating the virtualized network device is performed dynamically, and the method further includes: After assessing and restoring the target network device, the virtualized network device is deleted from the remote or cloud computing device.

5. The method according to any one of claims 1 to 4, wherein, The one or more debugging processes are used to profile the target network device or the network to optimize the data plane configuration and / or control plane configuration of the target network device.

6. The method according to any one of claims 1 to 4, wherein, The one or more debugging processes are used to profile the target network device or the network in order to optimize network operation, network policy or network configuration.

7. The method according to any one of claims 1 to 4, wherein, When the second control plane of the target network device is being debugged by one or more debugging processes performed at the virtualized network device, the second control plane in the active stack configuration provides near-uninterrupted operation for the target network device.

8. The method according to any one of claims 1 to 4, wherein, The operational image includes non-instrumented production images.

9. The method according to any one of claims 1 to 4, wherein, The target network device includes a non-redundant switch.

10. The method according to any one of claims 1 to 4, wherein, The target network device includes redundant switches.

11. The method according to any one of claims 1 to 4, wherein, The virtualized network device and the target network device are added to the stack configuration through stacking technology.

12. The method according to any one of claims 1 to 4, wherein, The switching operation is performed through at least one of the following: High Availability (HA) operation, Non-Disruptive Service Software Upgrade (ISSU) operation, and Stateful Switching (SSO) operation.

13. The method according to any one of claims 1 to 4, wherein, The virtualized network device further includes an instrumented data plane, which includes a data plane component and additional instruments for accessing the data plane component, wherein the one or more debugging processes or the additional instruments are configured to send one or more debugging packets to the data plane component, and wherein the instruments are configured to provide logging and / or analysis for the one or more debugging packets.

14. The method according to any one of claims 1 to 4, wherein, The remote or cloud computing device includes: Cloud server; Remote server; The evaluation platform for the target network device; or A custom computing server that includes one or more debugging or evaluation subsystems.

15. The method according to any one of claims 1 to 4, wherein, The steps of instantiating the virtualized network device include: Retrieve the operational image of the target network device from the image server; and Use the retrieved operational image to orchestrate the virtualization environment, operating system, and environment.

16. The method according to any one of claims 1 to 4, further comprising: When the virtualized network device and the target network device are added, a tunnel connection or a direct connection is established between the virtualized network device and the target network device, wherein the tunnel connection or the direct connection is used as a network connection for the virtualized network device to update the data plane of the target network device.

17. A system for establishing a debugging cloud or remote infrastructure for network devices, the system comprising: processor; as well as A memory having computer-readable instructions, wherein execution of the computer-readable instructions causes the processor to: At a remote or cloud computing device, a virtualized network device is instantiated using an operational image of a target network device, wherein the virtualized network device is configured by software instructions or hardware to perform one or more debugging processes not executed on the target network device, and wherein the target network device includes a first control plane and a data plane for receiving and forwarding network traffic in the network. The virtualized network device and the target network device are added to a stack configuration to synchronize the state of the first control plane of the target network device to the second control plane of the virtualized network device, wherein the first control plane of the target network device is initially executed in an active stack configuration, and wherein the second control plane of the virtualized network device is initially executed in a standby stack configuration; and A handover operation is triggered, wherein the first control plane of the target network device is switched from the active stack configuration to the standby stack configuration and disconnected, no longer updating the data plane of the target network device; and wherein the second control plane of the virtualized network device is switched from the standby stack configuration to the active stack configuration and connected via a network connection to update the data plane of the target network device. The one or more debugging processes are operatively executed concurrently with the second control plane of the virtualized network device to evaluate at least one of the following: i) the second control plane, ii) the hardware or firmware configuration of the target network device, and iii) the network.

18. The system according to claim 17, wherein, When the instruction is executed by the processor, it also causes the processor to: Perform one or more debugging procedures to recover the target network device from detected errors or invalid states associated with the target network device's data plane and / or control plane.

19. The system according to claim 17 or 18, wherein, The system is configured to: A set of bus interconnect transactions is read from the bus interconnect of the target network device, and this set of bus interconnect transactions is sent as a set of data plane transaction messages to the virtualized network device via the network interface. The virtualized network device is configured to use this set of data plane transaction messages to write the set of bus interconnect transactions to the bus interconnect or host processor of the virtualized network device to update the control plane state maintained by the virtualized network device. Based on a second set of data plane transaction messages received from the virtualized network device via the network interface, a second set of bus interconnect transactions is written to the bus interconnect of the target network device, wherein the second set of bus interconnect transactions updates a portion of a plurality of tables of the target network device associated with the data plane.

20. A computer-readable medium having instructions stored thereon, wherein, The instructions are executed by the processor, causing the processor to: At a remote or cloud computing device, a virtualized network device is instantiated using an operational image of a target network device, wherein the virtualized network device is configured by software instructions or hardware to perform one or more debugging processes not executed on the target network device, and wherein the target network device includes a first control plane and a data plane for receiving and forwarding network traffic in the network. The virtualized network device and the target network device are added to a stack configuration to synchronize the state of the first control plane of the target network device to the second control plane of the virtualized network device, wherein the first control plane of the target network device is initially executed in an active stack configuration, and wherein the second control plane of the virtualized network device is initially executed in a standby stack configuration; and A handover operation is triggered, wherein the first control plane of the target network device is switched from the active stack configuration to the standby stack configuration and disconnected, no longer updating the data plane of the target network device; and wherein the second control plane of the virtualized network device is switched from the standby stack configuration to the active stack configuration and connected via a network connection to update the data plane of the target network device. The one or more debugging processes are operatively executed concurrently with the second control plane of the virtualized network device to evaluate at least one of the following: i) the second control plane, ii) the hardware or firmware configuration of the target network device, and iii) the network.

21. An apparatus for establishing a debugging cloud or remote infrastructure for network devices, the apparatus comprising: Components for instantiating a virtualized network device at a remote or cloud computing device using an operational image of a target network device, wherein the virtualized network device is configured by software instructions or hardware to perform one or more debugging processes not performed on the target network device, and wherein the target network device includes a first control plane and a data plane for receiving and forwarding network traffic in the network. Components for joining the virtualized network device and the target network device into a stack configuration to synchronize the state of the first control plane of the target network device to the second control plane of the virtualized network device, wherein the first control plane of the target network device is initially executed in an active stack configuration, and wherein the second control plane of the virtualized network device is initially executed in a standby stack configuration; and A component for triggering a handover operation, wherein the handover operation switches the first control plane of the target network device from the active stack configuration to the standby stack configuration, and disconnects the first control plane from updating the data plane of the target network device; wherein the handover operation switches the second control plane of the virtualized network device from the standby stack configuration to the active stack configuration; and wherein the second control plane in the active stack configuration is connected to the data plane via a network connection and updates the data plane. The one or more debugging processes are operatively executed concurrently with the second control plane of the virtualized network device to evaluate at least one of the following: i) the second control plane, ii) the hardware or firmware configuration of the target network device, and iii) the network.

22. The apparatus of claim 21, further comprising: Components for implementing the method according to any one of claims 2 to 16.

23. A computer program product comprising instructions that, when executed by a computer, cause the computer to perform the steps of the method according to any one of claims 1 to 16.

Images