Systems, apparatus and methods for determining data for root cause analysis

Abstract

A system for determining data for root cause analysis for a computer network comprises a controller configured to receive current state data for a network device, compare at least some of the current state data with stored reference data for the network device, generate difference data in dependence on whether one or more differences are present between the current state data and the stored reference data, determine one or more candidate network devices for testing in dependence on one or more properties for the network device, responsive to one or more differences being present between the current state data and the stored reference data, and perform one or more tests for one or more of the candidate network devices to obtain test data indicative of one or more properties associated with one or more of the candidate network devices.

Description

[0001] SYSTEMS, APPARATUS AND METHODS FOR DETERMINING DATA FOR ROOT CAUSE

[0002] ANALYSIS

[0003] FIELD OF THE INVENTION

[0004] This present invention relates to systems, apparatus and methods for determining data for root cause analysis for a computer network.

[0005] It is known for network error monitoring systems to operate on a computer network and be operable to detect an error occurring and react to the error. Reacting to detection of an error may involve generating a support ticket comprising an error log with details of the observed error and sending the ticket to a technical support system. Once reported to the technical support system, a technical support engineer reads the details of the ticket, noting the reported data about the error, and then takes actions to rectify the error. Usually, the details provided about the error at the time the error is established by the network error monitoring system, provide only insight about the nature of the error and often the underlying cause of the error remains unknown. In a networked computing environment, an error may be caused by factors such as incompatible settings being applied to networking equipment providing the network functionality and / or to the devices connected to the network for communicating over the network, or issues with the networking hardware, firmware, software or infrastructure. More generally, errors for computing networks can have many possible causes and which can often be complex to diagnose.

[0006] It is an aim of the present invention to reduce or overcome one or more of the problems associated with prior art methods and associated systems.

[0007] SUMMARY OF THE INVENTION

[0008] According to an aspect of the invention, a system for determining data for root cause analysis for a computer network comprises a controller configured to: receive current state data for a network device; compare at least some of the current state data with stored reference data for the network device; generate difference data in dependence on whether one or more differences are present between the current state data and the stored reference data; determine one or more candidate network devices for testing in dependence on one or more properties for the network device, responsive to one or more differences being present between the current state data and the stored reference data; and perform one or more tests for one or more of the candidate network devices to obtain test data indicative of one or more properties associated with one or more of the candidate network devices.

[0009] The controller may determine one or more of the candidate network devices in dependence on at least one of a device type and a device role associated with the network device.

[0010] The controller may classify network devices based on at least one of a network protocol for a network device, a hostname for the network device, and an internet protocol “IP” address for the network device, and wherein one or more of the candidate network devices may have a same classification as the network device.

[0011] The difference data may be indicative of one or more differences between the current state data and the stored reference data, and the controller may be configured to determine one or more of the candidate network devices in dependence on one or more of the differences.

[0012] The controller may perform one or more diagnostic tests for a candidate network device to obtain diagnostic testing results for the candidate network device.

[0013] The controller may access stored diagnostic testing data associated with a candidate network device and use the stored diagnostic testing data to run one or more diagnostic tests associated with the candidate network device.

[0014] The controller may perform a difference test for a candidate network device and generate difference data for the candidate network device in dependence on whether at least one difference is present between current state data for the candidate network device and stored reference data for the candidate network device.

[0015] The controller may perform a status query test for a candidate network device to obtain status data for the candidate network device.

[0016] The controller may generate a support ticket comprising the test data for one or more of the candidate network devices.

[0017] The support ticket may further comprise the difference data generated forthe network device in which the difference data generated for the network device is indicative of one or more differences between the current state data and the stored reference data. The controller may receive a report indicative of at least one of potential error and a change in a current state from the network device, and request the current state data from the network device in response to the report.

[0018] The controller may request the current state data to include one or more attribute types corresponding attribute types included in the stored reference data for the network device.

[0019] The controller may receive a plurality of reports indicative of potential errors and / or changes in current states and analyse at least some of the plurality of reports to determine an order of priority for at least some of the plurality of reports.

[0020] The controller may analyse at least some of the plurality of reports to detect one or more keywords corresponding to names and / or types of predefined errors.

[0021] The controller may determine a degree of severity for a report in dependence one or more keywords corresponding to names and / or types of predefined errors and order at least some of the plurality of reports according to the degree of severity.

[0022] The controller may capture snapshots for network devices to obtain stored reference data for one or more of the network devices.

[0023] The system may further comprise a ticketing system that may receive a support ticket from the controller, in which the support ticket comprises at least some of the test data.

[0024] At least one of the controller and the ticketing system may generate one or more suggestions for root cause analysis in dependence upon the support ticket.

[0025] At least one of the controller and the ticketing system may access at least one of a database and an artificial intelligence ‘Al’ model to generate one or more of the suggestions for root cause analysis in dependence upon the support ticket.

[0026] According to another aspect of the invention, a computer-implemented method comprises: receiving current state data for a network device; comparing at least some of the current state data with stored reference data for the network device; generating difference data in dependence on whether one or more differences are present between the current state data and the stored reference data; determining one or more candidate network devices for testing in dependence on one or more properties for the network device, responsive to one or more differences being present between the current state data and the stored reference data; and performing one or more tests for one or more of the candidate network devices to obtain test data indicative of one or more properties associated with one or more of the candidate network devices.

[0027] Various aspects and features of the present disclosure are defined in the appended claims and within the text of the accompanying description. Example embodiments include at least a system, apparatus, a method, a computer program and a machine-readable, non-transitory storage medium which stores such a computer program.

[0028] BRIEF DESCRIPTION OF THE FIGURES

[0029] In orderthatthe present disclosure may be more readily understood, preferable embodiments thereof will now be described, by way of example only, with reference to the accompanying drawings, in which

[0030] Figure 1 is a schematic diagram illustrating a system in accordance with embodiments of the disclosure;

[0031] Figures 2 and 3 are schematic flowcharts illustrating methods; and

[0032] Figures 4 and 5 are schematic diagrams illustrating flow of data in a system.

[0033] DETAILED DESCRIPTION OF THE DISCLOSURE

[0034] The techniques of the present disclosure relate to determining data for root cause analysis for computer networks. Embodiments of the disclosure provide systems, apparatus and methods for determining data for root cause analysis for computer networks. Root cause analysis is generally the process for identifying root causes for errors, issues and other problems on a computer network.

[0035] References to a computer network refer to a system comprising a plurality of network devices that are connected to each other directly and / or indirectly via wired and / or wireless communications for communicating data electronically. The techniques of the present disclosure are applicable to computer networks comprising any suitable number of network devices. References to network devices, refer to devices including network functionality, including for example data transmission, secure network connectivity, data traffic flow optimisation, access control and threat detection or prevention, and device management and configuration. A non-exhaustive list of examples of network devices includes switches, routers, bridges, repeaters, gateways, firewalls, wireless access points, endpoint devices, user devices and so on. Referring now to Figure 1 , a system 100 is shown comprising a controller 110 and three network devices 120-1 , 120-2 and 120-3. Figure 1 provides an example of a system in which the controller 110 can act as a central controller capable of communicating with each of the network devices 120- 1 , 120-2 120-3. Figure 1 provides an example in which the controller 110 can receive data from any of the network devices 120-1 , 120-2 120-3 and perform operations to determine data for root cause analysis for the computer network comprising the network devices 120-1 , 120-2 120-3. In some examples, the controller 110 and the network devices 120-1 , 120-2, 120-3 may be part of a same Local Area Network (LAN), or part of two or more connected LANs. In some examples, the controller 110 and the network devices 120-1 , 120-2, 120-3 may be connected to each other in a same geographic area such as a building, office, school, university campus and so on. In some examples, each of the network devices 120-1 , 120-2, 120-3 may be part of a same LAN, and the controller 110 may be implemented as a remote server (e.g. a cloud-based server). The controller may be implemented as one or more data processing apparatuses. That is to say that the functionality of the controller may be implemented by a respective data processing apparatus or in a distributed manner using a combination of data processing apparatuses.

[0036] Whilst Figure 1 shows an example system 100 comprising the controller 110 and three network devices, embodiments of the disclosure provide systems comprising any suitable number of respective network devices. In some examples, a system may comprise a potentially large number N of respective network devices, where N may be any suitable value potentially of the order of tens or even hundreds. Therefore, more generally, the system 100 may comprise the controller 110 and a plurality of network devices.

[0037] Still referring to Figure 1 , any suitable wireless communication protocol may be used for communication between the controller 110 and the network devices. Suitable wireless communication protocols are generally known and are not discussed in detail. In some cases, a same wireless communication protocol may be used for communication between the controller 110 and each of the network devices. In other cases, two or more different wireless communication protocols may be used for communication between the controller 110 and the network devices. Examples of suitable communication protocols are discussed in more detail later.

[0038] In Figure 1 , the network devices 120-1 , 120-2, 120-3 are each capable of communicating at least one of a report and / or current state data. The controller 110 is operable to receive a report or current state data for the techniques to be discussed below. The controller 110 has access to stored reference data for one or more network devices. Stored reference data for a network device may be obtained by capturing snapshots of the computer network, or parts of the computer network. Snapshots may be captured for all or parts of the computer network at different intervals. In some embodiments of the disclosure, the controller is operable to capture snapshots of the network configuration or aspects of the configuration at intervals to thereby obtain stored reference data for one or more network devices. The snapshots may be stored at regular periodic intervals (e.g., every 10 minutes, hourly, every two hours, twice per day, or daily).

[0039] In particular, snapshots of network devices and their states (e.g. configuration settings, and / or settings associated with their network ports, and / or health status etc. ), can be obtained and stored for situations when a device is observed to be correctly configured and operating correctly. This may happen at the time of system set up, or shortly after, for example. In addition, alongside that stored ‘safe’ state data, a snapshot of the entire network (or at least a portion of the network), providing one or more of device details, device states, device configurations, and port configurations, may be stored as a known “safe” stage of the network in relation to that device, at least. In some examples, stored reference data for a network device may be manually specified by a user (e.g. admin user) or obtained from one or more other resources providing information for normal operation of network devices.

[0040] Hence more generally, the controller 110 is operable to access stored reference data for one or more network devices. In some embodiments of the disclosure, the controller 110 may comprise storage circuitry (e.g. any suitable memory such as volatile memory, non-volatile memory or a combination thereof) to store reference data for one or more network devices. In some embodiments of the disclosure, the controller 110 may communicate with another device (e.g. a storage device) to access the stored reference data. For example, one or more connected datastores may be accessed by the controller 110. The controller 110 comprises processing circuitry to perform any of the operations discussed herein. The processing circuitry may comprise any of a CPU, GPU, DPU or any combination thereof, for example, which may be part of a same processing apparatus or a combination of processing apparatuses.

[0041] In the techniques of the present disclosure, the controller 110 may receive a report for a network device and perform processing in response to the report to determine data for root cause analysis in relation to that report. A report may be indicative of any of a potential error and / or a change in a current state (e.g. change in configuration) for a network device. Moreover, a potential error and / or a change in a current state can be detected by a network device and reported to the controller. Hence, unless stated otherwise, references herein to a report refer to a report indicative of at least one of a detected potential error and / or a detected state change (e.g. detected configuration change). Generally, a report may indicate an error without substantive detail as to why the error is likely to have occurred or how to resolve the error. Therefore, traditionally, provision of such reports may be of little assistance for root cause analysis. In the techniques of the present disclosure the controller can receive a report and determine data for enriching the report, thereby assisting in root cause analysis. Processing by the controller 110 for determining data for root cause analysis may be triggered by any of: i) the controller receiving a report; and ii) the controller receiving current state data and performing an analysis for the current state data. For example, network devices may generate reports and communicate reports to the controller, and the controller may carry out further investigation by requesting current state data for a network device and comparing the current state data with stored reference data for the network device to identify deviation from the stored reference data. Alternatively or in addition, network devices may communicate current state data to the controller (i.e. without sending reports) and the controller may compare the current state data with stored reference data to identify deviation from the stored reference data. In response to deviation from the stored reference data, the controller determines one or more candidate devices for testing so that targeted testing of candidate devices can be performed to obtain test data useful for root cause analysis.

[0042] More generally, current state data for a network device can be received by the controller 110 as a result of a request from the controller (e.g. automatically requesting on a periodic basis and / or in response to a received report from a network device) and / or as a result of one or more network devices automatically communicating current state data (e.g. automatically sending on a periodic basis). In some examples, API calls, Secure Shell (SSH) commands, command-line interface (CLI) commands and / or streaming telemetry may be used for requesting and / or sending current state data. API calls between the controller and network devices may be used to request and / or send current state data. Streaming telemetry may be triggered for one or more network devices for communication of current state data to the controller. In some cases, the controller may connect to network devices to directly perform CLI commands and retrieve the outputs. Hence, more generally the controller 110 may receive current state data using one or more of API calls, SSH commands, CLI commands, and / or streaming telemetry. More generally, current state data can be provided to the controller which is indicative of current properties for a network device and may correspond to real-time data (e.g. live state data). Examples of attributes included in live state data may include any suitable attribute that can be queried via the above techniques. In some examples, attributes include current configuration settings for network devices. Hence, in some examples the controller may receive current state data comprising current configuration data for one or more current configurations associated with a network device.

[0043] Figures 2 and 3 are schematic flowcharts illustrating methods performed by the controller 1 10. The method 200 is triggered by the controller receiving a report, whereas the method 300 is triggered by the controller receiving current state data and proceeding to analyse the current state data.

[0044] Referring to Figure 2, the method 200 comprises: receiving (at a step 210) a report from a network device; receiving (at a step 220) current state data for the network device; comparing (at a step 230) at least some of the current state data with stored reference data for the network device; generating (at a step 240) difference data in dependence on whether one or more differences are present between the current state data and the stored reference data; determining (at a step 250) one or more candidate network devices fortesting in dependence on one or more properties for the network device, responsive to one or more differences being present between the current state data and the stored reference data; and performing (at a step 260) one or more tests for one or more of the candidate network devices to obtain test data indicative of one or more properties associated with one or more of the candidate network devices.

[0045] The method 200 can be performed by the controller 110 for reports received from various network devices. For example, the controller may receive a report from the network device 120-1 and in response request current state data from the network device 120-1. The current state data may provide a greater insight into the current operation of the network device and may be more useful for assisting in identifying a cause of the error. Current state data can thus be pulled by the controller and / or pushed by network devices for use in determining one or more candidate network devices to be subjected to testing to thereby obtain test data for enriching the information already included in the report and assisting in root cause analysis.

[0046] A report may be indicative of a potential error such as one or more of a network error and a hardware error. In some cases, a report may be indicative of a change in state for a network device. For example, one or more of a configuration change, state change, hardware change and interface change may be indicated by the report. More generally, a report may include information detected by a network device. In some cases, reports may include error messages indicative of any of a name associated with an error, a type associated with an error, and a time associated with an error.

[0047] In some embodiments of the disclosure, the controller 110 may perform the method 200 in respect of each received report. That is, in response to receiving a report, the controller may proceed to automatically request current state data from the network device and proceed according to the steps discussed above for enrichment thereof. In some embodiments of the disclosure, the controller may firstly analyse a received report to determine whether to proceed with the method 200 in respect of that report.

[0048] In some embodiments of the disclosure, the controller 110 may analyse a report using keyword detection techniques. In particular, the controller may analyse a report to detect one or more keywords corresponding to names (e.g. identifiers) and / or types of predefined errors. In response to detection of at least one keyword, the controller may determine to proceed with performing the method 200 in respect of the report. In response to absence of detection of at least one keyword, the controller may determine not to proceed with performing the method 200 in respect of the report, or the controller may assign the report to a queue with a lowest priority rating so that processing for enrichment for that report can be performed conditional on whether other higher priority reports require processing.

[0049] In some embodiments, keyword detection techniques may use keywords associated with one or more predetermined network protocols. The controller may use one or more predetermined keywords for one or more predetermined network protocols such as one or more of the Border Gateway Protocol (BGP), the Bidirectional Forwarding Protocol (BFD), the Open Shortest Path First protocol (OSPF), the Spanning Tree Protocol (STP), the Intermediate System to Intermediate System protocol (IS-IS); and the Locator / ldentifier Separation Protocol (LISP). For example, keywords associated with names and / or types of predefined errors for those protocols may be used. In some examples, the controller may store or otherwise access stored keyword information comprising a plurality of predetermined keywords association with names and / or types of errors. The stored keyword information may be updated (e.g. by an admin user) as needed and can be used for detecting presence of predetermined keywords in received reports.

[0050] In some embodiments of the disclosure, the controller 110 may receive a plurality of reports and analyse at least some of the plurality of reports to determine an order of priority for at least some of the plurality of reports. Reports may be received from various network devices and some reports can potentially be of a higher severity than others. Accordingly, the controller can analyse reports and determine an order of priority for at least some of the reports so that the reports can be processed according to the priority ordering. For example, the above-mentioned keyword detection techniques may be used to detect keywords for the reports and determine an order of priority so that reports of higher potential severity are preferentially handled before reports of lower potential severity. The controller may receive reports on an on-going basis, and whilst the controller is determining data for root cause analysis in respect of a current report (i.e. performing method 200 in respect of a current report), other reports may be queued by the controller and the received reports can be queued according to an order of priority. In other examples, the controller may potentially queue received reports and process received reports according to the order in which they are received, and optionally analysis of a report (e.g. keyword detection) may be used to determine whether to perform the method 200 in respect of a queued report.

[0051] Hence more generally, in some embodiments of the disclosure the controller 110 may determine a degree of severity for a received report in dependence one or more keywords corresponding to names and / or types of predefined errors and order at least some of a plurality of received reports according to the degree of severity. Accordingly, in some cases the controller may store a queued sequence of reports to be processed, and the queued sequence may be ordered according to a determined priority order so that higher severity reports are ordered before lower severity reports. The above discussion with respect to Figure 2 refers to arrangements for determining data for enriching a received report for assisting in root cause analysis. Alternatively or in addition to communicating reports, one or more network devices may automatically communicate current state data to the controller 110. Whereas reporting generally requires detection by a network device of a potential error and / or state change, the controller can receive current state data and perform comparison with stored reference data for checking deviations from normal behaviour and may identify errors, issues or other problems that can potentially go un-reported. Respective network devices may periodically communicate current state data for checking by the controller, and / or the controller may periodically request current state data for checking by the controller.

[0052] Referring to the example of Figure 1 , in some examples each of the network devices 120-1 , 120-2 and 120-3 may automatically communicate current state data for that network device. The network devices may periodically communicate current state data according to a given time period, and the given time period may be the same or different among the network devices 120-1 , 120-2 and 120-3.

[0053] In some examples, one of more of the network devices may be considered a core network device and other or more of the network devices may be considered non-core network devices. For example, the network device 120-1 may be a core network device, whereas network device 120-2 may be a non-core network device. A shorter time period may be used for communication of current state data for the network device 120-1 than for communication of current state data for the network device 120-2 so that checking can be performed for the network device 120-1 at shorter time intervals. More generally, potential errors and other issues for core network devices can be potentially more severe for network performance than for non-core network devices and checking of current state data with stored reference data can be performed more regularly for core network devices.

[0054] Network devices may be classified based on one or more properties for those network devices and each network device may communicate current state data at regular intervals based on a classification for that network device. Moreover, network devices having a first classification may communicate current state data according to a first time interval, and network devices having a second classification may communicate current state data according to a second time interval. In some cases, the controller may periodically receive current state data from one or more network devices designated as priority network devices. A user may specify for a given system one or more priority network devices for which current state data is to be periodically provided to the controller for error presence determining. In this way, one or more network devices considered to be of higher importance for the functioning of a network can be selected for current state data comparison techniques. Hence in some cases only a subset of the network devices for the computer network may periodically communicate current state data. Optionally, the network devices 120-1 , 120-2 and 120-3 may have report generating functionality. In some examples, each of the network devices 120- 1 , 120-2 and 120-3 may also have report generating functionality. In other examples, only some (i.e. a subset) of the network devices 120-1 , 120-2 and 120-3 may have report generating functionality.

[0055] Referring to Figure 3, the method 300 comprises: receiving (at a step 310) current state data for a network device; comparing (at a step 320) at least some of the current state data with stored reference data for the network device; generating (at a step 330) difference data in dependence on whether one or more differences are present between the current state data and the stored reference data; determining (at a step 340) one or more candidate network devices fortesting in dependence on one or more properties for the network device, responsive to one or more differences being present between the current state data and the stored reference data; and performing (at a step 350) one or more tests for one or more of the candidate network devices to obtain test data indicative of one or more properties associated with one or more of the candidate network devices.

[0056] The controller 110 may receive current state data for a given network device at the step 310 which may be sent by the given network device in response to a request from the controller for current state data and / or based on the given network device automatically sending the current state data. The controller 110 may receive current state data from any of the network devices 120-1 , 120-2 and 120- 3. Whereas the method 200 is triggered by receiving a report, the method 300 is triggered by receiving current state data which may be automatically sent by a network device or automatically requested by the controller. More generally, rather than relying on network devices to detect errors and / or changes and communicate reports to the controller, the controller 110 (also referred to as a controller apparatus) may receive current state data for a network device without having firstly received a report.

[0057] Figures 2 and 3 represent methods performed by the controller for determining data for root cause analysis for a computer network comprising a plurality of connected network devices. It will be appreciated that the methods may be suitably combined, or the methods may be performed as alternatives. For example, the controller may receive a report at a first time and perform processing operations to determine data in response to the received report. The controller may receive current state data at a second time and perform processing operations to determine data in response to the received current state data. More generally, the controller may be capable of using any of reports and current state data to trigger processing for determining data to assist root cause analysis.

[0058] In some examples, the controller may have different modes of operation. In a first mode of operation, the controller may receive one or more reports and determine data according to the method 200. In a second mode of operation, the controller may receive current state data for one or more network devices and determine data according to the method 300. The controller may thus have the first mode and second mode of operation and may switch between the first mode and second mode of operation based on a user input. For example, a user (e.g. admin user) may select a mode of operation. The controller may be configured to receive just reports indicative of errors. In other examples, the controller may be configured to receive just current state data for network devices.

[0059] More generally, in embodiments of the disclosure a system for determining data for root cause analysis for a computer network comprises the controller 1 10 configured to: receive current state data for a network device; compare at least some of the current state data with stored reference data for the network device; generate difference data in dependence on whether one or more differences are present between the current state data and the stored reference data; determine one or more candidate network devices for testing in dependence on one or more properties for the network device, responsive to one or more differences being present between the current state data and the stored reference data; and perform one or more tests for one or more of the candidate network devices to obtain test data indicative of one or more properties associated with one or more of the candidate network devices.

[0060] The stored reference data is considered representative of normal (i.e. error free) behaviour for one or more network devices. Comparison of current state data with stored reference data identifies whether at least one difference is present between the current state data and the stored reference data. At least one difference between the current and stored state data can be indicative of a potential error, issue or other similar problem. Therefore, in response to generating the difference data, the controller proceeds to determine one or more candidate network devices for testing and perform one or more tests to obtain resulting test data which can be of assistance for root cause analysis of the potential error, issue or other similar problem. A set of one or more network devices on the network can be determined as being candidates for testing, in which the set of one or more network devices includes one or more network devices other than the given network device from which the initial report (or initial state data) is received.

[0061] The controller thus obtains test data related to one or more of the candidate network devices which can supplement information already included in a report. The test data can be useful for indicating to a user (e.g. support engineer) a likely cause for an error or issue, or at least useful for indicating to the user one or more steps for determining the likely cause. More generally, test data can be obtained for specific network devices on the network and this can assist in improving a likelihood of a user successfully identifying a root cause and / or reducing time for a user to identify the root cause. For example, provision of the test data related to one or more of the candidate network devices may potentially reduce time between an error or issue being reported and the error being corrected thus resulting in improved system performance. System downtime during which one or more network devices and / or one or more capabilities of one or more network devices are unavailable may be reduced compared to traditional techniques. Furthermore, traditional techniques in which users rely simply on a report from a network device can result in some errors going unresolved. For networks comprising potentially large numbers of network devices it can be particularly challenging to identify root causes of errors. By determining one or more candidate network devices and performing one or more tests, as discussed above, this can improve a likelihood of identifying root causes of potentially complex errors.

[0062] The controller may receive files from network devices comprising state data. A file may comprise state data indicative of a plurality of different attributes a network device. In some examples, the file may be obtained by capturing outputs of various CLI commands, SSH commands, API commands and / or streaming telemetry. Simple Network Management Protocol (SNMP), Network Configuration Protocol (NETCONF), gNMI and / or gRPC are other suitable examples that may be used for this purpose. The controller may access a stored file for the network device which may similarly comprise a plurality of different reference attributes for the network device. The controller can be operable to compare the two files. Hence, in some examples the steps 230 and 320 may comprise comparing a received file with a stored reference file.

[0063] In some examples, each attribute in a received file may be compared with a corresponding attribute in a stored reference fie. In other examples, certain attributes may be included in the files which may be omitted from the comparison such that differences for certain attributes may be ignored. For example, attributes such as a BGP neighbour count; BGP table version, routing table version, memory usage, total memory and / or last reset time may not be compared with reference data because it can be expected that discrepancies for these attributes are not associated with errors.

[0064] In some examples, the current state data may comprise one or more of: a current BGP neighbour state for one or more BGP neighbours; and / or a number of learned prefixes from a BGP neighbour. A current BGP neighbour state may correspond to one of idle, connect, active, open sent, open confirm, and established. Differences (or put differently changes) for one or more of these attributes may be associated with errors. Accordingly, in response to comparison of the current state data and the stored reference data indicating a change in an attribute (e.g. a changing in a BGP neighbour state and / or a change in a number of learned prefixes from a BGP neighbour) then the controller generates difference data indicative of at least presence of a difference and proceeds to determine one or more candidate network devices for testing.

[0065] The controller may thus compare two files and generate difference data depending on whether one or more differences are present between the two files. In some examples, the controller may be operable to generate a difference file in dependence upon the comparison. The presence / absence of the difference file may potentially indicate presence / absence of at least one difference (e.g. in the case of no differences the difference file may not be generated or may be empty). Optionally, the difference file may be generated to include details of one or more of the differences. If the difference file indicates presence of at least one difference, the controller proceeds with determining one more candidate network devices for testing. However, if the difference file indicates that there is no difference (e.g. an empty difference file or potentially no difference file being generated at all), the controller ends the processing and may proceed to handle a next request. Hence, referring to figures 2 and 3, the methods may perform the comparing operation (steps 230 and 320 in Figures 2 and 3, respectively) and end in the case of in the case of normal behaviour.

[0066] In some embodiments of the disclosure, the controller may request the current state data from a given network device, and the request may specify one or more attribute types to be included in the sent current state data. In particular, the request may specify one or more attribute types that correspond to attribute types included in the stored reference data for the network device. In this way, data can be requested which enables comparison with the stored reference data. Moreover, the request can specify a plurality of different attribute types for which comparison can be performed and this may potentially avoid wasteful use of processing resources and communication bandwidth that may arise from sending of redundant information.

[0067] The current state data may be indicative of one or more configuration settings and / or statuses such as one or more of: an interface connectivity (e.g. connection status of one or more network ports); a routing protocol (e.g. routing protocol neighbours and routes); a VLAN status (e.g. VLAN forwarding); a security enforcement; and a device health status or platform health status. In some examples, a border gateway routing protocol (BGP) may be used and the current state data may be indicative of at last one of: a current BGP neighbour state for one or more BGP neighbours; and / or a number of learned prefixes from a BGP neighbour.

[0068] In some examples, a received report may comprise an error message indicative of an error associated with a BGP neighbour. For example, a first network device (e.g. first router device) may communicate a report comprising an error message indicative of an error associated with a BGP neighbour network device (e.g. a second router device). In response to such a report, the controller may request current state data for one or both of the first network device and the second network device and perform the method steps 220-260 for both the first current state data for the first network device and the second current state data for the second network device.

[0069] In both of the methods 200 and 300, the controller 110 determines one or more candidate network devices to be subjected to one or more tests for obtaining test data that can be used for root cause analysis. Candidate network devices for testing are determined in dependence on at least one or more properties for a network device for which the current state data results in generating difference data. Moreover, for current state data for a given network device resulting in generating difference data, the controller can be operable to determine at least one candidate network device in dependence on at least one property for that given network device. In some cases, the controller may determine a plurality of candidate devices. Generally, for a computing network comprising multiple network devices, the controller can determine a set of one or more candidate network devices corresponding to a subset of the multiple network devices. Therefore, a specific set of candidate network devices can be identified which are of relevance forthat respective network device and one or more tests can be performed for some or all of the set of candidate network devices to gather test data (i.e. further data) for assisting in root cause analysis.

[0070] The controller can be operable to perform property comparison between one or more properties associated with a given network device and one or more properties associated with other network devices to determine one or more candidate network devices. Properties used for property comparison may include one or more of: associated network protocols; hostnames; media access control (MAC) addresses; and / or internet protocol (IP) addresses. For example, if a given network device supports a first communication protocol (e.g. BGP), the controller may determine one or more other network devices that also support the first communication protocol as being candidate network devices. In some examples, the controller may specify a condition such as at least two or more communication protocols in common with the given network device for determining candidate network devices. Alternatively or in addition, hostnames may be compared and devices having similar hostnames may be selected as candidate network devices. Hostnames may be used to distinguish devices on the network and can be freely set by users. In some examples, hostnames may be assigned to devices based on device role and / or device type. For example, a server or other network equipment (e.g. network management controllers, routers, switches and so on) on a network may be assigned a hostname such as server! , and another server (or other similar network equipment) may be assigned a hostname server2 and so on. The controller can be operable to compare hostnames to identify similar hostnames to a hostname for a given network device resulting in generating difference data and thereby determine one or more candidate network devices. For example, keyword matching techniques may be used. Alternatively or in addition, MAC addresses and / or IP addresses may be compared for the purposes of determining one or more candidate network devices. For example, MAC addresses may be used to select candidate devices having a same type, manufacturer and / or model as the given network device.

[0071] More generally, the controller 110 may store or otherwise access data indicative of properties for network devices. In response to generating a difference file based on current state data and stored data for a given network device, operations may be performed based on one or more properties for that network device to identify other network devices having matching and / or similar properties. In some examples, one or more look-up tables may be used. In some embodiments of the disclosure, the controller 110 may classify network devices based on at least one of a network protocol for a network device, a hostname for the network device, and an internet protocol “IP” address for the network device. The controller may thus use classifications for network devices so that the candidate network devices each have at least one same classification as the given network device for which the current state data differs from the stored reference data.

[0072] It will be appreciated that various types of network devices may be used and different types of devices may be used for different sites / premises. In some examples, network device roles may be classified as one of core and non-core. Accordingly, in response to a report of an error from a core network device, one or more other core network devices can be determined as candidates for testing. It will be appreciated that a more granular classification of role-type may be used. In a similar manner, in some examples, network devices may be classified based on a plurality of different device types (e.g. switches, routers, network management controllers and so on). Accordingly, in response to a report of an error from a first type of network device, one or more other devices also of the first type can be determined as candidates for testing.

[0073] In some embodiments of the disclosure, the controller may classify network devices based on one or more predetermined network protocols supported, or currently being run, by the network devices. In particular, the controller may classify network devices based on one or more communication protocols comprising one or more of: BGP; BFD; OSPF; STP; IS-IS; and LISP. In some examples, a single set of communication protocols may be defined and used by the controller 110 for classifying network devices so that devices are either assigned a first classification or not assigned a classification. For example, a single set of communication protocols may be defined which corresponds to a set of communication protocols for a given device role. Accordingly, devices may be either classified as having that given device role or not assigned a classification. For example, a defined set of communication protocols may comprise the BGP, BFD, OSPF and STP protocols. These protocols may be expected to be supported by core network devices. Accordingly, network devices supporting the BGP, BFD, OSPF and STP protocols may be classified as core network devices.

[0074] In some examples, devices supporting a first set of communication protocols may be assigned a first classification, and devices supporting a second set of communication protocols may be assigned a second classification. For example, network devices supporting the BGP, BFD, OSPF and STP protocols may be classified as core network devices, and network devices supporting the IS-IS, BFD and LISP protocols may be classified as edge network devices. More generally, the controller may be operable to determine device roles associated with network devices in dependence upon supported communication protocols. Hence whereas device type (e.g. a switch type of device) relates to a fixed characteristic, device role is specific to the deployment of the network device and a given device can have different roles for different deployments (e.g. a core switch and an edge switch).

[0075] Hence more generally, the controller may determine one or more candidate network devices in dependence on at least one of a device type and a device role associated with a network device. In some examples, metadata associated with reports and / or received state data (e.g. in a received file) may specify one or both of a device type and device role. Referring again to the method 200, at the step 250 (and similarly the step 340 in the case of the method 300), the controller may determine one or more candidate network devices for testing in dependence on a device type and / or device role for the network device so that the candidate network devices each correspond to a same device type and / or device role as the network device.

[0076] In some cases, a set of candidate network devices may be determined which each have a same device type as the network device. For example, if the current state data for a network switch (as an example of a device type) differs from the stored reference data, the controller may determine one or more other network switches of the network as candidate network devices to be tested. In some cases, a set of candidate network devices may be determined which each have a same device role in the computing network as the network device. For example, if the current state data for a core network device (as an example of a device role) differs from the stored reference data, the controller may determine one or more other core network devices of the network as candidate network devices to be tested. In some cases, a set of candidate network devices may be determined which each have a same device type and a same device role in the computing network as the network device. For example, if the current state data for a core switch device differs from the stored reference data, the controller may determine one or more other core switch devices of the network as candidate network devices to be tested.

[0077] The above discussion refers to using properties for a given network device to determine one or more candidate network devices. In some embodiments of disclosure, the controller may compare current state data with stored reference data and generate difference data indicative of one or more differences between the current state data and the stored reference data. The controller may determine one or more of the candidate network devices in dependence on one or more of the differences. For example, difference data may indicate a deviation from the normal behaviour such as a difference in BGP neighbour state for one or more BGP neighbours and / or a difference in a number of learned prefixes from a BGP neighbour. This information may be used for determining one or more candidate devices fortesting. In particular, one or more BGP neighbours can be selected as candidate network devices for testing based on the difference data. A difference may relate to a BGP neighbour being down (or having a state change) which may be due to any of a configuration change, offline network device, cabling issue and optical transceiver hardware fault. The BGP neighbour can be identified as a candidate network device for testing. Any of a diagnostic test, difference test, and status query test can be performed for the BGP neighbour to obtain test data useful for root cause analysis. It will be appreciated that this is one example of using difference data (e.g. a difference file) to determine one or more candidate network devices. More generally, the difference data may indicate a deviation for an attribute associated with a connected network device (i.e. another device connected to the given network device for which current state data comparison generates difference data) and the connected network device can be selected as a candidate network device.

[0078] The controller is operable to perform one or more tests for one or more of the candidate network devices. References to the controller performing one or more tests encompass the controller itself running one or more tests and / or the controller triggering remote running of one or more tests on one or more candidate devices and obtaining the results from such testing. Hence, the controller can identify candidate network devices to be targeted for testing, and perform one or more tests for at least some of the candidate network devices to obtain resulting test data. The tests may comprise any of: one more diagnostic tests; one or more difference tests; and / or one or more status query tests. The resulting test data is useful for root cause analysis. The controller may gather the resulting test data together and store the test data in a suitable form for evaluation by a user. The controller may generate a support ticket comprising the resulting test data. The support ticket may be communicated to a service management apparatus for evaluation by a user and the inclusion of the resulting test data can assist in root cause analysis. For cases in which a report is initially received by the controller, the support ticket may include both the report (i.e. the initially received report with little to no substantive detail) and also the test data obtained by the controller.

[0079] The controller may have access to stored diagnostic testing data relating to diagnostic tests that can be run for one or more network devices in order to generate diagnostic test results about operation of those network devices. For example, diagnostic test data may include commands for running network utilities such as packet sniffers, device I port mapping tools, input / output port data flow reports, hardware configuration reports, software configuration reports, or any other suitable diagnostic tools as are known in the field. The diagnostic testing data may be sufficient for the controller to run one or more such tests, and / or to trigger remote running of the diagnostic utility on a network device remotely from the controller, for example. Diagnostic testing data may be stored by the controller or otherwise generally accessed by the controller (e.g. from a connected apparatus such as a datastore).

[0080] The controller may perform a difference test for a candidate network device and generate difference data for the candidate network device in dependence on whether differences are present between current state data for the candidate network device and stored reference data for the candidate network device. In other words, the method steps 310, 320 and 330 can be similarly performed for one or more of the candidate network devices to generate difference data depending on whether one or more differences are present between current state data for a candidate network device and stored reference data for that candidate network device. In more detail, for a given candidate network device, the controller may request current state data for that candidate network device and perform comparison of received state data with respect to stored reference data.

[0081] The test data resulting from the difference test may comprise difference data (e.g. a difference file) indicative or presence / absence of a difference. Optionally, the test data resulting from the difference test may comprise difference data (e.g. a difference file) indicative of details of one or more of the differences. The controller may perform a difference test for each of a plurality of candidate network devices identified by the controller. The difference test for each candidate network device may indicate presence / absence of a difference with respect to stored reference data. This test data may be included in a same support ticket.

[0082] The controller may perform a status query test for a candidate network device to obtain status data for the candidate network device. A status query test may be performed to check a current (live) status for one or more of the candidate network devices. For example, a result of a status query test may indicate any of a plurality of possible states (e.g. available, not available, number of connected devices, connection status for one or more ports, whether a port is experiencing a network error and so on).

[0083] The controller may generate a support ticket comprising the test data for one or more of the candidate network devices. The support ticket may include test data for any combination of the tests discussed above for some or all of the candidate network devices. In some embodiments of the disclosure, the support ticket may comprise results from only one of diagnostic testing, difference testing, and status query testing. For example, the controller may firstly perform one type of testing for one or more candidate network devices and if the results are conclusive performance of other testing may not be called for. In some examples, a support ticket may comprise results for two or more different types of test for a same candidate network device.

[0084] In some examples, the controller may firstly perform difference testing for each of the candidate network devices. The results of the difference testing can be included in a same support ticket. If the results indicate that the current state data differs from the stored reference state data for each of the candidate network devices, this can potentially be indicative of a significant error or issue. In particular, if the results indicate that the current state data differs from the stored reference state data for each of the candidate network devices and each candidate network device is of a same device type and / or same device role this can indicate that a network error or issue is present which is likely to impact network service. The controller can thus be operable to identify such an occurrence and may generate the support ticket to include the test data and metadata indicating that the difference testing failed (i.e. resulted in difference data indicating presence of at least one difference) for each candidate network device.

[0085] Hence, in the techniques of the present disclosure in response to generating difference data for a respective network device, the controller may perform difference testing for a plurality of candidate network devices each of a same device type and / or device role as the respective network device and the resulting test data can be gathered into a support ticket. In this way, presence of an error or issue affecting devices of a same device type and / or role can be quickly identified and indicated in a support ticket. Metadata may be included in the support ticket for specifying a device type and / or role for the respective network device and one or more of: a number of candidate network devices for which the difference testing failed; and whether the difference testing failed for each of the candidate network devices subjected to difference testing.

[0086] In some examples, the controller may generate the metadata to provide an indication of a severity rating for the support ticket. The controller may determine the severity rating in dependence on the test data obtained from performing the tests. For example, the controller may determine a severity rating in dependence upon a proportion of the candidate network devices that failed the difference testing. A highest severity rating may be determined in the case of each candidate network device failing the difference testing. Moreover, if devices of a same device type and / or device role are impacted (e.g. each fail difference testing and / or have similar or matching difference files), then that device functionality and / or role can be assumed to have failed and is not available and the network is severely impacted. This information can be included in a support ticket provided to a user.

[0087] In some examples, the controller may also include the current state data for one or more of the candidate network devices in the support ticket. For example, first current state data for a first candidate network device and second current state data for a second candidate network device may each be included in the support ticket. More generally, the controller can obtain current state data and test data for candidate network devices and potentially all of this data may be included in the support ticket.

[0088] Results from performing difference testing for some or all of the candidate network devices may each be included in a same support ticket. The difference data for a given candidate network device (also referred to as candidate network device difference data) may be indicative of one or more differences between the current state data and the stored reference data for the candidate network device. For example, the candidate network device difference data may specify one or more attributes for which a difference has been determined, or the candidate network device difference data may may specify both the current state for a given attribute and the stored state for that given attribute. The candidate network device difference data can be included in the support ticket and can quickly provide an indication to a user (e.g. at a support ticket evaluation stage) of both a current state (e.g. current configuration) and an expected state (e.g. expected configuration) which are different from each other, which may assist in faster and more efficient root cause analysis. More generally, candidate network device difference data for one or more candidate network devices indicating at least attributes for which differences have been identified, and potentially current and stored states (e.g. configurations), can be provided in a same support ticket.

[0089] In some embodiments of the disclosure, a system comprises the controller 110 and a ticketing system. Figure 4 is a schematic diagram illustrating a system 400 comprising the controller 110 and a ticketing system 140. The communications between the controller, network devices and ticketing system may use any suitable communication techniques Figure 4 schematically illustrates an example of flow of data for the system 400. At 401 , communications between the network devices 120 and the controller 110 allow the controller 110 to achieve the functionality discussed with respect to Figures 2 and 3. At 402, the controller provides at least some of the test results to the ticketing system. In some cases, the controller sends support tickets to the ticketing system, in which a respective support ticket comprises at least some of the test data generated from a set of candidate network devices. Hence, the controller can be operable to perform tests and gather test data for a set of candidate network devices and assemble the tests results into a respective support ticket. Comparative to a technique in which reports from network devices are merely sent to the ticketing system, the present arrangement provides support tickets which are enriched with test data obtained by the controller which can assist in discovering root causes of errors, issues and other problems for computer networks.

[0090] In some embodiments of the disclosure, at least one of the controller 110 and the ticketing system 140 may analyse a support ticket and generate one or more suggestions for root cause analysis for the network. Figure 5 schematically illustrates an example of a system 500. The system 500 comprises the controller 110, network devices 120 and ticketing system 140 (as already discussed with respect to Figure 4) and further comprises an artificial intelligence (Al) model 150. At 403, the controller can input a support ticket (or potentially at least some of the test data for a support ticket) to the Al model 150. The Al model has been trained to output root cause suggestions in dependence on the data received from the controller 110. In response to receiving the input support ticket, at 404 the Al model outputs one or more suggestions for root cause analysis for the support ticket. The one or more suggestions may comprise one or more of: one or more corrective actions to be performed by the user for correcting an error; one or more investigative actions to be performed by the user for obtaining further details in relation to an error; and one or more suggested tests to be performed for obtaining further test data. In some examples, the Al model may output a list of suggested actions to be performed for root cause analysis.

[0091] More generally, a support ticket can be input to the Al model 150 and in response one or more suggestions can be output for assisting in root cause analysis. Information received from the Al model 150 may be added to the support ticket and sent to the ticketing system 140 for evaluation by a user. The ticketing system may be operable to display user interface images so as to display information included in a support ticket. The ticketing system may comprise a display unit for displaying the user interface images and a user input unit operable to receive one or more user inputs. Upon reviewing a support ticket, the user may instruct further actions and commands can be sent from the ticketing system 140 to the controller 110 for performing such actions (e.g. further testing and / or corrective actions may be instructed). Further testing (e.g. diagnostic testing, difference testing and / or status query testing) may be instructed for one or more of the candidate network devices and / or other network devices not included in the set of one or more candidate network devices. In some examples, the ticketing system may instruct one or more corrective actions for network devices and such commands can be communicated from the ticketing system to the network devices (e.g. directly or via the controller 110).

[0092] Figure 5 provides an example in which the controller 110 is operable to access to the Al model 150. The Al model 150 may be accessible by any of the controller 110 and the ticketing system 140. In some examples, support tickets may be provided from the controller 110 to the ticketing system 140, and the ticketing system may input support tickets to the Al model. Hence, processing for Al-based supplementing of a support ticket may be performed at the controller and / or at the ticketing system. The Al model may be implemented as a processor-implemented Al model provided as part of a local or remote processing device (e.g. a remote server, such as a cloud-based server). In some cases, the Al model may be implemented by one or more processors of the controller and / or ticketing system. Hence, in some examples at least one of the controller 110 and the ticketing system 140 may run the Al model for outputting suggestions for root cause analysis for support tickets. The Al model may be generative Al model.

[0093] Figure 5 shows an example in which the Al model 150 provides suggestions for root cause analysis. In some cases, one or both of the Al model and a database may be used for providing suggestions for root cause analysis. For example, referring to Figure 5 in some examples the Al model 150 may be replaced (or complemented) with a database. Hence more generally, in some embodiments of the disclosure at least one of the controller 110 and the ticketing system 140 may have access to a database storing information that associates test data with root cause suggestions. The database may for example store historical information including previous test data and previous actions (e.g. corrective actions and / or investigative actions and / or testing) for solving errors, issues and other similar problems. The controller and / or ticketing system can thus use current test data (or a current support ticket) to identify root cause suggestions from the database. The ticketing system may thus display the root cause suggestions to a user for assistance in root cause analysis.

[0094] Referring to the methods 200 and 300, test data can be obtained and provided to a user for assisting in root cause analysis and, upon identifying an appropriate solution for an error or issue, the user may log the test data and also the solution (one or more actions) in the database. Accordingly, the database can be populated with information logged from root cause analysis for previous cases. The database can thus store associations between previous instances of errors or issues and test data for network devices that were linked to those errors. The database may thus be used with current test data for a current error or issue to provide root cause suggestions. The database may store a set of associated entries in which a first entry includes test data obtained from candidate network devices, and a second entry includes one or more root cause suggestions. Such entries may be stored for each of a plurality of previously fixed errors or issues. Comparison of current test data for a current error or issue (e.g. obtained at the steps 260 and 350) against entries in the database may be used to identify a same or similar set of test data and the associated one or more root cause suggestions can be provided to the user as a suggestion.

[0095] In some examples, the controller and / or ticketing system may firstly use the database to obtain one or more suggestions for root cause analysis, and if needed the controller and / or ticketing system may subsequently input the test data and / or one or more of the suggestions obtained from the data to the Al model 150. Hence, in some cases the database may be used as a first pass and the Al model may be used as an optional second pass. More generally, at least one of the controller 110 and the ticketing system 140 may generate one or more suggestions for root cause analysis in dependence upon the support ticket. At least one of the controller and the ticketing system may access at least one of the database and the Al model generate one or more of the suggestions for root cause analysis in dependence upon the support ticket.

[0096] The invention may also broadly consist in the parts, elements, steps, examples and / or features referred to or indicated in the specification individually or collectively in any and all combinations of two or more said parts, elements, steps, examples and / or features. In particular, one or more features in any of the embodiments described herein may be combined with one or more features from any other embodiment(s) described herein.

[0097] When used in this specification and claims, the terms "comprises" and "comprising" and variations thereof mean that the specified features, steps or integers are included. The terms are not to be interpreted to exclude the presence of other features, steps or components.

[0098] It will be appreciated that example embodiments can be implemented by computer software operating on a general purpose a computing system. In these examples, computer software, which when executed by a computer, causes the computerto carry out any of the methods discussed above is considered as an embodiment of the present disclosure. Similarly, embodiments of the disclosure are provided by a non-transitory, machine-readable storage medium which stores such computer software. Although certain example embodiments of the invention have been described, the scope of the appended claims is not intended to be limited solely to these embodiments. The claims are to be construed literally, purposively, and / or to encompass equivalents. It will also be apparent that numerous modifications and variations of the present disclosure are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims, the disclosure may be practised otherwise than as specifically described herein.

Claims

CLAIMS1 . A system for determining data for root cause analysis for a computer network, the system comprising a controller configured to: receive current state data for a network device; compare at least some of the current state data with stored reference data for the network device; generate difference data in dependence on whether one or more differences are present between the current state data and the stored reference data; determine one or more candidate network devices for testing in dependence on one or more properties for the network device, responsive to one or more differences being present between the current state data and the stored reference data; and perform one or more tests for one or more of the candidate network devices to obtain test data indicative of one or more properties associated with one or more of the candidate network devices.

2. The system according to claim 1 , wherein the controller is configured to determine one or more of the candidate network devices in dependence on at least one of a device type and a device role associated with the network device.

3. The system according to claim 1 or claim 2, wherein the controller is configured to classify network devices based on at least one of a network protocol, a hostname, and an internet protocol “IP” address, and wherein one or more of the candidate network devices have a same classification as the network device.

4. The system according to any preceding claim, wherein the difference data is indicative of one or more differences between the current state data and the stored reference data, and the controller is configured to determine one or more of the candidate network devices in dependence on one or more of the differences.

5. The system according to any preceding claim, wherein the controller is configured to perform one or more diagnostic tests for a candidate network device to obtain diagnostic testing results for the candidate network device.

6. The system according to claim 5, wherein the controller is configured to access stored diagnostic testing data associated with the candidate network device and use the stored diagnostic testing data to run one or more diagnostic tests associated with the candidate network device.

7. The system according to any preceding claim, wherein the controller is configured to perform a difference test for a candidate network device and generate difference data for the candidate network device in dependence on whether at least one difference is present between current state data for the candidate network device and stored reference data for the candidate network device.

8. The system according to any preceding claim, wherein the controller is configured to perform a status query test for a candidate network device to obtain status data for the candidate network device.

9. The system according to any preceding claim, wherein the controller is configured to generate a support ticket comprising the test data for one or more of the candidate network devices.

10. The system according to claim 9, wherein the support ticket further comprises the difference data generated for the network device in which the difference data generated for the network device is indicative of one or more differences between the current state data and the stored reference data.11 . The system according to any preceding claim, wherein the controller is configured to: receive a report indicative of at least one of potential error and a change in a current state from the network device; and request the current state data from the network device in response to the report.

12. The system according to claim 11 , wherein the controller is configured to request the current state data to include one or more attribute types corresponding attribute types included in the stored reference data for the network device.

13. The system according to any preceding claim, wherein the controller is configured to receive a plurality of reports indicative of potential errors and / or changes in current states and analyse at least some of the plurality of reports to determine an order of priority for at least some of the plurality of reports.

14. The system according to claim 13, wherein the controller is configured to analyse at least some of the plurality of reports to detect one or more keywords corresponding to names and / or types of predefined errors.

15. The system according to claim 14, wherein the controller is configured to determine a degree of severity for a report in dependence one or more keywords corresponding to names and / or types of predefined errors and order at least some of the plurality of reports according to the degree of severity.

16. The system according to any preceding claim, wherein the controller is configured to capture snapshots for network devices to obtain stored reference data for one or more of the network devices.

17. The system according to any preceding claim, further comprising a ticketing system configured to receive a support ticket from the controller, in which the support ticket comprises at least some of the test data.

18. The system according to claim 17, wherein at least one of the controller and the ticketing system is configured to generate one or more suggestions for root cause analysis in dependence upon the support ticket.

19. The system according to claim 18, wherein at least one of the controller and the ticketing system is configured to access at least one of a database and an artificial intelligence ‘Al’ model to generate one or more of the suggestions for root cause analysis in dependence upon the support ticket.

20. A computer-implemented method comprising: receiving current state data for a network device; comparing at least some of the current state data with stored reference data for the network device; generating difference data in dependence on whether one or more differences are present between the current state data and the stored reference data; determining one or more candidate network devices for testing in dependence on one or more properties for the network device, responsive to one or more differences being present between the current state data and the stored reference data; and performing one or more tests for one or more of the candidate network devices to obtain test data indicative of one or more properties associated with one or more of the candidate network devices.

21. A computer program comprising instructions which, when executed by a computer, cause the computer to perform the method according to claim 20.

Images