Machine learning based firmware version recommender

Abstract

Embodiments of the present disclosure relate to machine learning based firmware version recommenders. Examples of the presently disclosed technology provide an automated firmware recommendation system that injects machine learned intelligence into the firmware recommendation process. To accomplish this, examples train a machine learning model on a dynamic basis against a large volume of historical customer firmware update data (e.g., examples can train a machine learning model weekly to predict firmware updates accepted by a vendor’s customers in the last 6 months). From this dynamic training, the machine learning model can learn to predict / recommend the best firmware version for a customer / network device cluster based on firmware related features, recent customer preferences, and other customer specific factors. Once trained, examples can deploy the machine learning model to make highly customized firmware recommendations for individual network device clusters of individual customers, taking into account the aforementioned factors.

Description

BACKGROUND

[0001] Firmware is a type of software that is embedded in hardware (e.g., network devices such as access points, switches, gateways, etc.). Firmware of network devices can be periodically updated to (1) take advantage of new / improved functionality of new firmware versions of network devices (e.g., enhanced security functionality, new / improved functionality, fewer bugs, etc.); and (2) ensure that network devices remain compatible with other network devices that use new firmware versions. As such, vendors typically want customers to run the latest, stable firmware version on their network devices. Firmware upgrades can be automatically provided to customers via a vendor-specific network management system (as used herein, a network management system can refer to a computerized system that automates various aspects of network management / administration - Aruba Central by HPE is one example of a network management system as a unified, cloud-based network operations and security platform).

[0002] When deployed, network devices typically operate together as a group / cluster within a common network (as used herein, a “network device cluster” can refer to a group / cluster of network devices of the same type - e.g., access points can be a first type of network device, switches can be a second type of network device, gateways can be a third type of network device, etc. - that operate together within a common network). For example, a cluster of access points deployed at a customer site can operate together to provide wireless access to users (it should be understood that the access points can be the same network device type - i.e., access points - but can include different models within the network device type - e.g., a first access point model, a second access point model, etc.). In large deployments (e.g., deployments of 100 or more access points), a network device cluster can include different network device models with different ranges of compatible firmware (e.g., a particular firmware version can be compatible on model A access points within the network device cluster, but not on model B access points within the network device cluster). Ideally, a network device cluster would utilize the same firmware version to ensure that network devices within the cluster remain interoperable / compatible. However, interoperability / compatibility within a network device cluster is not always achievable due to inherent deficiencies in traditional firmware recommendation processes. BRIEF DESCRIPTION OF DRAWINGS

[0003] The present disclosure is described in detail with reference to one or more various examples in accordance with the principles described herein. These examples are described merely for illustration and are not intended to limit the present disclosure in any way.

[0004] Figure 1 Example diagrams illustrating various examples in accordance with the presently disclosed technology are shown in which an automatic firmware recommendation system makes firmware version recommendations for a group of network devices.

[0005] Figure 2 An example workflow is shown that can be used to: (1) train a machine learning model to predict firmware version updates; and (2) deploy the machine learning model at inference time to automatically recommend compatible firmware versions for a set of network devices, in accordance with various examples of the presently disclosed technology.

[0006] Figure 3 An example computing system is shown that can be used to train a machine learning model to predict acceptance of firmware version updates, in accordance with various examples of the presently disclosed technology.

[0007] Figure 4 An example computing system is shown that can be used to automatically recommend compatible firmware versions for a set of network devices, in accordance with various examples of the presently disclosed technology.

[0008] Figure 5 A block diagram of an example computer system in which various examples described herein can be implemented is shown.

[0009] The drawings are not exhaustive and do not limit the disclosure to the precise forms disclosed. DETAILED DESCRIPTION

[0010] Providing customized firmware recommendations (e.g., firmware recommendations that are customized for individual customers or individual clusters of network devices for individual customers) is a serious challenge. Vendors can have millions of network devices deployed in the field at thousands / tens of thousands of customer sites. The millions of network devices will have different ranges of compatible firmware versions based on type (e.g., access points, switches, gateways, etc.), model (e.g., access point model, access point model B, etc.), and configuration (e.g., configuration 1 for access point model A, configuration 2 for access point model A, etc.). Even among similar network devices (e.g., network devices of the same type / model), the best firmware version for one customer can be different than the best firmware version for another customer based on various customer-specific factors (e.g., (a) size and number of customer sites; (b) customer-specific deployment configurations; (c) customer-specific network device configurations; (c) customer-specific vulnerability issues; (d) customer-specific bugs, etc.).

[0011] This challenge is complicated by the fact - and as noted above - that the best firmware version recommendation often needs to ensure that the set of network devices operating together (i.e., the network device cluster) are running on the same firmware version. This aspect of the firmware recommendation problem can be particularly challenging in large customer deployments (e.g., customer deployments of 100 or more access points), where the network device cluster can include different network device models with different ranges of compatible firmware (e.g., a particular firmware version can be compatible on access points of model A within the cluster, but not on access points of model B within the cluster).

[0012] The traditional process of recommending firmware versions is completely incapable of addressing the above challenges. Currently, this process involves repeated discussions between product managers and engineering teams about which firmware version updates should be recommended for various types and models of network devices of a vendor. After reaching cross-organizational consensus, the agreed firmware versions are typically subjected to an internal testing process. After internal testing, the final recommendations are provided to customers.

[0013] Due to this manual, time-consuming process that typically requires cross-organizational communication / consensus between different product managers and engineering teams, firmware version update / recommendation can not be frequent (e.g., once every 2-3 months) and slow to react to new firmware version releases. The delay between firmware version releases and recommendations deprives customers of the opportunity to immediately (or almost immediately) take advantage of new / improved features of newly released firmware versions. These delays also degrade / limit the performance of a vendor’s network management system (e.g., HPE’s Aruba Central) that can rely on using the latest firmware version updates for customers’ network devices to provide the best service.

[0014] Another problem with the above human-driven recommendation process is that it is fundamentally incapable of providing customized recommendations (e.g., recommendations customized for individual customers or recommendations customized for individual clusters of network devices of individual customers). As mentioned above, a vendor can have thousands / tens of thousands of different customers, and each customer can have multiple sites / networks. Due to having such a large customer base, it is difficult for a vendor’s engineering and product management teams to have customer-specific (or network device cluster-specific) discussions about firmware recommendations, which is both difficult and expensive, and inefficient. Thus, the bundle (i.e., non-customer-specific) recommendations that are typically provided can be suboptimal for various customers / customer sites.

[0015] For example, the above described bundle of recommendations can result in a‘non- compatible’ firmware version for a cluster of network devices. This is because the recommended firmware version can only be compatible on certain network devices in the cluster of network devices. Thus, if the customer follows the recommendation, then some of their network devices can have incompatible firmware installed. Alternatively, the customer can follow the recommendation for the subset of network devices within the cluster that are compatible with the recommended firmware version, resulting in the network devices of the cluster having different firmware versions. Either of these scenarios is suboptimal for network operation / health, and can result in: (a) a poor customer / user experience, (b) high troubleshooting costs for the vendor; and (c) a reduction in performance of the vendor’s network management system, which relies on customer network devices running on compatible firmware versions to provide optimal service. For example, incompatible firmware-network devices (i.e. network devices with incompatible firmware versions installed) can interfere with the network management system’s ability to collect network telemetry data - which can hinder the network management system’s ability to provide dynamic data-driven insights to customers.

[0016] Due to the poor user experience of updating firmware versions, customers can be discouraged from upgrading to new firmware versions provided to them. This can result in more firmware versions running on the vendor’s network devices than desired (e.g. a customer can have 50+ firmware versions running on the vendor’s access points, whereas ideally the customer should have the latest 10 firmware versions compatible across different hardware models of the vendor’s access points). Thus, the vendor’s technical support center can be overwhelmed with issues related to firmware upgrades. As described above, incompatible firmware network devices also reduce the performance of the vendor’s network management system that interacts with / manages the incompatible firmware network devices. For example, incompatible firmware-network devices can interfere with the vendor’s network management system’s ability to collect network telemetry data - which can hinder the vendor’s network management system’s ability to provide dynamic data-driven insights to customers.

[0017] In this context, examples of the presently disclosed technology provide an automated firmware recommendation system that injects machine-learned intelligence into the firmware recommendation process. To accomplish this, examples train a machine learning model on a dynamic basis, on a large amount of historical customer firmware update data (e.g., examples can train a machine learning model every week to predict acceptable firmware updates made by a vendor’s customers in the last 6 months). From this dynamic training, the machine learning model can learn to predict / recommend the best firmware version for a customer / network device cluster based on firmware-related features, recent customer preferences, and other customer-specific factors. Once trained, examples can deploy the machine learning model to make highly customized firmware recommendations for individual network device clusters of individual customers, taking into account the aforementioned factors. In certain examples, the automated firmware recommendation system can be incorporated into a vendor’s network management system (e.g., HPE’s Aruba Central). Thus, examples can improve the functionality of a vendor’s network management system while also improving its performance (as noted above, by reducing the number of incompatible firmware network devices with which the network management system interacts, examples can improve the performance of the network management system).

[0018] Further (and as noted above), examples can also ensure that all network devices within a network device cluster are compatible with the recommended firmware version. Thus, examples can reduce the number of incompatible firmware network devices in the field—improving the customer / user experience, reducing vendor support costs, improving the performance of a vendor’s network management system, etc.

[0019] In various examples, examples can use a machine learning model to compute a plurality of firmware update likelihood scores for each network device of a cluster of network devices, which are based on: (1) features of a “pre-update” firmware version currently installed on the network devices of the cluster of network devices (e.g., a number of customer-induced bugs - weighted by bug severity - for the pre-update firmware version, a number of internal bugs - weighted by bug severity - for the pre-update firmware version, an age of the pre-update firmware version, a numerical value measuring a popularity level of the pre-update firmware version, a numerical value measuring a security level of the pre-update firmware version, etc.); (2) features of all available / expected “update” firmware versions that are compatible on the network devices of the cluster of network devices (e.g., a number of customer-induced bugs - weighted by bug severity - for the update firmware versions, a number of internal bugs - weighted by bug severity - for the update firmware versions, an age of the update firmware versions, a numerical value measuring a popularity level of the update firmware versions, a numerical value measuring a security level of the update firmware versions, a numerical value measuring an operational performance level of the update firmware versions, etc.); and (3) various customer / customer site-specific factors (e.g., (a) a size and number of customer sites; (b) customer-specific deployment configurations; (c) customer-specific network device configurations; (d) customer-specific vulnerability issues; (e) customer-specific bugs; etc.). For a first network device: (1) the first firmware update likelihood score can include a numerical score quantifying a likelihood that the first network device will be updated from its pre-update firmware version to a first update firmware version, the first update firmware version being compatible with the first network device; and (2) the plurality of firmware update likelihood scores can include firmware update likelihood scores associated with all available / expected update firmware versions that are compatible on the first network device. Examples can then compute an aggregated firmware update likelihood score across the cluster of network devices for each firmware version available to the network devices of the cluster of network devices. Examples can then recommend to at least one network device in the cluster of network devices an update to an available update firmware version that has the highest aggregated firmware update likelihood score among the available update firmware versions that are compatible on all network devices of the cluster of network devices.

[0020] In comparison to current human-driven firmware recommendation processes, examples of the presently disclosed technology provide a number of advantages. As a first example, examples can provide customized recommendations for individual network device clusters of individual customers. Such customized recommendations are generally unattainable for current human processes due to a lack of artificial intelligence / automation. As a second (and related) example, examples can ensure that all network devices in a network device cluster are compatible with the recommended firmware version. As noted above, recommending incompatible firmware versions for a customer’s network device cluster is a common failure of the current firmware recommendation process for bundles of recommendations. As a third example, examples can provide recommendations more quickly than current firmware recommendation processes. For example, examples can leverage the automation / intelligence of machine learning to dynamically make recommendations in response to new firmware version releases— which, in various examples, can be released by a network management system that coordinates firmware update recommendations— immediately. Based on all of the above advantages, examples of the presently disclosed technology can reduce the number of field-incompatible firmware network devices— thereby improving customer / user experience, reducing vendor support costs, improving performance of a vendor’s network management system, etc.

[0021] Before describing examples of the presently disclosed technology in more detail, it should be understood that the automatic firmware recommendation system of the presently disclosed technology does not simply recommend the “latest” firmware version that is compatible on individual network devices. Rather, in addition to the various firmware version characteristics described above, the automatic firmware recommendation system of the presently disclosed technology makes network device cluster-specific recommendations based on a number of customer / site-specific factors taken together (informed by training on a large amount of historical customer data). In some cases, this can result in recommending a firmware version for a network device cluster that is not the latest firmware version that is compatible on all network devices of the network device cluster. Relatedly, this can also result in recommending a firmware version for a network device cluster that is not the best / “best” firmware version for some network devices of the network device cluster— but is best for the network device cluster as a whole— taking into account customer- and customer site-specific factors.

[0022] Figure 1Various examples are shown of the automatic firmware recommendation system 122 making firmware version recommendations for a set of network devices 106a-c of a customer network deployment 100 in accordance with the presently disclosed technology. The automatic firmware recommendation system 122 can be incorporated into a computerized network management system 120 (e.g., HPE’s Aruba Central) that manages a wide range of network deployments including the network deployment 100 (where the set of network devices 106a-c are separate). In other words, it should be understood that the automatic firmware recommendation system 122 is a multi-tenant solution that utilizes data from many customers / customer deployments (e.g., customer deployments 200, 300, 400, and any number of additional customer deployments - not fully detailed in the example Figure 1

[0023] The network deployment 100 can include a host network, which can be, for example, an office network, a home network, or other network setup. The network deployment 100 network can be a private network, e.g., a network that can include security and access controls to limit access to authorized users of the private network. Authorized users can include, for example, employees of a company, residents of a house, customers of a business, and so on.

[0024] In the illustrated example, the network deployment 100 includes a controller 104 that communicates with the automatic firmware recommendation system 122. The controller 104 can provide communication with the automatic firmware recommendation system 122 for the network deployment 100, although it can not be the only point of communication with the automatic firmware recommendation system 122 for the network deployment 100. A single controller 104 is shown, although the network deployment 100 can include multiple controllers and / or multiple points of communication with the automatic firmware recommendation system 122. In certain examples (e.g., in the case that the automatic firmware recommendation system 122 is a cloud-based deployment), the controller 104 communicates with the automatic firmware recommendation system 122 over the Internet (not shown). In these examples, the automatic firmware recommendation system 122 can communicate / connect with the Internet directly (where the controller 104 provides router functionality) or through a router (not shown). In certain examples, the controller 104 can communicate with the automatic firmware recommendation system 122 directly or through a router (not shown). As noted above, and described in more detail below, the automatic firmware recommendation system 122 can provide firmware recommendations for network devices (e.g., wireless APs 106a-c) of the network deployment 100 to the controller 104.

[0025] The controller 104 can operate to configure and manage network devices, such as network devices of the network deployment 100. The controller 104 can operate to configure and / or manage switches, routers, access points, and / or client devices connected to a network. The controller 104 itself can be an access point or provide the functionality of an access point. ​

[0026] The controller 104 can communicate with one or more switches 108 and / or wireless access points (APs) 106a-c. The switches 108 and wireless APs 106a-c provide network connectivity to various client devices 110a-j. Using a connection to a switch 108 or wireless AP 106a-c, the client devices 110a-j can access network resources (including other devices of the network deployment 100) and the automatic firmware recommendation system 122.

[0027] Examples of client devices can include desktop computers, laptop computers, servers, web servers, authentication servers, authentication-authorization-accounting (AAA) servers, domain name system (DNS) servers, dynamic host configuration protocol (DHCP) servers, Internet protocol (IP) servers, virtual private network (VPN) servers, network policy servers, mainframes, tablet computers, e-readers, netbook computers, televisions and similar monitors (e.g., smart TVs), content receivers, set-top boxes, personal digital assistants (PDAs), mobile phones, smart phones, smart terminals, dumb terminals, virtual terminals, video game consoles, virtual assistants, Internet of Things (IOT) devices, and the like.

[0028] Within the network deployment 100, the switches 108 are included as an example of an access point to the network established by the client devices 110i-j in the network deployment 100. The client devices 110i-j can connect to the switches 108 and, through the switches 108, have access to other devices within the network deployment 100. The client devices 110i-j can communicate with the switches 108 through wired 112 connections. In the example shown, the switches 108 communicate with the controller 104 through wired 112 connections, although the connections can also be wireless.

[0029] The wireless APs 106a-c are included as another example of an access point to the network established by the client devices 110a-h by the network deployment 100. Each of the wireless APs 106a-c can be a combination of hardware, software, and / or firmware configured to provide wireless network connectivity to the wireless client devices 110a-h. In the example shown, the wireless APs 106a-c can be managed and configured by the controller 104. The wireless APs 106a-c communicate with the controller 104 and the network through connections 112, which can be wired or wireless interfaces.

[0030] As noted above, the wireless APs 106a-c can comprise a network device cluster. A network device cluster can refer to a group / cluster of the same type of network devices (e.g., access points, switches, gateways, etc.) that operate together in a common network. While the wireless APs 106a-c can have a common vendor, they can be different models with different sets of compatible firmware versions available for their use. As noted above, the cluster of wireless APs 106a-c should utilize the same firmware version in order to maintain optimal interoperability / compatibility. Unfortunately, one issue with the conventional manual firmware version recommendation process is that it often recommends ‘incompatible’ firmware versions for a network device cluster. For example, the process can recommend a firmware version that is only compatible on the wireless APs 106a and 106b. Thus, if the customer follows the recommendation, the wireless AP 106c can have incompatible firmware installed. Alternatively, the customer can follow the recommendations of the wireless APs 106a and 106b, resulting in the wireless APs 106a and 106b having different firmware versions than the wireless AP 106c. Either of these scenarios is suboptimal for network operation / health and can result in: (a) a poor customer / user experience; (b) high troubleshooting costs for the vendor; (c) and a decrease in performance of the vendor’s network management system that relies on customer network devices running on compatible (and / or the latest) firmware versions to provide optimal service. As noted above, and will be described below, by making network device cluster specific recommendations that ensure the recommended firmware version is compatible on all network devices of the network device cluster, the examples can improve the current firmware recommendation process.

[0031] Figure 2 An example workflow 200 is shown that can be used for (1) training a machine learning model to predict firmware version updates; and (2) deploying the machine learning model in inference to automatically recommend compatible firmware versions for a group of network devices, in accordance with the presently disclosed technology. As shown, the top of the workflow 200 includes a workflow that can be used for offline training, while the bottom of the workflow 200 includes a workflow that can be used for real-time prediction / inference.

[0032] Offline training

[0033] The training data 220 can be used to train a machine learning model to predict firmware version updates. In various examples, the training data 220 can be extracted from a data center 202, which can be a repository of network related data. In various examples, the data stored in the data center 202 can be ingested (and / or stored) by a computerized network management system (e.g., Aruba Central by HPE).

[0034] The training data 220 can include a historical firmware update dataset. The historical firmware update dataset can include data related to a plurality of historical firmware updates made on a plurality of network devices of a customer.

[0035] In various examples, the historical firmware update dataset can be vendor-specific and network device type-specific. For example, the historical firmware update dataset can include data related to all firmware updates (or some subset) made by a customer on access points of a vendor in the past 6 months. In certain examples, the historical firmware update data can be vendor-specific, but include data related to updates made on various network device types (e.g., data related to updates made on access points, switches, and gateways). Here, it should be appreciated that firmware version updates can be upgrades (i.e., from a pre-update firmware version to a newer / younger firmware version) and downgrades (i.e., from a pre-update firmware version to an older firmware version).

[0036] In the historical firmware update dataset, individual historical firmware update data can relate to individual firmware updates made by customers on network devices. Individual historical firmware update data can include information about pre-update firmware versions (i.e., firmware versions installed on network devices prior to the update) and post-update firmware versions (firmware versions to which network devices were updated). Such information can include characteristics of pre-update firmware versions and post-update firmware versions, such as: number of errors (as used herein, “errors” can refer to errors against firmware versions, such as coding errors) — weighted by error severity — for firmware versions caused by customers; number of internal errors — weighted by error severity — for firmware versions; age of firmware versions, a numerical value measuring a popularity level of firmware versions; a numerical value measuring a security level of firmware versions; a numerical value measuring an operational performance level (e.g., SLA, device time, crashes, speed, etc.) of firmware versions; and the like. Individual historical firmware update data can also include similar information related to all firmware versions available to and compatible with network devices at the time of the historical firmware update. From this information, the machine learning model can learn to predict firmware updates given a wide selection. In various examples, individual historical firmware update data can also include information specific to customers / network devices, such as: (a) size of customer sites where network devices are deployed; (b) customer-specific deployment configurations; (c) customer-specific network device configurations; (c) customer-specific vulnerability issues; (d) customer-specific errors; and the like). From this information, the machine learning model can learn to consider customer-specific factors in predicting firmware updates. In certain examples, all of the above information can be contemporaneous with the historical firmware update (e.g., measure a popularity level of firmware versions available at the time of the historical firmware update, customer-specific network device configurations at the time of the historical firmware update, etc.).

[0037] During model development / tuning and evaluation 224, the example training machine learning model is trained to predict firmware version updates using the training data 220. The training will be described in more detail in connection with Figure 3 This training is described in more detail.

[0038] The machine learning model (which can be various types of machine learning models, including classification-based machine learning models such as random forest models, k-means clustering models, etc.) can update its model parameters 226 according to its training. Thus, the updated model parameters 226 can be used for the machine learning model deployed in inference. The model parameters 226 can be updated dynamically (e.g., once a week, once a day, etc.).

[0039] Real-time speculation in production

[0040] Inference data 230 can be used by a machine learning model deployed at inference time to automatically recommend a compatible firmware version for a set of network devices. In various examples, inference data 230 can be extracted from a data center 202, which can be a repository of network-related data. In various examples, data stored in data center 202 can be sourced (and / or stored) by a network management system (e.g., HPE’s Aruba Central).

[0041] Inference data 230 can include network-related information for a plurality of network devices. In various examples, inference data 230 can be vendor-specific and network device type-specific. For example, inference data 230 can include data related to all access points (or some subset) of a vendor. In certain examples, inference data 230 can be vendor-specific but include data related to various network device types (e.g., access points, switches, and gateways).

[0042] Inference data 230 can include firmware-related information for a plurality of network devices. For example, for a network device of the plurality of network devices, firmware-related information can include: (1) a firmware version currently installed on the network device; (2) features of the firmware version currently installed on the network device; (3) a list of all firmware versions that are compatible with the network device (this can include firmware versions that would constitute an upgrade or a downgrade); (4) features of all firmware versions that are compatible with the network device; and so on.

[0043] Inference data 230 can also include customer / network device-specific information that a machine learning model can use to recommend optimal / improved firmware versions. For example, for a network device of the plurality of network devices, such information can include: (a) a size of a customer site where the network device is deployed; (b) a customer-specific deployment configuration; (c) a customer-specific network device configuration; (c) a customer-specific vulnerability issue; (d) a customer-specific bug; and so on.

[0044] As described above, the machine learning model can use the inference data 230 to recommend the best / improved firmware version for a group of network devices. In some cases, the group of network devices can include a cluster of network devices. As described above, the cluster of network devices can be a group / cluster of network devices of the same type (e.g., access points, switches, gateways, etc.) that operate together in a common network. For example, the cluster of network devices can include a cluster of access points that work together to provide wireless access to users at a customer location. While these access points can have a common vendor, they can be different models with different sets of compatible firmware versions available for their use. The cluster of network devices should utilize the same firmware version in order to maintain optimal interoperability / compatibility. As described above, and will be described below, by making cluster-specific recommendations that ensure the recommended firmware version is compatible on all network devices of the cluster of network devices, the examples can improve current firmware recommendation processes.

[0045] Accordingly, the inference data 230 can also include information related to the cluster of network devices that enables the machine learning model to determine which firmware versions are available to and compatible with each network device within the cluster of network devices.

[0046] During the model inference 234, the machine learning model deployed at inference uses the inference data 230 to recommend the best / improved firmware version for the group of network devices. As described above, the model parameters 226 of the machine learning model deployed at inference can be dynamically updated based on the training (e.g., once a week, once a day, etc.).

[0047] Figure 3 FIG. 3 illustrates an example computing system 300 that can be used to train a machine learning model to predict firmware version updates, in accordance with the presently disclosed technology. In certain examples, the computing system 300 can be incorporated into a vendor’s network management system (e.g., Aruba Central by HPE).

[0048] Referring now to Figure 3 The computing component 310 can be, for example, a server computer, a controller, or any other similar computing component capable of processing data. In Figure 3 In example implementations, the computing component 310 includes a hardware processor 312 and a machine-readable storage medium 314.

[0049] The hardware processor 312 can be one or more central processing units (CPUs), semiconductor-based microprocessors, and / or other hardware devices suitable for retrieval and execution of instructions stored in the machine-readable storage medium 314. The hardware processor 312 can fetch, decode, and execute instructions, such as instructions 316-320, for example, for controlling processes and operations for burst preloading for available bandwidth estimation. As an alternative or in addition to retrieving and executing instructions, the hardware processor 312 can include one or more electronic circuits comprising electronic components for performing the functionality of one or more instructions, such as a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or other electronic circuits.

[0050] A machine-readable storage medium, such as the machine-readable storage medium 314, can be any electronic, magnetic, optical, or other physical storage device that contains or stores executable instructions. Thus, the machine-readable storage medium 314 can be, for example, Random Access Memory (RAM), non-volatile RAM (NVRAM), electrically erasable programmable read-only memory (EEPROM), a storage device, an optical disc, and the like. In some implementations, the machine-readable storage medium 314 can be a non-transitory storage medium, where the term “non-transitory” does not encompass transitory propagating signals. As described in detail below, the machine-readable storage medium 314 can be encoded with executable instructions, such as instructions 316-320.

[0051] As described above, the computing system 300 can be used to train a machine learning model to predict firmware version updates. The machine learning model can be various types of machine learning models, such as a random forest model.

[0052] The hardware processor 312 can execute instructions 316 to generate a historical firmware update dataset. The historical firmware update dataset can include data related to historical firmware updates made on customer’s network devices.

[0053] In various examples, the historical firmware update dataset can be vendor-specific and network device type-specific. For example, the historical firmware update dataset can include data related to all firmware updates (or some subset) made by customers on the vendor’s access points in the past 6 months. In certain examples, the historical firmware update dataset can be vendor-specific but include data related to updates made on various network device types (e.g., data related to updates made on access points, switches, and gateways). Here, it should be understood that firmware version updates can be upgrades (i.e., from a pre-update firmware version to a newer / later firmware version) and downgrades (i.e., from a pre-update firmware version to an older firmware version).

[0054] In the historical firmware update dataset, individual historical firmware update data can relate to individual firmware updates made by customers on network devices. Individual historical firmware update data can include information about pre-update firmware versions (i.e., firmware versions installed on network devices prior to the update) and post-update firmware versions (firmware versions updated to). Such information can include characteristics of pre-update firmware versions and post-update firmware versions, such as: the number of errors caused by the customer - weighted by error severity - for a firmware version; the number of internal errors - weighted by error severity - for a firmware version; the age of a firmware version; a numerical value measuring a popularity level of a firmware version; a numerical value measuring a security level of a firmware version; a numerical value measuring an operational performance level of a firmware version; and so on. Individual historical firmware update data can also include similar information related to all firmware versions available to and compatible with network devices at the time of the historical firmware update. From this information, a machine learning model can learn to predict firmware updates given a broad selection. In various examples, individual historical firmware update data can also include information specific to a customer / network device, such as: (a) the size of the customer site where the network device is deployed; (b) customer-specific deployment configurations; (c) customer-specific network device configurations; (c) customer-specific vulnerability issues; (d) customer-specific errors; (e) the operational mode of customer network device SSIDs, including whether they are open or secured; (f) customer site-specific network device mix; (g) customer preferences for types of SSIDs configured (e.g., guest, employee, voice); (h) what frequency bands customer network devices are configured for (e.g., 2.4 GHz, 5 GHz, 6 GHz, or any combination of the three); (i) what protocols the customer has chosen to run on their network; (j) customer-specific upgrade behavior, including risk appetite; (k) customer-specific applications served by customer network devices; and so on. From this information, a machine learning model can learn to consider customer-specific factors when predicting firmware updates. In certain examples, all of the above information can be contemporaneous with the historical firmware update (e.g., measuring a numerical value of a popularity level of available firmware versions at the time of the historical firmware update, customer-specific network device configurations at the time of the historical firmware update, and so on).

[0055] In various examples, the historical firmware update dataset can be generated by a computerized network management system of a vendor (e.g., Aruba Central by HPE).

[0056] The hardware processor 312 can execute the instructions 318 to train a machine learning model using the historical firmware update dataset to predict acceptance of firmware updates based on computed firmware update likelihood scores for historical firmware updates. In various examples, the hardware processor 312 can train the machine learning model to predict acceptance of firmware version updates made on the network device for all historical firmware updates based on the computed firmware update likelihood scores.

[0057] Here, the predicted accepted firmware version for the historical firmware update can be a concurrently available expected update firmware version having a highest computed firmware update likelihood score for the historical firmware update.

[0058] The machine learning model can compute the firmware update likelihood scores for historical firmware updates made on the network device based on features of the pre-update firmware version installed on the network device and features of one or more (in some cases, all) concurrently available update firmware versions that are compatible on the network device. Here, the first concurrently available update firmware version can be a first firmware version available to the network device contemporaneously with the historical firmware update and compatible therewith (different from the pre-update firmware version installed on the network device). The first firmware update likelihood score can include a numerical score quantifying a likelihood that the network device will be updated from the pre-update firmware version installed on the network device to the first concurrently available update firmware version (e.g., a probability that the network device will be updated from the pre-update firmware version to the first concurrently available update firmware version). The firmware update likelihood scores for historical firmware updates made on the network device can include firmware update likelihood scores associated with all update firmware versions available to the network device contemporaneously with the historical firmware update and compatible therewith (including the first firmware update likelihood score associated with the first concurrently available update firmware version).

[0059] As noted above, the features of the pre-update firmware version installed on the network device can include: (1) a number of customer-caused errors for the pre-update firmware version at the time of historical firmware updates, weighted by error severity; (2) a number of internal errors for the pre-update firmware version at the time of historical firmware updates, weighted by error severity; (3) an age of the pre-update firmware version at the time of historical firmware updates; (4) a value measuring a popularity level of the pre-update firmware version at the time of historical firmware updates; (5) a value measuring a security level of the pre-update firmware version at the time of historical firmware updates; (6) a value measuring an operational performance level (e.g., SLA, device uptime, crashes, speed, etc.) of the pre-update firmware version at the time of historical firmware updates. Relatedly, the features of the first concurrently available update firmware version of the network device can include: (1) a number of customer-caused errors for the first concurrently available update firmware version at the time of historical firmware updates, weighted by error severity; (2) a number of internal errors for the first concurrently available update firmware version at the time of historical firmware updates, weighted by error severity; (3) an age of the first concurrently available update firmware version at the time of historical firmware updates; (4) a value measuring a popularity level of the first concurrently available update firmware version at the time of historical firmware updates; (5) a value measuring a security level of the first concurrently available update firmware version at the time of historical firmware updates; (6) a value measuring an operational performance level (e.g., SLA, device uptime, crashes, speed, etc.) of the first concurrently available update firmware version at the time of historical firmware updates; etc.

[0060] As noted above, in various examples, in making the firmware update likelihood score calculation, the machine learning model can consider customer-specific factors, such as: (a) a size of the customer site where the network device is deployed; (b) a customer-specific deployment configuration; (c) a customer-specific network device configuration; (c) a customer-specific vulnerability issue; (d) a customer-specific error; etc.

[0061] As noted above, the hardware processor 312 can dynamically train the machine learning model. For example, the hardware processor 312 can train the machine learning model on a weekly rolling basis, where the historical firmware update dataset includes data related to all historical firmware updates made by customers of the vendor in the last 6 months (or more specifically, all historical firmware updates made by customers of the vendor - for a particular type of network device - in the last 6 months). Through such training, the machine learning model can derive dynamic insights related to concurrent customer preferences from recent customer behavior related data.

[0062] Hardware processor 312 can execute instructions 320 to refine the machine learning model based on a comparison between the predicted acceptance of the firmware update by the machine learning model and historical firmware updates. In this way, the machine learning model can be refined to make predictions / recommendations that more closely track the behavior / preferences of the customer. In some examples, user feedback can be used to train / refine the machine learning model.

[0063] Figure 4 An example computing system 400 is shown that can be used to automatically recommend a compatible firmware version for a group of network devices, in accordance with the presently disclosed technology. In certain examples, computing system 400 can be incorporated into a vendor’s network management system (e.g., Aruba Central by HPE).

[0064] Referring now to Figure 4 Computing component 410 can be, for example, a server computer, a controller, or any other similar computing component capable of processing data. In Figure 4 example implementations, computing component 410 includes a hardware processor 412 and a machine-readable storage medium 414.

[0065] Hardware processor 412 and machine-readable storage medium 414 can be the same / similar to hardware processor 312 and machine-readable storage medium 314, respectively. Thus, machine-readable storage medium 414 can be encoded with executable instructions, such as instructions 416-420. In certain examples, hardware processor 412 can execute these instructions dynamically.

[0066] Hardware processor 412 can execute instructions 416 to compute, using a machine learning model, a firmware version score for an expected update firmware version for each network device in a group of network devices. In various examples, model parameters of the machine learning model can be adjusted in accordance with training described in connection with Figure 3

[0067] For an example first network device in the group of network devices, the first expected update firmware version can be a firmware version that is compatible on the first network device and is not currently installed on the first network device. Likewise, for a second network device in the group of network devices, the first expected update firmware version can also be a firmware version that is compatible on the second network device and is not currently installed on the second network device.

[0068] ​In certain examples, for the first network device, hardware processor 412 can compute a firmware version score for each prospective update firmware version (including the first prospective update firmware version) that is compatible on the first network device. Likewise, for the second network device, hardware processor 412 can compute a firmware version score for each prospective update firmware version (including the first prospective update firmware version) that is compatible on the second network device, and so on. For example, if the first prospective firmware version is not compatible on a third network device of the cluster of network devices, then for the third network device— hardware processor 412 can not compute a firmware version score for the first prospective firmware version, or hardware processor 412 can compute a third firmware version score that is a null score (e.g., a zero score).

[0069] An example first firmware version score for the group aggregation algorithm for the first network device and the first prospective update firmware version can include a first firmware update likelihood score that includes a numerical score (which can be positive or negative) that quantifies a likelihood that the first network device will be updated from the first pre-update firmware version installed on the first network device to the first prospective update firmware version. However, in other examples, the first firmware version score can be other types of numerical scores that quantify a value level (e.g., a customer-specific value level) for the firmware version.

[0070] Here, the group of network devices can be various types of groups. For example, the group of network devices can be all network devices of a customer, or all network devices of a particular type of customer (e.g., all access points of a customer). In certain examples, the group of network devices can include network devices of multiple networks (e.g., gateways that span multiple networks of a customer).

[0071] In some examples, the group of network devices can include a cluster of network devices of a customer. In examples where the group of network devices includes a cluster of network devices, hardware processor 412 can identify the group of network devices as a cluster of network devices prior to executing instructions 416. For example, an individual access point can identify a local access point as a virtual controller. Hardware processor 412 can identify access points that specify the same virtual controller as a single cluster of network devices that must all be compatible with each other. For mixed-type groups of network devices, hardware processor 412 can use several methods to identify the network topology. For example, hardware processor 412 can use LLDP (Link Layer Discovery Protocol) to identify Aruba APs connected to an Aruba switch.

[0072] As described above, a network device cluster can be a group / cluster of network devices of the same type (e.g., access points, switches, gateways, etc.) that operate together in a common network. For example, the group of network devices can include a cluster of access points that work together to provide wireless access to users at a customer location. While these access points (of the common network device cluster) can be of a common vendor, they can be different models with different sets of compatible firmware versions available for their use. As described above, a network device cluster should utilize the same firmware version in order to maintain optimal interoperability / compatibility. Unfortunately, one problem with the conventional manual firmware version recommendation process is that it often recommends 'incompatible' firmware versions for a network device cluster. For example, the process can recommend a firmware version that is only compatible on certain devices within the cluster. Thus, if the customer follows the recommendation, then some of their network devices can have incompatible firmware installed. Alternatively, the customer can follow the recommendation for the subset of network devices within the cluster that are compatible with the recommended firmware version, resulting in the network devices of the cluster having different firmware versions. Either of these scenarios is suboptimal for network operation / health and can result in: (a) a poor customer / user experience; (b) high troubleshooting costs for the vendor; (c) and a degradation in performance of the vendor's network management system that relies on customer network devices running on compatible firmware versions to provide optimal service. As described above, and will be described below, the example can improve the current firmware recommendation process by making network device cluster specific recommendations that ensure the recommended firmware version is compatible on all network devices of the network device cluster.

[0073] In certain examples, the machine learning model can compute an example first firmware version score for the combination of the first network device and the first prospective update firmware version based on features of the first firmware version and features of a first "pre-update" firmware version currently installed on the first network device. The features of the first pre-update firmware version can include: (1) a number of customer-induced errors for the first pre-update firmware version, weighted by error severity; (2) a number of internal errors for the first pre-update firmware version, weighted by error severity; (3) an age of the first pre-update firmware version; (4) a value measuring a popularity level of the first pre-update firmware version; (5) a value measuring a security level of the first pre-update firmware version; (6) a value measuring an operational performance level (e.g., SLA, device uptime, crashes, speed, etc.) of the first pre-update firmware version. Relatedly, the features of the first prospective update firmware version can include: (1) a number of customer-induced errors for the first prospective update firmware version, weighted by error severity; (2) a number of internal errors for the first prospective update firmware version, weighted by error severity; (3) an age of the first prospective update firmware version; (4) a value measuring a popularity level of the first prospective update firmware version; (5) a value measuring a security level of the first prospective update firmware version; (6) a value measuring an operational performance level (e.g., SLA, device uptime, crashes, speed, etc.) of the first prospective update firmware version; etc.

[0074] As noted above, in various examples, in computing the firmware version score for the first network device, the machine learning model can take into account customer-specific factors, such as: (a) a size of the customer site at which the first network device is deployed; (b) a customer-specific deployment configuration; (c) a customer-specific network device configuration; (c) a customer-specific vulnerability issue; (d) a customer-specific error; etc. In this way, the hardware processor 412 can provide firmware version update recommendations that are tailored to individual customers / customer sites.

[0075] Hardware processor 412 can execute instructions 418 to calculate an aggregated firmware version score across the group of network devices for at least one (and in some cases, each) prospective update firmware version. In some examples, the aggregated firmware version score for a first prospective update firmware version can include a sum of all calculated firmware version scores (including the example first firmware version score) for the first prospective update firmware version. In other examples, the aggregated firmware version score for a first prospective update firmware version can include an average of all calculated firmware version scores for the first prospective update firmware version. Here, by basing its recommendations on aggregated firmware version scores (rather than individual firmware version scores from individual network devices), hardware processor 412 can improve its recommendations for the group of network devices by recommending the prospective update firmware version that is best for the largest number of network devices.

[0076] As noted above, in some cases, if a prospective update firmware version is only compatible on a subset of the network devices of a network device cluster - then the firmware version score for the prospective update firmware version can be calculated only for the subset of network devices for which the prospective firmware update version is compatible. Thus, the aggregated firmware score for the prospective firmware version across the network device cluster can be an aggregated firmware score that is based on the firmware version scores calculated for the subset of network devices. In other cases, if a prospective update firmware version is only compatible on a subset of the network devices of a network device cluster - then a null / zero score for the prospective firmware update version can be calculated for the network devices for which the prospective firmware update version is not compatible. In these cases, when calculating the aggregated firmware score for the prospective firmware version across the network device cluster, hardware processor 412 can take into account these null / zero scores.

[0077] Hardware processor 412 can execute instructions 420 to recommend, for the group of network devices, the prospective update firmware version having the highest aggregated firmware version score among the compatible prospective update firmware versions (as used herein, a "compatible prospective update firmware version" can refer to a prospective update firmware version that is compatible on all network devices of the group of network devices). As noted above, the group of network devices can include a network device cluster of a customer. Thus, by recommending a prospective update firmware version that is compatible on all network devices of the group of network devices / network device cluster, hardware processor 412 can reduce the occurrence of incompatible firmware network devices - thereby improving the customer / user experience, reducing vendor support costs, improving the performance of the vendor's network management system, etc.

[0078] In various examples, prior to making the recommendation of instruction 420, hardware processor 412 can identify one or more of the compatible prospective update firmware versions as being unsafe (e.g., hardware processor 412 can measure the level of safety of the prospective update firmware versions according to a numerical score, and can identify all prospective update firmware versions that do not meet a“safety threshold” as being unsafe). In these examples, hardware processor 412 can execute instruction 420 to recommend, for the group of network devices, the compatible prospective update firmware version of the compatible prospective update firmware versions that has the highest aggregated firmware version score that has not been identified as unsafe.

[0079] In various examples, hardware processor 412 can provide its recommendation to at least one network device of the group of network devices. In certain examples, hardware processor 412 can provide its recommendation to a network / system administrator.

[0080] As noted above, in certain instances, hardware processor 412 can be incorporated into a vendor’s network management system (e.g., HPE’s Aruba Central). In these examples, hardware processor 412 can also automatically update at least one network device of the group of network devices to the recommended compatible prospective update firmware version. For example, according to an agreement between the vendor and a particular customer, hardware processor 412 can periodically (e.g., monthly, quarterly, etc.) push the recommended update to the customer’s network devices.

[0081] In certain examples, hardware processor 412 can receive user feedback based on its recommendation (to, e.g., a network administrator). Such user feedback can be considered in future recommendations and / or during machine model training.

[0082] Figure 5 A block diagram of an example computer system 500 in which various embodiments described herein can be implemented is shown. Computer system 500 includes a bus 512 or other communication mechanism for communicating information, a one or more hardware processors 504 coupled with bus 512 for processing information. Hardware processor(s) 504 can be, for example, one or more general purpose microprocessors.

[0083] Computer system 500 also includes a main memory 506, such as a random access memory (RAM), cache and / or other dynamic storage devices, coupled to bus 512 for storing information and instructions to be executed by processor 504. Main memory 506 also can be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor 504. Such instructions, when stored in storage media accessible to processor 504, render computer system 500 into a special-purpose machine that operates to perform the operations specified in the instructions.

[0084] The computer system 500 also includes a read only memory (ROM) 508 or other static storage device coupled to the bus 512 for storing static information and instructions for the processor 504. A storage device 510, such as a magnetic disk, optical disk, or USB thumb drive (flash drive), and the like, can be provided and coupled to the bus 512 for storing information and instructions.

[0085] The computer system 500 can be coupled via the bus 512 to a display 512, such as a liquid crystal display (LCD) (or touch screen), for displaying information to a computer user. An input device 514, including alphanumeric and other keys, can be coupled to the bus 512 for communicating information and command selections to the processor 504. Another type of user input device is cursor control 516, such as a mouse, a trackball, or cursor direction keys, for communicating direction information and command selections to the processor 504 and for controlling cursor movement on the display 512. In some embodiments, the same direction information and command selections can be implemented via receiving touches on a touch screen without a cursor.

[0086] The computing system 500 can include a user interface module to implement a GUI that can be stored in the mass storage device as executable software code that is executed by the computing device(s). This and other modules can include, for example, components, such as software components, object-oriented software components, class components and task components, processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables.

[0087] Generally, the words "component," "engine," "system," "database," "data store," and the like, used in the description above, can refer to logic embodied in hardware or firmware, or to a collection of software instructions, possibly having entry and exit points, written in a programming language, such as, for example, Java, C or C++. A software component can be compiled and linked into an executable program, or it can be interpreted on-the-fly by a software interpreter. A software component can be stored in dynamic link libraries, static or dynamic libraries, object or class libraries, or in any other memory locations or formats. The software component can also include decentralized software components, such as but not limited to, web services, distributed databases, and e-mail handling software. The software component can also include de-centralized software components, such as but not limited to, web services, distributed databases, and e-mail handling software. A software component can also be a software thread, a process, or a thread within a process. A software component can be compiled into an executable program or interpreter readable by a computer system. The software component can be written in any combination of one or more programming languages. Software components can communicate over various mechanisms, including application program interfaces, function calls, socket and http requests, and remote procedure calls.

[0088] Computer system 500 can implement the techniques described herein using customized hard-wired logic, one or more ASICs or FPGAs, firmware and / or program logic which in combination with the computer system causes computer system 500 to be a special purpose machine. According to one embodiment, the techniques herein are performed by computer system 500 in response to the

[0089] The term "non-transitory medium" and similar terms as used herein refer to any medium that stores the data and / or instructions that cause a machine to operate in a specific manner. Such non-transitory media can include non-volatile media and / or volatile media. Non-volatile media includes, for example, optical or magnetic disks, such as the storage device 510. Volatile media includes dynamic memory, such as the main memory 506. Common forms of non-transitory media include, for example, a floppy disk, a flexible disk, a hard disk, a solid-state drive, a magnetic tape, or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, NVRAM, any other memory chip or cartridge, and a networked version of any of the above.

[0090] Non-transitory media differ from transmission media, which are involved with transferring information from one place to another. Transmission media include coaxial cables, copper wire, and fiber optic cables, including the wires that comprise bus 512. Transmission media can also take the form of acoustic or light waves, such as those generated during radio and infrared data communications.

[0091] Computer system 500 also includes a communication interface 518 coupled to bus 512. Network interface 518 provides a two-way data communication coupling to one or more network links that are connected to one or more local networks. For example, communication interface 518 can be an integrated services digital network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, network interface 518 can be a local area network (LAN) card to provide a data communication connection to a compatible LAN (or WAN component to have WAN communication). Wireless links can also be implemented. In any such implementation, network interface 518 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.

[0092] Network links typically provide data communication through one or more networks to other data devices. For example, a network link can provide a connection through a local network to a host computer or to data equipment operated by an Internet Service Provider (ISP). The ISP in turn provides data communication services through the world wide packet data communication network now commonly referred to as the "Internet". Local networks and the Internet use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on network link and through communication interface 518, which carry the digital data to and from computer system 500, are example forms of transmission media.

[0093] Computer system 500 can send messages and receive data, including program code, through the network(s), network link and communication interface 518. In the Internet example, a server might transmit a requested code for an application program through the Internet, ISP, local network and communication interface 518.

[0094] The received code can be executed by processor 504 as it is received, and / or stored in storage device 510, or other non-volatile storage for later execution.

[0095] Each of the processes, methods, and algorithms described in the preceding sections can be embodied in, and fully or partially automated by, code components of one or more computer systems or computer processors, including the processor 504. The one or more computer systems or computer processors can also operate to support performance of the relevant operations described in connection with the techniques of this disclosure. Such components can be created using software 512 instructions that are executable by such computer systems or computer processors. "Code components" of programs, or instructions, can be stored in any desired computer-readable medium, including memories 514, storage devices 510, and / or other computer-readable media.

[0096] As used herein, a circuit can be implemented using any form of hardware, software, or combinations thereof. For example, one or more processors, controllers, ASICs, PLAs, PALs, CPLDs, FPGAs, logical components, software routines, or other mechanisms might be implemented to make up a circuit. In implementation, the various circuits described herein might be implemented as discrete circuits or the functions and features described can be shared in part or in total among one or more circuits. Even though individual features or elements of functionality can be described or claimed as part of a circuit, these features and elements can be shared among one or more circuits, and the description should not be limited to a single circuit unless specifically recited by the claim. In some embodiments, the circuit can comprise a plurality of circuits, such as one or more circuits distributed among a plurality of physically separate devices. Even though individual features or elements of functionality can be described or claimed as part of a circuit, these features and elements can be shared among one or more circuits, and the description should not be limited to a single circuit unless specifically recited by the claim. In some embodiments, the circuit can comprise a plurality of circuits, such as one or more circuits distributed among a plurality of physically separate devices. In some embodiments, the circuit can comprise a plurality of circuits, such as one or more circuits distributed among a plurality of physically separate devices.

[0097] As used herein, the term “or” can be construed in either an inclusive or exclusive sense. Furthermore, the description of resources, operations, or structures as being singular or multiple does not preclude their being integrated or multiple. Conditional language, such as “can,” “could,” “might,” or “may,” among others, unless specifically stated otherwise, or otherwise understood within the context as used, is generally intended to convey that certain embodiments include, while other embodiments do not include, certain features, elements, and / or steps.

[0098] Unless otherwise expressly stated, terms and phrases used in this document and variations thereof, unless otherwise indicated or unless the context clearly dictates otherwise, are to be interpreted in an open and inclusive manner. Adjectives such as “conventional,” “traditional,” “normal,” “standard,” “known,” and terms of similar meaning, when used to describe items used, for example, in making or using a product or process, should not be construed to limit such items to a given time period or to items that are already in use, but should be construed to include items that are or can become future items in the future. In some instances, phrases such as “one or more” or “at least” will not be construed in an exclusive sense unless specifically stated otherwise.

[0099] Data systems, platforms, and frameworks can apply machine learning (ML) or other models or algorithms (referred to herein as “ML models”) to data inputs to generate various analyses. Generally, these ML models can be trained to generate outputs based on inputs received during operation (i.e., when the ML model is placed in a production / speculative environment). Training of the ML model can involve providing known training data to the ML model that produces known outputs. Such training can teach the ML model to predict what output based on particular inputs. In order for the ML model to have accurate performance, the training data and operational data (i.e., real-world data) can share various features used to predict corresponding outputs.

[0100] As used herein, “one or more processing resources” for a network management system can refer to any number of physical processors on any number of devices.

Claims

1. A network management system comprising: one or more processing resources; and a non-transitory computer-readable medium coupled to the one or more processing resources, the non-transitory computer-readable medium having stored therein instructions that, when executed by the processing resources, cause the system to perform a method comprising: receiving a machine learning model, the machine learning model trained during a training phase by: receiving a historical firmware update dataset comprising firmware versions and features of a first cluster of network devices, and acceptance by customers associated with the firmware versions for the first cluster of network devices, updating model parameters based on the firmware versions of the first cluster of network devices, the features of the first cluster of network devices, and the acceptance by the customers of the firmware versions at the first cluster of network devices, and during an inference phase, enabling generation of a likelihood score of acceptance based on the model parameters generated during training; identifying a second cluster of network devices, wherein the second cluster of network devices comprises network devices associated with the customers that are of a same network device type for which the machine learning model was trained; during the inference phase, using the machine learning model to compute, for network devices of the second cluster of network devices, a plurality of firmware version scores for a plurality of prospective update firmware versions, the plurality of firmware version scores based on: features of a pre-update firmware version installed on the network devices of the second cluster of network devices, features of the prospective update firmware versions, and site-specific factors at which the second cluster of network devices is deployed; computing, for at least one of the plurality of prospective update firmware versions, an aggregated firmware version score across the second cluster of network devices; and recommending, for the second cluster of network devices, an update to the at least one of the plurality of prospective update firmware versions having a highest aggregated firmware version score among the compatible prospective update firmware versions.

2. The network management system of claim 1, wherein computing, for network devices of the second cluster of network devices, the plurality of firmware version scores for the plurality of prospective update firmware versions comprises: computing, for a first network device of the second cluster of network devices, a first firmware version score for a first prospective update firmware version based on features of a first pre-update firmware version installed on the first network device and features of the first prospective update firmware version; and computing, for a second network device of the second cluster of network devices, a second firmware version score for the first prospective update firmware version based on features of a second pre-update firmware version installed on the second network device and features of the first prospective update firmware version, and ​ ​ ​ wherein the first firmware version score computed for the first intended update firmware version includes a first firmware update likelihood score, the first firmware update likelihood score including a numerical score quantifying a likelihood that a network device in the first network device cluster will update from a first pre-update firmware version installed on the network device to the first intended update firmware version.

3. The network management system of claim 1, wherein computing, for network devices of the second network device cluster, the plurality of firmware version scores for the plurality of intended update firmware versions includes: computing, for a first network device of the second network device cluster, a first firmware version score for the first intended update firmware version based on characteristics of a first pre-update firmware version installed on the first network device and characteristics of the first intended update firmware version; and computing, for a second network device of the second network device cluster, a second firmware version score for the first intended update firmware version based on characteristics of a second pre-update firmware version installed on the second network device and characteristics of the first intended update firmware version, and wherein computing, for at least one intended update firmware version, an aggregated firmware version score across the second network device cluster includes: computing an aggregated firmware version score for the first intended update firmware version based on the first firmware version score and the second firmware version score.

4. The network management system of claim 1, wherein: the method further includes identifying one or more of the compatible intended update firmware versions as unsafe; and recommending, for the second network device cluster, an update to the plurality of intended update firmware versions having the highest aggregated firmware version score among the compatible intended update firmware versions includes recommending the plurality of intended update firmware versions having the highest aggregated firmware version score among the compatible intended update firmware versions that have not been identified as unsafe.

5. The network management system of claim 1, wherein the method further includes: updating at least one network device in the second network device cluster in accordance with the recommendation.

6. The network management system of claim 1, wherein the at least one intended update firmware version to which an update is recommended is not a latest firmware version.

7. The network management system of claim 1, wherein the at least one intended update firmware version to which an update is recommended is not a first best firmware version for network devices in the first network device cluster, but is a second best firmware version for the first network device cluster as a whole.

8. The network management system of claim 1, wherein the plurality of firmware version scores includes a numerical score quantifying a likelihood that a network device in the first network device cluster will update from a first pre-update firmware version installed on the network device to a first intended update firmware version.

9. The network management system of claim 1, wherein the processing resource further causes the system to perform a method including: adjusting the plurality of firmware version scores based on one or more of: (a) a size of a customer site at which the first network devices are deployed; (b) a customer-specific deployment configuration; (c) a customer-specific network device configuration; (d) a customer-specific vulnerability issue; and (e) a customer-specific bug.

10. A non-transitory computer-readable medium storing instructions that, when executed by one or more processing resources, cause the one or more processing resources to: receive a machine learning model, the machine learning model trained during a training phase by: receiving a historical firmware update dataset comprising firmware versions and features of a first cluster of network devices and acceptance by customers associated with the firmware versions for the first cluster of network devices, updating model parameters based on the firmware versions of the first cluster of network devices, the features of the first cluster of network devices, and the acceptance by the customers of the firmware versions at the first cluster of network devices, during an inference phase, enabling generation of a likelihood score of acceptance based on the model parameters generated during training; identify a second cluster of network devices, wherein the second cluster of network devices comprises network devices associated with the customers that are of a same network device type for which the machine learning model was trained; during the inference phase, for each network device of the second cluster of network devices, use the machine learning model to compute a plurality of firmware upgrade likelihood scores for a plurality of upgrade firmware versions, the plurality of firmware upgrade likelihood scores based on: (a) features of a pre-upgrade firmware version installed on the network device of the second cluster of network devices, (b) features of the upgrade firmware versions, and (c) site-specific factors at which the second cluster of network devices is deployed; compute an aggregated firmware version upgrade likelihood score across the second cluster of network devices for at least one of the plurality of upgrade firmware versions; and recommend, for the second cluster of network devices, the at least one of the plurality of upgrade firmware versions having a highest aggregated firmware upgrade likelihood score among the compatible upgrade firmware versions.

11. The non-transitory computer-readable medium storing instructions of claim 10, wherein, for a first network device of the second cluster of network devices, the machine learning model computes the plurality of firmware upgrade likelihood scores for the plurality of upgrade firmware versions compatible with the first network device based on features of a first pre-upgrade firmware version installed on the first network device and the plurality of upgrade firmware versions compatible with the first network device.

12. The non-transitory computer-readable medium storing instructions of claim 11, wherein a first firmware upgrade likelihood score comprises a numerical score quantifying a likelihood that the first network device of the second cluster of network devices will upgrade from the first pre-upgrade firmware version to a first upgrade firmware version. ​ 13. The non-transitory computer-readable medium storing instructions of claim 12, wherein the first firmware upgrade likelihood score comprises a probability that the first network device of the second network device cluster will upgrade from the first pre-upgrade firmware version to the first upgrade firmware version.

14. The non-transitory computer-readable medium storing instructions of claim 13, wherein the aggregate firmware version upgrade likelihood score across the second network device cluster for the first upgrade firmware version comprises an average firmware upgrade probability for the first upgrade firmware version across the second network device cluster.

15. The non-transitory computer-readable medium storing instructions of claim 11, wherein the features for the first pre-upgrade firmware version installed on the first network device comprise at least one of: a number of customer-induced errors for the first pre-upgrade firmware version, weighted by error severity; a number of internal errors for the first pre-upgrade firmware version, weighted by error severity; an age of the first pre-upgrade firmware version; a numerical value measuring a popularity level for the first pre-upgrade firmware version; and a numerical value measuring a security level for the first pre-upgrade firmware version.

16. A method for network management, comprising: generating a historical firmware update dataset related to a first network device cluster, the historical firmware update dataset comprising firmware versions and features of the first network device cluster, and customer acceptance associated with the firmware versions for the first network device cluster; and during a training phase, training a machine learning model using the historical firmware update dataset, the training phase being implemented by: updating model parameters based on the firmware versions of the first network device cluster, the features of the first network device cluster, and the customer acceptance of implementing the firmware versions at the first network device cluster, and during an inference phase, enabling generation of a likelihood score of acceptance based on the model parameters generated during training; during the inference phase, enabling the machine learning model to compute firmware update scores for a plurality of prospective update firmware versions on a second network device, wherein the firmware update scores are based on: (a) features of a pre-update firmware version installed on the second network device, (b) features of one or more concurrently available update firmware versions that are compatible on the second network device, and (c) site-specific factors in which a second network device cluster associated with the second network device is deployed.

17. The method of claim 16, wherein a first firmware update score for the second network device comprises a numerical score quantifying a likelihood that the second network device will update from the pre-update firmware version to a first concurrently available update firmware version.

18. The method of claim 16, wherein the method further comprises: based on the firmware update scores, predicting accepted firmware updates; ​ Based on the comparison of the predicted accepted firmware update to the historical firmware updates, refine the machine learning model.

19. The method of claim 16, wherein the first cluster of network devices comprises the same type of network device deployed across multiple customer networks.

20. The method of claim 16, wherein the machine learning model comprises a random forest model.

Images