Network device, computer networking method and computer readable storage medium

Abstract

The invention discloses a network device, a computer networking method and a computer readable storage medium. Techniques are described for classifying packet flows into different paths of a computer network based at least in part on an energy cost of each of the different paths. In one example, a network device determines a weighted equal cost multipath (wECMP) cost for each of different paths on which packets are forwarded. The different paths may include, for example, different links of different interfaces of the network device, different links connected with the network device, or an aggregated bundle of Ethernet links. The network device determines an energy cost for each of the paths. The network device modifies the wECMP cost for each path based at least in part on the energy cost of the path to obtain a modified wECMP cost. And the network equipment carries out load balancing on the packet on the path according to the modified wECMP cost.

Description

[0001] This application claims the benefit of U.S. Patent Application No. 18 / 823,384 filed on September 3, 2024, which claims the benefit of Indian Provisional Patent Application No. 202341060922 filed on September 11, 2023, the entire contents of each of which are incorporated herein by reference. Technical Field

[0002] The present disclosure relates to computer networks, and more particularly to improving energy efficiency in computer networks. Background Art

[0003] A computer network is a collection of interconnected network devices that can exchange data and share resources. In a packet-based network (such as Ethernet), network devices communicate data by dividing the data into variable-length blocks called packets, which are individually routed from a source device to a destination device across the network. The destination device extracts the data from the packet and assembles the data into its original form.

[0004] Certain devices (i.e., nodes), such as routers, maintain routing information that describes routes through a network. A "route" can generally be defined as the path between two locations on a network. Upon receiving an incoming packet, a router examines key information within the packet to identify the destination of the packet. Based on the destination, the router forwards the packet according to the routing information.

[0005] In a typical cloud data center environment, there is a large collection of interconnected routers and servers that provide computing and / or storage capacity to run various applications. For example, a data center may include facilities that host applications and services for subscribers (i.e., customers of the data center). For example, a data center may host all infrastructure equipment, such as networking and storage systems, redundant power supplies, and environmental controls. In a typical data center, clusters of storage servers and application servers (computing nodes) are interconnected via a high-speed switching fabric provided by one or more layers of physical network switches and routers. More complex data centers provide infrastructure throughout the world using subscriber support equipment located in various physical hosting facilities.

[0006] As data centers become larger, the energy usage of data centers increases. Some large data centers require a large amount of power (e.g., around 100 megawatts), which is enough to power many homes (e.g., around 80,000). Data centers can also run compute and data intensive application workloads (such as crypto mining and machine learning applications) and consume a lot of energy. Data center customers and data center providers themselves are pushing for more energy efficient data centers and / or applications. In order to be more energy efficient, traditional data centers may obtain some of their energy from renewable energy sources. However, the configuration of the data center and / or the applications running on the data center are constantly changing, and these traditional data centers are not able to dynamically increase their energy efficiency. Summary of the invention

[0007] In general, techniques are described for classifying packet flows into different paths of a computer network based at least in part on an energy cost of each of a plurality of different paths. In one example, a network device determines a weighted equal cost multi-path (wECMP) cost for each of the different paths over which a packet is forwarded. The different paths may include, for example, different interfaces of a network device, different links connected to a network device, or different links of an aggregated bundle of Ethernet links.

[0008] The network device determines the energy cost of each of the paths. In some examples, the network device may calculate the energy cost based on the number of hops of each path. In some examples, the network device may calculate the energy cost based on the geographical distance of each path. In other examples, the network device may calculate the energy cost by calculating a green bit cost metric for each path, which represents the energy cost of each path. For example, the green bit cost metric may be calculated based at least in part on physical parameters of each path, such as the distance, speed, or optical type of each path, device characteristics of one or more network devices implementing the path, such as a model of the network device or the type of one or more components of the network device, or the watt per bit cost of the path. The network device modifies the wECMP cost of each path based at least in part on the energy cost of the path to obtain a modified wECMP cost. The network device load balances packets on the path based on the modified wECMP cost.

[0009] In some examples, the techniques of the present disclosure may apply different network policies based on the utilization of a computer network. For example, during "off-peak" or low-usage times of a computer network, a computer system as described herein may apply a first network policy that considers the energy cost of each path when classifying packet flows to corresponding paths. Additionally, during "peak" or high-usage times of a computer network, the computer system may apply a second network policy that applies a shortest path or equal-cost algorithm to classify packet flows to corresponding paths without considering the energy cost of each path. In this way, the computer system may maximize energy-saving techniques when network congestion is not a significant factor and maximize network throughput when network congestion adversely affects the computer network.

[0010] The disclosed techniques may provide specific improvements to computer-related fields of computer networking having one or more practical applications. For example, the techniques disclosed herein may modify the use of ECMP or wECMP techniques to distribute packet flows to network paths, thereby taking into account the energy usage of each path. Thus, the disclosed techniques may significantly reduce the energy cost of forwarding network traffic without adversely affecting the load balancing of such network traffic across the network. Such techniques may reduce the energy usage of computer networks, reduce the environmental impact of computer networks, and reduce the cost of infrastructure used to power such computer networks.

[0011] In one example, the technology describes a network device comprising: processing circuitry in communication with a storage medium and configured to: determine a weighted equal cost multi-path (wECMP) cost for each of a plurality of paths over which packets of a plurality of packet flows are forwarded; determine an energy cost for each of the plurality of paths; modify the wECMP cost for each of the plurality of paths based at least in part on the energy cost of a corresponding path in the plurality of paths to obtain a modified wECMP cost; and load balance a plurality of packets over the plurality of paths based on the modified wECMP costs.

[0012] In another example, the technology describes a method comprising: determining, by a network device, a weighted equal cost multi-path (wECMP) cost for each of a plurality of paths on which packets of a plurality of packet flows are forwarded; determining, by the network device, an energy cost for each of the plurality of paths; modifying, by the network device, the wECMP cost for each of the plurality of paths based at least in part on the energy cost of a corresponding path in the plurality of paths to obtain a modified wECMP cost; and load balancing, by the network device, a plurality of packets on the plurality of paths based on the modified wECMP cost.

[0013] In another example, the technology describes a non-transitory computer-readable medium comprising instructions that, when executed, are configured to cause a processing circuit to: determine a weighted equal-cost multi-path (wECMP) cost for each of a plurality of paths over which packets of a plurality of packet flows are forwarded; determine an energy cost for each of the plurality of paths; modify the wECMP cost for each of the plurality of paths based at least in part on the energy cost of a corresponding path in the plurality of paths to obtain a modified wECMP cost; and load balance a plurality of packets over the plurality of paths based on the modified wECMP costs.

[0014] The details of one or more examples of the disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a block diagram illustrating an example network system of a data center having examples in which the techniques described herein may be implemented.

[0016] Figure 2 is a block diagram illustrating an example computing device using a network interface card with a separate processing unit, with which examples of the techniques described herein may be performed.

[0017] Figure 3 is a conceptual diagram illustrating a data center with servers that each include a network interface card with a separate processing unit controlled by an edge service platform.

[0018] Figure 4 is a schematic diagram illustrating an exemplary data center with servers connected through a switch fabric, wherein the NICs form a separate NIC fabric.

[0019] Figure 5 is a flow chart of example operations of an edge service controller for providing a self-learning green network.

[0020] Figure 6 is a flow chart of another example operation of an edge service controller for providing self-learning application workloads.

[0021] Figure 7 is a block diagram of an exemplary network system in accordance with the techniques of this disclosure.

[0022] Figure 8 is a block diagram of an exemplary aggregated link bundle in accordance with the techniques of this disclosure.

[0023] Fig. 9 is a block diagram of an exemplary network system that performs network traffic routing according to conventional ECMP techniques.

[0024] Fig.10 is a block diagram of an example network system that performs network traffic routing according to energy-aware ECMP routing using hop count in accordance with the techniques of this disclosure.

[0025] Fig.11 is a block diagram of an example network device that performs network traffic routing according to energy-aware ECMP routing using hop count in accordance with the techniques of this disclosure.

[0026] Fig.12 is a block diagram of an example network device that performs network traffic routing according to energy-aware ECMP routing using green bit costs in accordance with the techniques of this disclosure.

[0027] Fig.13 is a block diagram of an exemplary computing device in accordance with the techniques of this disclosure.

[0028] Fig.14 is a flow chart illustrating exemplary operations according to the technology of this disclosure.

[0029] Like reference numerals refer to like elements throughout the specification and drawings. DETAILED DESCRIPTION

[0030] Figure 1 is a block diagram illustrating an exemplary system 8 having an example computing infrastructure in which the techniques described herein may be implemented. In general, a data center 10 provides an operating environment for applications and services at one or more customer sites 11A to 11N (collectively referred to herein as "customer sites 11" or "customers 11"), which have one or more customer networks coupled to the data center through a service provider network 7. For example, the data center 10 may host infrastructure equipment such as networking and storage systems, redundant power supplies, and environmental controls. The service provider network 7 is coupled to a public network 4, which may represent one or more networks managed by other providers, and thus may form part of a large-scale public network infrastructure (e.g., the Internet). For example, the public network 4 may represent a local area network (LAN), a wide area network (WAN), the Internet, a virtual LAN (VLAN), an enterprise LAN, a layer 3 virtual private network (VPN), an Internet Protocol (IP) intranet operated by a service provider operating the service provider network 7, an enterprise IP network, or some combination thereof.

[0031] Although customer site 11 and public network 4 are primarily shown and described as edge networks of service provider network 7, in some examples, one or more of customer site 11 and public network 4 may be tenant networks within data center 10 or another data center. For example, data center 10 may host multiple tenants (customers) each associated with one or more virtual private networks (VPNs), where each tenant may implement one of customer sites 11.

[0032] The service provider network 7 provides packet-based connectivity to attached customer sites 11, data centers 10, and public networks 4. The service provider network 7 may represent a network owned and operated by a service provider to interconnect multiple networks. The service provider network 7 may implement Multi-Protocol Label Switching (MPLS) forwarding, and in such instances, may be referred to as an MPLS network or MPLS backbone. In some instances, the service provider network 7 represents multiple interconnected autonomous systems, such as the Internet, that provide services from one or more service providers.

[0033] In some examples, data center 10 may represent one of many geographically distributed network data centers. Figure 1 As shown in the example of , data center 10 can be a facility that provides network services to customers. Customers of the service provider can be collective entities such as enterprises and governments or individuals. For example, a network data center can host network services for several enterprises and end users. Other exemplary services may include data storage, virtual private networks, traffic engineering, file services, data mining, scientific or supercomputing, etc. Although shown as a separate edge network of the service provider network 7, elements of the data center 10 (such as one or more physical network functions (PNFs) or virtualized network functions (VNFs)) may be included in the core of the service provider network 7.

[0034] In this example, data center 10 includes storage and / or computing servers interconnected via a switching fabric 14 provided by one or more layers of physical network switches and routers, and servers 12A to 12X (referred to herein as "servers 12") are depicted as coupled to top-of-rack (TOR) switches 16A to 16N. Servers 12 may also be referred to herein as "hosts" or "host devices." Data center 10 may include many additional servers coupled to other TOR switches 16 of data center 10.

[0035] In the example shown, the switch fabric 14 includes interconnected top-of-rack (or other "leaf") switches 16A to 16N (collectively, "TOR switches 16") coupled to a distribution layer of chassis (or "spine" or "core") switches 18A to 18M (collectively, "chassis switches 18"). Although not shown, the data center 10 may also include, for example, one or more non-edge switches, routers, hubs, gateways, security devices (such as firewalls, intrusion detection, and / or intrusion prevention devices), servers, computer terminals, laptops, printers, databases, wireless mobile devices such as cellular phones or personal digital assistants, wireless access points, bridges, cable modems, application accelerators, or other network devices.

[0036] In this example, the TOR switch 16 and the chassis switch 18 provide redundant (multi-homed) connections to the IP structure 20 and the service provider network 7 to the server 12. The chassis switch 18 aggregates traffic flows and provides connections between the TOR switches 16. The TOR switch 16 can be a network device that provides Layer 2 (MAC) and / or Layer 3 (e.g., IP) routing and / or switching functions. The TOR switch 16 and the chassis switch 18 can each include one or more processors and memories, and can execute one or more software processes. The chassis switch 18 is coupled to the IP structure 20, which can perform Layer 3 routing to route network traffic between the data center 10 and the customer site 11 through the service provider network 7. The switching architecture of the data center 10 is only an example. For example, other switching architectures can have more or fewer switching layers.

[0037] Each of the servers 12 may be a computing node, an application server, a storage server, or another type of server. For example, each of the servers 12 may represent a computing device configured to operate according to the techniques described herein, such as an x86 processor-based server. The servers 12 may provide a network function virtualization infrastructure (NFVI) for the NFV architecture.

[0038] Servers 12 are hosted as endpoints of one or more virtual networks operating on a physical network represented here by IP fabric 20 and switch fabric 14. Although primarily described with respect to data center based switched networks, other physical networks such as service provider network 7 may underlie one or more virtual networks.

[0039] Servers 12 each include at least one network interface card (NIC) NIC 13A to 13X (collectively, "NIC 13"), each of which includes at least one port for sending and receiving packets over a communications link and exchanging packets therewith. For example, server 12A includes NIC 13A. NIC 13 provides a connection between the server and the switching fabric. In some examples, NIC 13 includes additional processing units in the NIC itself to offload at least some processing from a host CPU (e.g., a CPU of a server including the NIC) to the NIC, such as for performing supervisory and other high-level functions (referred to as a "data path").

[0040] In some examples, each of the NICs 13 provides one or more virtual hardware components for virtualized input / output (I / O). The virtual hardware component for I / O can be a virtualization of a physical NIC 13 ("physical function"). For example, in single root I / O virtualization (SR-IOV) described by the Peripheral Component Interface Special Interest Group SR-IOV specification, the PCIe physical function of a network interface card (or "network adapter") is virtualized to provide one or more virtual network interface cards as "virtual functions" for use by corresponding endpoints running on the server 12. In this way, virtual network endpoints can share the same PCIe physical hardware resources, and the virtual function is an example of a virtual hardware component. As another example, one or more servers 12 can implement Virtio (a paravirtualization framework available for example for the Linux operating system), which provides emulated NIC functions as a type of virtual hardware component. As another example, one or more servers 12 can implement Open vSwitch to perform distributed virtual multi-layer switching between one or more virtual NICs (vNICs) of hosted virtual machines, where such a vNIC can also represent a type of virtual hardware component. In some instances, the virtual hardware component is a virtual I / O (e.g., NIC) component. In some instances, the virtual hardware component is an SR-IOV virtual function, and a direct process user space access based on a data plane development kit (DPDK) can be provided to SR-IOV.

[0041] In some examples (including Figure 1 In the example shown in FIG. 1 , one or more of the NICs 13 may include multiple ports. The NICs 13 may be connected to each other via the ports and communication links of the NICs 13 to form a NIC fabric 23 having a NIC fabric topology. The NIC fabric 23 is a collection of NICs 13 connected to at least one other NIC 13.

[0042] The NICs 13 each include a processing unit to offload aspects of the data path. The processing unit in the NIC may be, for example, a multi-core ARM processor with hardware acceleration provided by a data processing unit (DPU), a field programmable gate array (FPGA), and / or an ASIC. Alternatively, the NIC 13 may be referred to as a SmartNIC or a GeniusNIC.

[0043] Edge service controller 28 may, in part, orchestrate services to be performed by processing unit 25 (e.g., Figure 2 The operation of the edge service platform within the NIC 13 is managed by: API-driven deployment of services 233 on the NIC 13; addition, deletion, and replacement of NICs 13 within the edge service platform; monitoring of services 233 and other resources on the NIC 13; and management of connections between various services 233 running on the NIC 13.

[0044] The edge service controller 28 can transmit information describing the services available on the NIC 13, the topology of the NIC fabric 13, or other information about the edge service platform to an orchestration system (not shown) or a network controller 24. Exemplary orchestration systems include OpenStack, VMWARE's vCenter, or Microsoft's System Center (System Center by MICROSOFT). Exemplary network controllers 24 include Juniper Networks' Contrail controller or Tungsten Fabric's controller. Additional information regarding the controller 24 operating in conjunction with other devices in the data center 10 or other software defined networks may be found in International Application No. PCT / US2013 / 044378, filed on June 5, 2013, and entitled “PHYSICAL PATH DETERMINATION FOR VIRTUAL NETWORK PACKET Flows,” and in U.S. Patent Application No. 14 / 226,509, filed on March 26, 2014, and entitled “Tunneled Packet Aggregation for Virtual Networks,” each of which is incorporated herein by reference as if fully set forth herein.

[0045] In some examples, the edge service platform determines the energy efficiency of the data center 10 and / or the energy efficiency of the data center 10 when deploying application workloads, and can invoke one or more actions to improve the energy efficiency of the data center 10. In some examples, the edge service controller 28 determines the energy efficiency of the data center 10 and utilizes the processing unit 25 of the NIC 13 to enhance the processing and network functions of the switch fabric 14 and / or the server 12 including the NIC 13 to improve the energy efficiency of the data center 10.

[0046] As an example, the edge service controller 28 may include an energy efficiency module 32 configured to determine the energy efficiency of the data center (referred to herein as the Green Quotient (GQ) of the data center) based on the energy usage of the data center 10 in relation to the percentage (g) of energy provided to the data center by one or more renewable energy sources (also referred to herein as “green energy”, such as solar energy, wind energy, hydroelectric energy, etc.). dc )).

[0047] exist Figure 1 In the example of FIG. 1 , the data center 10 may obtain energy from one or more power sources 30 of the data center 10. The power sources 30 may include renewable energy, non-renewable energy, or a combination of renewable energy and non-renewable energy. The energy efficiency module 32 obtains telemetry data including energy usage data of the data center 10, including current energy data (E C ) and the total energy usage (E ) of data center 10 when the data center is operating at full capacity (e.g., as determined from information provided by the manufacturer). T ). For example, the current energy usage data may include the amount of power currently used by one or more devices within the data center, or other metrics indicating the energy currently being consumed. The total energy usage of the data center 10 may indicate the maximum amount of power that one or more devices can use when operating at maximum capacity. The energy efficiency module 32 may obtain the energy usage data of the data center 10 directly from the servers 12 and / or network devices (e.g., chassis switches 18 and TOR switches 16) of the data center 10, from a centralized controller (which collects energy usage data from the servers 12 and / or network devices of the data center), and / or from static information provided by the manufacturers of the servers and / or network devices (e.g., the maximum capacity of the servers and / or computing nodes).

[0048] The energy efficiency module 32 determines the current energy usage (E C ) Total energy consumption (E T ) is referred to in this paper as the data center's "energy quotient" (E Q ), such as the following examples:

[0049]

[0050] The energy efficiency module 32 converts the energy quotient (E Q ) is compared to the percentage of energy (g) provided to data center 10 by one or more renewable energy sources (referred to herein as the "green energy percentage" of the data center). As an example of a green energy percentage, if power source 30 includes 20% renewable energy and 80% non-renewable energy, then the percentage of energy provided to data center 10 by one or more renewable energy sources is 20%. Based on the energy quotient (E Q ) and the green energy percentage (g) of the data center 10, the energy efficiency module 32 can calculate the green quotient (GQ) of the data center 10. dc ) specifies a value indicating whether the data center 10 is energy efficient, such as shown in the following example.

[0051]

[0052] In this example, if the energy quotient (E Q ) is less than or equal to the green energy percentage (g) of the data center 10 (e.g., E Q ≤g), the energy efficiency module 32 is the green quotient (GQ) of the data center 10 dc ) specifies a value (e.g., 100) indicating that the energy usage of the data center 10 is energy efficient (e.g., "green"). For example, if 20% of the energy provided or allocated to the data center 10 comes from renewable energy, then a value of 100 assigned to the Green Quotient of the data center indicates that the energy usage of the data center 10 does not exceed 20% of the renewable energy provided or allocated to the data center 10. If the Energy Quotient (E Q ) is greater than the green energy percentage (g) of the data center 10 (e.g., E Q >g), the energy efficiency module 32 is the green quotient (GQ dc ) specifies a value (e.g., Continuing with the above example, if 20% of the energy provided or allocated to data center 10 comes from renewable energy, the value of the green quotient of data center 10 (e.g., ) indicates the amount by which the energy usage of the data center 10 exceeds the renewable energy provided or allocated to the data center 10. As can be inferred from the formula, as the renewable energy percentage increases from g% to 100%, GQ dc Growing exponentially.

[0053] Green Quotient (GQ) based on Data Center 10 dc), the energy efficiency module 32 may cause the edge service controller 28 to invoke one or more actions to improve the energy usage of the data center 10. For example, the energy efficiency module 32 may cause the edge service controller 28 to modify the network protocol implemented by the devices within the data center 10 to another network protocol that is more energy efficient (e.g., uses less power) in response to determining that the Green Quotient of the data center 10 is a value (e.g., not 100) indicating that the data center 10 is not energy efficient. For example, network devices such as the chassis switch 18 and the TOR switch 16 and / or the virtual router implemented by the server 12 may use one or more network protocols, among which some network protocols may cause the devices (e.g., by processing more data) to consume more energy than other network protocols. In these examples, the energy efficiency module 32 can cause the edge service controller 28 to modify the network protocols implemented by the virtual routers and / or physical network devices (e.g., switches 18 and TOR switches 16) in the data center 10 (e.g., changing the implementation of a first network protocol to a second network protocol that is more energy efficient than the first network protocol) in response to determining that the green quotient of the data center 10 is a value (e.g., not 100) indicating that the data center 10 is not energy efficient.

[0054] In some examples, the energy efficiency module 32 can cause the edge service controller 28 to modify the tunneling protocol implemented in the underlying network of the data center to another tunneling protocol that is more energy efficient in response to determining that the green quotient of the data center 10 is a value (e.g., not 100) indicating that the data center 10 is not energy efficient. For example, physical and / or virtual network devices within the data center 10 can perform encapsulation and / or decapsulation of network packets to tunnel the network packets through the physical network infrastructure of the data center 10. Certain tunneling protocols may cause the device to consume more energy (e.g., by processing more data) than other tunneling protocols. In these examples, the energy efficiency module 32 can cause the edge service controller 28 to modify the tunneling protocol implemented by the network devices in the data center 10 (e.g., changing the implementation of the first tunneling protocol to a second tunneling protocol that is more energy efficient than the first tunneling protocol) in response to determining that the green quotient of the data center 10 is a value (e.g., not 100) indicating that the data center 10 is not energy efficient.

[0055] In some examples, the energy efficiency module 32 may cause the edge service controller 28 to automatically scale the port speed of the high-speed interface of one or more devices within the data center 10 in response to determining that the Green Quotient of the data center 10 is a value (e.g., not 100) indicating that the data center 10 is not energy efficient. For example, the NIC 13 may include one or more high-speed interfaces configured to exchange packets using links of an underlying physical network. These interfaces may include a port interface card having one or more network ports. In these examples, the energy efficiency module 32 may cause the edge service controller 28 to automatically scale the port speed of the high-speed interface of the NIC 13 in response to determining that the Green Quotient of the data center 10 is a value (e.g., not 100) indicating that the data center 10 is not energy efficient.

[0056] In some examples where devices within the data center 10 can achieve high availability to provide redundancy and reliability for packet-based communications, the energy efficiency module 32 can cause the edge service controller 28 to reduce the number of duplicate and / or equal cost multi-path (ECMP) paths used to provide high availability in response to determining that the green quotient of the data center 10 is a value (e.g., not 100) indicating that the data center 10 is not energy efficient. In some examples, a computing system as described herein modifies an equal cost multi-path (ECMP) algorithm or a weighted ECMP (wECMP) algorithm to consider the energy costs of different paths (e.g., using a target distance and / or a number of hops) when classifying packet flows to corresponding paths of different paths, as further described below. In some examples where high availability is disabled, the energy efficiency module 32 can cause the edge service controller 28 to shut down one or more network devices (e.g., half) configured to achieve high availability in response to determining that the green quotient of the data center 10 is a value (e.g., not 100) indicating that the data center 10 is not energy efficient.

[0057] In some examples, energy efficiency module 32 may cause edge service controller 28 to proactively activate a cooling system of the data center in response to determining that the Green Quotient of data center 10 is a value indicating that data center 10 is not energy efficient (e.g., not 100). In some examples, edge service controller 28 may activate the cooling system of the data center before the temperature of the data center exceeds a configured temperature threshold that automatically activates the cooling system of the data center.

[0058] In some examples, the energy efficiency module 32 may cause the edge service controller 28 to automatically expand the packet size (e.g., maximum transmission unit (MTU) size) of the packets in response to determining that the Green Quotient of the data center 10 is a value (e.g., not 100) indicating that the data center 10 is not energy efficient. For example, the payload size in each packet may affect energy usage. For example, a smaller MTU size of the packets may cause more packets to be generated, which may increase energy usage. In these examples, the energy efficiency module 32 may cause the edge service controller 28 to automatically expand the MTU size of the packets in response to determining that the Green Quotient of the data center 10 is a value (e.g., not 100) indicating that the data center 10 is not energy efficient.

[0059] The above actions are examples only. Energy efficiency module 32 may perform other actions, such as alerting an administrator that data center 10 is not energy efficient, and in some examples, providing recommended actions to reduce energy consumption of data center 10.

[0060] As another example, the energy efficiency module 32 is configured to determine the energy efficiency of the application workload deployed by the data center 10, referred to herein as the Green Quotient (GQ) of the application. app In these examples, energy efficiency module 32 obtains telemetry data (including energy usage data of data center 10) and determines a Green Quotient (GQ) of data center 10 based on the energy usage data, which indicates the energy efficiency of data center 10 when the application is running. dc2 ), and determining a green quotient (GQ) of the data center 10 indicating the energy efficiency of the data center 10 when the application is not running. dc1 ). The green quotient of the data center 10 when the application is not running can represent the energy efficiency of the stable data center 10. As described above, the energy efficiency module 32 can be based on the "energy quotient" (E Q ) is calculated to determine the green quotient of the data center. In this example, the energy efficiency module 32 can calculate the "energy quotient" (E Q For example, the energy efficiency module 32 may determine the current energy usage (E C ) Total energy consumption (E T ) and is based on the “energy quotient” (E Q ) is calculated to determine the Green Quotient (GQ) of the data center when the application is not running. dc1 The energy efficiency module 32 can also determine the current energy usage (E C ) Total energy consumption (E T ) and is based on the “energy quotient” (E Q) is calculated to determine the Green Quotient (GQ) of the data center when the application is running. dc2 In some examples, the Green Quotient (GQ) of a data center at the time the application is running is dc2 ) may represent the average value of the green quotient calculated for the data center 10 when the application runs for a period of time.

[0061] The energy efficiency module 32 calculates the green quotient (GQ) of the data center 10 when the application is running. dc2 ) and the Green Quotient (GQ) of the data center 10 when the application is not running dc1 ) increment (GQ Δ ) to determine the energy usage of your application, as shown in the following example.

[0062] GQ Δ =GQ dc2 –GQ dc1

[0063] The energy efficiency module 32 then calculates the energy usage (eg, GQ Δ ) is compared with the green energy percentage (g) of the data center 10, and a green quotient (GQ) of the application is calculated based on the comparison. app ) specifies a value indicating whether the data center 10 where the application is deployed is energy efficient, such as shown in the following example.

[0064]

[0065] In this example, if the energy usage of the application is less than or equal to the green energy percentage of data center 10 (e.g., GQ Δ ≤g), the energy efficiency module 32 is the green quotient (GQ app ) specifies a value (e.g., 100) indicating that the data center 10 in which the application is deployed is energy efficient. For example, if 20% of the energy provided or allocated to the data center 10 comes from renewable energy sources, then a value of 100 specified for the Green Quotient of the application indicates that the energy usage of the data center 10 in which the application is deployed does not exceed 20% of the renewable energy provided or allocated to the data center 10. If the energy usage of the application is greater than the percentage of energy provided to the data center by one or more renewable energy sources (e.g., GQ Δ >g), the energy efficiency module 32 is the green quotient (GQ app ) specifies a value indicating that the data center 10 where the application is deployed is not energy efficient (e.g., GQ Δ ). Continuing with the above example, if 20% of the energy provided or allocated to data center 10 comes from renewable energy, then a non-100 value specified for the Green Quotient of the application indicates that the energy usage of data center 10 in which the application is deployed exceeds 20% of the renewable energy provided or allocated to data center 10.

[0066] Green Quotient (GQ) based on application app ), the energy efficiency module 32 may cause the edge service controller 28 to invoke one or more actions to improve energy usage of the data center 10 where the application is deployed. In some examples, the energy efficiency module 32 may cause the edge service controller 28 to automatically deploy the application workload to one or more selected servers to reduce energy consumption of the data center 10 in response to determining that the green quotient of the application is a value (e.g., non-100) indicating that the data center 10 is not energy efficient. For example, some servers 12 may consume more energy due to overload of application workloads. In these examples, the energy efficiency module 32 may cause the edge service controller 28 to automatically deploy the application workload to the selected servers 12 in response to determining that the green quotient of the application is a value (e.g., non-100) indicating that the data center 10 is not energy efficient, or to deploy the application workload in a manner that reduces energy consumption of the data center 10 (e.g., by not deploying the application workload to servers that currently consume more energy).

[0067] In some examples, the energy efficiency module 32 may cause the edge service controller 28 to migrate the application workload to fewer servers and shut down the remaining unused servers in response to determining that the green quotient of the application is a value (e.g., non-100) indicating that the data center 10 is not energy efficient. For example, the application workload may be deployed on multiple servers 12 (e.g., servers 12A to 12C). In these examples, the energy efficiency module 32 may cause the edge service controller 28 to migrate the application workload to fewer servers (e.g., servers 12A and 12B) and shut down the remaining unused servers (e.g., server 12C) in response to determining that the green quotient of the application is a value (e.g., non-100) indicating that the data center 10 is not energy efficient.

[0068] In some examples, energy efficiency module 32 may cause edge service controller 28 to activate a cooling system in response to determining that the green quotient of the application is a value indicating that data center 10 is not energy efficient (e.g., not 100). In some examples, edge service controller 28 may proactively activate a cooling system of the data center 10 before the temperature of the data center 10 exceeds a configured temperature threshold.

[0069] In some examples, the energy efficiency module 32 may, in response to determining that the Green Quotient of the application is not energy efficient (e.g., not 100), cause the edge service controller 28 to disable scaling of the application if the energy usage of the application exceeds a configured threshold. For example, the application workload may be deployed on multiple servers 12. The deployment of additional application workloads on the servers 12 may cause the energy usage of the data center 10 to be not energy efficient. In these examples, the energy efficiency module 32 may, in response to determining that the Green Quotient of the application is a value (e.g., not 100) indicating that the data center 10 is not energy efficient, cause the edge service controller 28 to disable scaling of the application workload on the servers 12.

[0070] In some examples, a customer may prefer performance over energy efficiency for data center 10, or vice versa. In these examples, the customer may specify a requirement when the Green Quotient of an application exceeds a configured threshold. For example, one or more servers 12 may run a crypto mining application. In this example, a customer may specify a configured threshold for certain applications that enables data center 10 to operate with less energy efficiency to allow for higher performance. In this example, the customer may specify a configured threshold for certain applications that enables data center 10 to operate with less energy efficiency to allow for higher performance. app ) exceeds a configured threshold, the customer can specify a requirement to invoke one or more actions to improve the energy usage of the data center 10. In this way, the customer can specify a requirement to control whether the data center 10 is more energy efficient or has more performance.

[0071] The above actions are examples only. Energy efficiency module 32 may perform other actions, such as alerting an administrator that data center 10 where application workloads are deployed is not energy efficient, and in some examples, providing recommended actions to reduce energy consumption of data center 10.

[0072] In some examples, the energy efficiency module 32 can determine a pattern of energy usage of the data center 10 and / or one or more applications running on the data center 10. For example, the energy efficiency module 32 can implement one or more machine learning models (e.g., supervised learning, unsupervised learning, or other machine learning models) to determine a pattern of energy usage of the data center 10 and / or a pattern of energy usage of an application that repeatedly exceeds the green energy percentage (g) of the data center 10. Based on the determined pattern, the energy efficiency module 32 can cause the edge service controller 28 to increase the energy usage of the data center 10 before it becomes less energy efficient (e.g., before specifying the green quotient (GQ) of the data center 10 with a value other than 100). dc ) or the green quotient (GQ) of the application app) to improve energy usage of the data center 10 and / or the application. In some examples, the energy efficiency module 32 may cause the edge service controller 28 to preset the cooling system of the data center to activate before the temperature of the data center exceeds a configured temperature threshold. For example, the energy efficiency module 32 may determine that a particular application running on the servers 12A to 12C may repeatedly cause the temperature of the data center 10 to rise above the configured temperature threshold. In this example, the energy efficiency module 32 may cause the edge service controller 28 to preset the cooling system to activate when the application begins running on the servers 12A to 12C, before the temperature of the data center 10 rises above the configured temperature threshold.

[0073] In some examples, energy efficiency module 32 may cause edge service controller 28 to proactively migrate applications to fewer servers. For example, energy efficiency module 32 may determine that any application running on servers 12A-12C may repeatedly cause energy usage of data center 10 to exceed the green energy percentage of data center 10. In this example, energy efficiency module 32 may cause edge service controller 28 to migrate application workloads to fewer servers (e.g., servers 12A and 12B) before running the application on servers 12 in data center 10.

[0074] In accordance with the techniques of the present disclosure, one or more network devices (such as edge service controller 28, switches 16 and 18, server 12, or other types of computing devices) can classify packet flows into different paths of a computer network based at least in part on the energy cost of each of a plurality of different paths.

[0075] ECMP is a network routing strategy in which multiple packets sent to a single destination are load balanced across multiple paths of equal (or substantially equal) cost. In traditional routing, packets are sent along a single optimal path. However, when ECMP is used, network devices distribute packets across several equally efficient routes. ECMP works by identifying multiple paths with the same cost between a source and a destination. Conventionally, the cost is determined based on a metric for each path, such as the number of hops, bandwidth, or latency. Once multiple equal-cost paths are identified, the routing process distributes traffic across these paths in a balanced manner. In some examples, the distribution of packets is based on a hashing algorithm that takes into account factors such as source and destination IP addresses, port numbers, or packet size to ensure even distribution of traffic across multiple paths.

[0076] In some cases, if the local links of adjacent network devices toward the destination have unequal capacities, uniform traffic distribution across multiple paths may not be expected. Typically, the traffic distribution between two links is equal, and therefore the link utilization may be the same. However, if the capacity of the aggregated Ethernet bundle changes, the equal traffic distribution leads to an imbalance in link utilization. For example, this may occur when one link of the aggregated Ethernet bundle provides a much higher bandwidth than another link of the aggregated Ethernet bundle. In this case, wECMP implements traffic load balancing between equal-cost paths in proportion to the capacity of the local links. wECMP is a network routing strategy similar to ECMP, but wECMP attempts to balance traffic loads between multiple equal-cost paths in proportion to the different metrics or "weights" of each path. In some examples, the wECMP load balances traffic between multiple equal-cost paths in proportion to the capacity or bandwidth of each corresponding path.

[0077] As an example, chassis switch 18A determines a wECMP cost for each of the different paths 102A-102B (collectively, “paths 102”) over which to forward packets to server 12A. Figure 1 In the example of FIG. 1 , the different paths include a first path 102A from chassis switch 18A through TOR switch 16A to server 12A, and a second path 102B from chassis switch 18B through TOR switch 16N to server 12B. However, in other examples, the different paths may include, for example, different interfaces of a network device, different links connected to a network device, or different links of an aggregated bundle of Ethernet links between two network devices.

[0078] The chassis switch 18A determines the energy cost of each of the paths. In some examples, the chassis switch 18A calculates the energy cost based on the number of hops of each path. In some examples, the chassis switch 18A calculates the energy cost based on the geographical distance of each path. In other examples, the chassis switch 18A may calculate the energy cost by calculating a green bit cost metric for each path, which represents the energy cost of each path. As described in more detail below, for example, the green bit cost metric may be calculated based at least in part on a physical parameter of each path, such as the distance, speed, or optical type of each path, device characteristics of one or more network devices that implement the path, such as a model of the network device or a type of one or more components of the network device, or a watt per bit cost of the path.

[0079] The chassis switch 18A modifies the wECMP cost of each of the plurality of different paths based at least in part on the determined energy cost of the corresponding path to obtain a modified wECMP cost. The chassis switch 18A load balances the packets on the different paths according to the modified wECMP cost. Figure 1 In the example of FIG. 1 , chassis switch 18A load balances packets on paths 102A and 102B based on the modified wECMP costs. Thus, using the techniques described herein, chassis switch 18A can modify the wECMP costs of paths 102A and 102B to account for the energy cost of each corresponding path 102A.

[0080] In some examples, the techniques of the present disclosure may enable a network device to apply different network policies based on the utilization of a computer network. For example, during "off-peak" or low-usage times of a computer network, a network device as described herein may apply a first network policy that considers the energy cost of each path when classifying packet flows to corresponding paths. Additionally, during "peak" or high-usage times of a computer network, a network device may apply a second network policy that applies a shortest path or equal-cost algorithm to classify packet flows to corresponding paths without considering the energy cost of each path. In this way, network devices may maximize energy-saving techniques when network congestion is not a significant factor and maximize network throughput when network congestion adversely affects the computer network.

[0081] Figure 1 The foregoing example provides an example in which a chassis switch 18A of a switch fabric 14 within a data center 10 includes the energy cost of each corresponding path 102 when performing path selection for forwarding a packet across the switch fabric 14 to a server 12A. However, the techniques described herein may be used to enable a network device to consider the energy cost of a path during path selection in a variety of different scenarios. For example, the techniques of the present disclosure may be applied to enable a network device to consider the energy cost of a path during path selection between network devices within a data center 10, such as between two servers 12 or between a server 12 and a network device 16, 18, or additionally (or in the alternative), in forming two customer sites 11, a data center 10, another data center ( Figure 1 When selecting a path between network devices in the service provider network 7 (not shown) and / or the public network 4, the energy cost of the path is considered during path selection between network devices in the service provider network 7.

[0082] Figure 2 is a block diagram illustrating an example computing device using a network interface card with a separate processing unit, with which examples of the techniques described herein may be performed. Figure 2The computing device 200 may represent a real server or a virtual server, and may represent Figure 1 240 . In this example, the computing device 200 includes a bus 242 coupled to hardware components of the computing device 200 hardware environment. The bus 242 is coupled to a network interface card (NIC) 230 with SR-IOV function, a storage disk 246, and a microprocessor 210. In some cases, the front side bus can be coupled to the microprocessor 210 and the memory device 244. In some examples, the bus 242 can be coupled to the memory device 244, the microprocessor 210, and the NIC 230. The bus 242 can represent a high-speed peripheral component interface (PCI) (PCIe) bus. In some examples, a direct memory access (DMA) controller can control DMA transfers between components coupled to the bus 242. In some examples, components coupled to the bus 242 control DMA transfers between components coupled to the bus 242.

[0083] Microprocessor 210 may include one or more processors, each of which includes an independent execution unit ("processing core") that executes instructions consistent with an instruction set architecture. The execution units may be implemented as separate integrated circuits (ICs) or may be combined within one or more multi-core processors (or "multi-core" processors) each implemented using a single IC (i.e., a chip multiprocessor).

[0084] Disk 246 represents a computer-readable storage medium, including volatile and / or nonvolatile, removable and / or non-removable media implemented in any method or technology for storing information such as processor-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), EEPROM, flash memory, CD-ROM, digital versatile disk (DVD), or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage, or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by microprocessor 210.

[0085] The main memory 244 includes one or more computer-readable storage media, which may include random access memory (RAM) such as various forms of dynamic RAM (DRAM) (e.g., DDR2 / DDR3 SDRAM), or static RAM (SRAM), flash memory, or any other form of fixed or removable storage media that can be used to carry or store the required program code and program data in the form of instructions or data structures and that can be accessed by the computer. The main memory 144 provides a physical address space consisting of addressable memory locations.

[0086] The network interface card (NIC) 230 includes one or more interfaces 232 configured to exchange packets using the links of the underlying physical network. The interface 232 may include a port interface card having one or more network ports. The NIC 230 also includes, for example, on-card memory 227 for storing packet data. Direct memory access transmissions between the NIC 230 and other devices coupled to the bus 242 may be read from the memory 227, or may be written to the memory 227.

[0087] The memory 244 , NIC 230 , storage disk 246 , and microprocessor 210 provide an operating environment for executing a software stack of a hypervisor 214 and one or more virtual machines 228 managed by the hypervisor 214 .

[0088] Typically, a virtual machine provides a virtualized / guest operating system for executing applications in an isolated virtual environment. Because the virtual machine is visualized from the physical hardware of the host server, the executing application is isolated from both the host's hardware and other virtual machines.

[0089] An alternative to virtual machines is virtualized containers, such as those provided by the open source DOCKER container application. Like virtual machines, each container is virtualized and can remain isolated from the host and other containers. However, unlike virtual machines, each container can omit an independent operating system and only provide an application suite and application-specific libraries. The container is run by the host as an isolated user-space instance and can share an operating system and common libraries with other containers running on the host. As a result, a container may require less processing power, storage, and network resources than a virtual machine. As used herein, a container may also be referred to as a virtualization engine, a virtual private server, a silo, or a prison. In some instances, the techniques described herein with respect to containers and virtual machines or other virtualization components.

[0090] although Figure 2 The virtual network endpoints in the description are illustrated and described with respect to virtual machines, but other operating environments such as containers (e.g., Docker containers) can implement virtual network endpoints. Figure 2 ) may execute in kernel space 243 and may include, for example, Linux, Berkeley Software Distribution (BSD), a kernel of another Unix variant, or a Windows server operating system kernel commercially available from MICROSOFT.

[0091] The computing device 200 executes a hypervisor 214 to manage virtual machines 228. Exemplary hypervisors include Kernel-based Virtual Machine (KVM) for the Linux kernel, Xen, ESXi commercially available from VMWARE, Windows Hyper-V commercially available from MICROSOFT, and other open source and proprietary hypervisors. The hypervisor 214 may represent a virtual machine manager (VMM).

[0092] The virtual machine 228 can execute in the user space 245 and can host one or more applications, such as virtual network function instances. In some examples, the virtual machine 228 can host one or more VNF instances, where each of the VNF instances is configured to apply a network function to a packet.

[0093] Hypervisor 214 includes a physical driver 225 that uses physical functions 221 provided by network interface card 230. In some cases, network interface card 230 can also implement SR-IOV to enable sharing of physical network functions (I / O) between virtual machines 224. Each port of NIC 230 can be associated with a different physical function. Shared virtual devices (also referred to as virtual functions) provide dedicated resources so that each of virtual machines 228 (and corresponding guest operating systems) can access the dedicated resources of NIC230, so that the NIC appears to be a dedicated NIC for each of virtual machines 224. Virtual function 217 can represent a lightweight PCIe function that shares physical resources with physical function 221 and with other virtual functions 216. NIC 230 can have thousands of available virtual functions that comply with the SR-IOV standard, but for I / O intensive applications, the number of configured virtual functions is typically much smaller.

[0094] The virtual machine 228 includes a corresponding virtual NIC 229 that is provided directly into the guest operating system of the virtual machine 228, thereby providing direct communication between the NIC 230 and the virtual machine 228 via the bus 242 using the virtual functions assigned to the virtual machine. This can reduce the hypervisor 214 overhead associated with software-based VIRTIO and / or vSwitch implementations, where the hypervisor 214 memory address space of the memory 244 stores packet data and the copying of packet data from the NIC 230 to the hypervisor 214 memory address space and from the hypervisor 214 memory address space to the virtual machine 228 memory address space consumes cycles of the microprocessor 210.

[0095] The NIC 230 may also include a hardware-based Ethernet bridge or embedded switch 234. The Ethernet bridge 234 may perform layer 2 forwarding between virtual functions and physical functions of the NIC 230. Thus, in some cases, the bridge 234 provides hardware acceleration of packet forwarding between virtual machines 224 via the bus 242 and packet forwarding between the hypervisor 214 and any virtual machines 224 accessing the physical functions via the physical driver 225. The embedded switch 234 may be physically separate from the processing unit 25.

[0096] Computing device 200 may be coupled to a physical network switch fabric that includes an overlay network that extends the switch fabric from physical switches to software or "virtual" routers (including virtual router 220) on physical servers coupled to the switch fabric. A virtual router may be a physical server (e.g., Figure 1 The virtual router dynamically creates and manages one or more virtual networks that can be used for communication between virtual network endpoints (a process or thread executed by a server 12), or a combination thereof. In one example, the virtual routers each implement a virtual network using an overlay network that provides the ability to decouple the virtual address of an endpoint from the physical address (e.g., IP address) of the server on which the endpoint executes. Each virtual network can use its own addressing and security scheme and can be considered orthogonal to the physical network and its addressing scheme. Various techniques can be used to transmit packets within and across virtual networks on top of a physical network. At least some of the functions of the virtual router can be performed as one of the services 233.

[0097] exist Figure 2 In the exemplary computing device 200 of , the virtual router 220 executes within the hypervisor 214 using the physical function 221 for I / O, but the virtual router 220 can execute within the hypervisor, the host operating system, the host application, one of the virtual machines 228, and / or the processing unit 25 of the NIC 230.

[0098] Typically, each virtual machine 228 may be assigned a virtual address for use within a corresponding virtual network, wherein each of the virtual networks may be associated with a different virtual subnet provided by virtual router 220. For example, a virtual machine 228 may be assigned its own third virtual layer (L3) IP address to send and receive communications, but the virtual machine may not be aware of the IP address of the computing device 200 on which the virtual machine is running. In this manner, a "virtual address" is an address of an application that is distinct from the logical address of the underlying physical computer system (e.g., computing device 200).

[0099] In one implementation, the computing device 200 includes a virtual network (VN) agent (not shown) that controls the overlay of the virtual network of the computing device 200 and coordinates the routing of data packets within the computing device 200. Typically, the VN agent communicates with a virtual network controller for multiple virtual networks, which generates commands to control the routing of packets. The VN agent can operate as a proxy for control plane messages between the virtual machine 228 and a virtual network controller such as the controller 24. For example, a virtual machine can request to send a message using its virtual address via the VN agent, and the VN agent can in turn send the message and request to receive a message response for the virtual address of the virtual machine that initiated the first message. In some cases, the virtual machine 228 can call a program or function call presented by the application programming interface of the VN agent, and the VN agent can also handle the encapsulation of the message (including addressing).

[0100] In one example, a network packet (e.g., a layer 3 (L3) IP packet or a layer 2 (L2) Ethernet packet generated or consumed by an instance of an application executed by a virtual machine 228 within a virtual network domain) can be encapsulated in another packet (e.g., another IP or Ethernet packet) transmitted by a physical network. Packets transmitted in a virtual network may be referred to herein as "inner packets," while physical network packets may be referred to herein as "outer packets" or "tunnel packets." Encapsulation and / or decapsulation of virtual network packets within physical network packets may be performed by a virtual router 220. This function is referred to herein as tunneling and may be used to create one or more overlay networks. In addition to IPinIP, other exemplary tunneling protocols that may be used include IP over generic routing encapsulation (GRE), VxLAN, Multiprotocol Label Switching (MPLS) over GRE, MPLS over User Datagram Protocol (UDP), and the like.

[0101] As described above, a virtual network controller may provide a logical, centralized controller that facilitates the operation of one or more virtual networks. For example, a virtual network controller may maintain a routing information base, e.g., one or more routing tables that store routing information about a physical network and one or more overlay networks. The virtual router 220 of the hypervisor 214 implements network forwarding tables (NFTs) 222A to 222N for the N virtual networks for which its virtual router 220 operates as a tunnel endpoint. Typically, each NFT 222 stores forwarding information for the corresponding virtual network and identifies where a data packet will be forwarded to and whether the packet will be encapsulated in a tunneling protocol, such as with a tunnel header that may include one or more headers for different layers of the virtual network protocol stack. Each of the NFTs 222 may be an NFT for a different routing instance implemented by the virtual router 220 (not shown).

[0102] The edge service platform utilizes the processing unit 25 of the NIC 230 to enhance the processing and networking capabilities of the computing device 200. The processing unit 25 includes a processing circuit 231 to execute services orchestrated by the edge service controller 28. The processing circuit 231 may represent any combination of a processing core, an ASIC, an FPGA, or other integrated circuits and programmable hardware. In an example, the processing circuit may include a system on a chip (SoC) having, for example, one or more cores, a network interface for high-speed packet processing, one or more acceleration engines for specialized functions (e.g., security / cryptography, machine learning, storage), programmable logic, integrated circuits, and the like. Such a SoC may be referred to as a data processing unit (DPU). A DPU may be an example of a processing unit 25.

[0103] In the exemplary NIC 230, the processing unit 25 executes an operating system kernel 237 and a user space 241 for services. The kernel 237 may be a Linux kernel, a Unix or BSD kernel, a real-time OS kernel, or other kernels for managing the hardware resources of the processing unit 25 and the user space 241.

[0104] According to the techniques described in this disclosure, services 233 may include network, security, storage, data processing, co-processing, machine learning, or other services, such as energy efficiency services. Processing unit 25 may execute services 233 and edge service platform (ESP) agent 236 as processes and / or execute the services and edge service platform agent within a virtual execution element such as a container or a virtual machine. As described elsewhere herein, services 233 may enhance the processing power of a host processor (e.g., microprocessor 210) by, for example, enabling computing device 200 to offload packet processing, security, or other operations that would be performed by the host processor.

[0105] Processing unit 25 executes edge service platform (ESP) agent 236 to exchange data and control data with an edge service controller of the edge service platform. Although shown in user space 241, in some examples, ESP agent 236 can be a kernel module 237.

[0106] As an example, ESP agent 236 may collect and send to the ESP controller telemetry data generated by services 233 that describes traffic in the network, computing device 200 or network resource availability, resource availability (such as memory or core utilization) of resources of processing unit 25, and / or resource energy usage. As another example, ESP agent 236 may receive from the ESP controller service code for executing any service 233, service configuration for configuring any service 233, packets for injection into the network, or other data.

[0107] The edge service controller 28 manages the operation of the processing unit 25 by, for example, orchestrating and configuring services 233 performed by the processing unit 25; deploying services 233; adding, removing, and replacing NICs 230 within the edge service platform; monitoring of services 233 and other resources on the NIC 230; and managing connections between various services 233 running on the NIC 230. Exemplary resources on the NIC 230 include memory 227 and processing circuitry 231. In some examples, the edge service controller 28 may invoke one or more actions to improve energy usage of the data center 10 via managing the operation of the processing unit 25. In some examples, the edge service controller 28 may set a target green quotient for the processing unit 25, which causes the processing unit 25 to select or adjust a specific routing or tunneling protocol, a specific algorithm, an MTU size, an interface, and / or any service 233.

[0108] Figure 3 is a conceptual diagram showing a data center with servers each including a network interface card with a separate processing unit controlled by an edge service platform. A rack of computing nodes may correspond to Figure 1 The server 12, and the switch 308 may correspond to Figure 1 The switch of structure 14. Agent 302 or orchestrator 304 represents a processing unit (in Figure 3 304 and receives configuration information for the processing unit and sends telemetry and other information including the NIC of the processing unit to orchestrator 304. Network services, L4 to L7 services, and telemetry services (such as telemetry exporter 312) may represent examples of services 233. Orchestrator 304 may represent Figure 1 An example of an edge service controller 28.

[0109] The network automation platform 306 connects to and manages the network devices and orchestrator 304, through which the network automation platform 306 can utilize the edge service platform. For example, the network automation platform 306 can deploy network device configurations, manage the network, extract telemetry, and analyze and provide indications of network status.

[0110] The energy efficiency module 330 may represent Figure 1 In some examples, the energy efficiency module 330 can be based on the green quotient (GQ) of the data center 10. dc) is a value (e.g., not 100) indicating that the data center is not energy efficient, to instruct the network automation platform 306 to manage the network device configuration (such as modifying the network protocol implemented by the network device), modifying the tunneling protocol implemented by the network device, scaling the port speed, scaling the MTU size of the packet, reducing the number of duplicate / ECMP paths to achieve high availability, disabling the network device if high availability is disabled, etc. In some examples, the energy efficiency module 330 can instruct the network automation platform 306 to manage the cooling system to actively activate the cooling system and / or preset the configuration of the cooling system. In some examples, the energy efficiency module 330 can determine the green quotient (GQ) of the data center 10 based on the green quotient (GQ) of the data center 10. dc ) or the green quotient (GQ) of the application running on the data center 10 app ) is a determination of a value (e.g., non-100) indicating that the data center 10 is not energy efficient, to instruct the network automation platform 306 to manage network device configurations, such as managing the deployment of application workloads (e.g., migrating application workloads to fewer hosts and disabling unused hosts, disabling workload expansion).

[0111] Figure 4 is a schematic diagram showing an exemplary data center with servers connected via a switch fabric, wherein the NICs form an independent NIC fabric. Figure 4 The edge service controller 28 can manage one or more data centers, but for easier explanation, the edge service controller 28 is shown in FIG. Figure 4 A single data center is being managed. Figure 4 Examples of three different connections between the NICs in each rack and the TOR are shown. Rack 1 shows a traditional data center connection. In this example, each NIC is directly connected to a TOR port. In this configuration, applications (e.g., App1 and App2) can only communicate through the TOR, thereby increasing latency. Rack 2 shows a connection where application latency optimizes the connection. In this example, some NICs are connected to the TOR, while the remaining NICs are indirectly connected to the TOR. In this example, applications (e.g., App1 and App2) can communicate directly with each other (via back-to-back NIC connections). Rack 3 is similar to Rack 2 in connection (e.g., some NICs are connected to the TOR, while the remaining NICs are indirectly connected to the TOR), but high availability is increased using multiple connections between the NICs.

[0112] As an example, the energy efficiency module 32 of the edge service controller 28 can calculate the green quotient of the data center and determine that the green quotient of the data center is a value (e.g., non-100) indicating that the data center is not energy efficient. In this example, the energy efficiency module 32 can, in response to determining that the green quotient of the data center is a value indicating that the data center is not energy efficient, instruct the edge service controller 28 to manage the configuration of network devices in the rack 3 to reduce the number of duplicate and / or equal cost multi-path (ECMP) paths used to provide high availability. For example, the edge service controller 28 can send instructions to the agent 302 executed by the processing unit of the NICs 404A and 404B to reduce the number of duplicate and / or ECMP paths used to provide high availability. In some examples, the energy efficiency module 32 can, in response to determining that the green quotient of the data center is a value indicating that the data center is not energy efficient, instruct the edge service controller 28 to manage the configuration of network devices in the rack 3 to modify an equal cost multi-path (ECMP) algorithm or a weighted ECMP (wECMP) algorithm to consider the energy costs of different paths (e.g., using a target distance and / or a number of hops) when classifying packet flows to corresponding ones of the different paths, as further described below. In some examples of disabling high availability, the energy efficiency module 32 can, in response to determining that the green quotient of the data center is a value indicating that the data center is not energy efficient, cause the edge service controller 28 to shut down one or more network devices (e.g., half) configured to achieve high availability, for example, network devices including NICs 404C and 404D.

[0113] As another example, the energy efficiency module 32 can, in response to determining that the green quotient of the data center is a value indicating that the data center is not energy efficient, instruct the edge service controller 28 to manage the network device configuration in rack 1 to modify the tunneling protocol implemented by the NICs in rack 1 to implement a different tunneling protocol that is more energy efficient, e.g., NICs 406A and 406B.

[0114] As another example, the energy efficiency module 32 may instruct the edge service controller 28 managing one or more cooling systems 402 of the data center to actively activate a cooling system 402 and / or preset a configuration of the cooling system 402 in response to determining that the green quotient of the data center is a value indicating that the data center is not energy efficient.

[0115] As another example, energy efficiency module 32 may, in response to determining that the green quotient of the application is a value indicating that the data center is not energy efficient, instruct edge service controller 28 to manage deployment of application workloads in rack 2. For example, edge service controller 28 may deploy App2 in different servers (such as a server including NIC 408B) and manage network device configuration to enable NIC 408B to send and receive packets for App2.

[0116] Figure 5 is a flow chart of exemplary operations of the edge service controller 28 for providing a self-learning green network. Figure 5 In the example of , the energy efficiency module 32 of the edge service controller 28 can determine the green quotient of the data center and invoke one or more actions to improve energy usage of the data center based on the green quotient of the data center.

[0117] For example, the energy efficiency module 32 of the edge service controller 28 may obtain energy usage data of the data center (502). The energy usage data may include the current energy usage (E C ) and the total energy usage of the data center when the data center is running at full capacity (E T As an example, the edge service controller 28 can be configured to receive the data from a computing device (e.g., Figure 2 The edge service platform (ESP) agent (e.g., Figure 2 In some examples, the edge service controller 28 may obtain total energy usage data from information provided by the manufacturer of the computing devices in the data center, such as a manufacturer-provided file that specifies the maximum energy capacity of the computing devices.

[0118] The energy efficiency module 32 of the edge service controller 28 may determine an energy quotient (E quotient ) of the data center indicating a percentage of the total energy usage of the data center consumed by the current energy usage of the data center based on the energy usage data of the data center. Q )(504). The energy efficiency module 32 of the edge service controller 28 may compare the energy quotient of the data center to the percentage of energy provided to the data center by one or more renewable energy sources (eg, green energy percentage (g)) (506).

[0119] Based on the energy quotient (E Q ) and the green energy percentage (g) of the data center, the energy efficiency module 32 of the edge service controller 28 can specify the green quotient (GQ) of the data center. dc ) indicates whether the data center is energy efficient (508). For example, if the energy quotient (E Q ) is less than or equal to the green energy percentage (g) of the data center (e.g., E Q ≤g), the energy efficiency module 32 is the green quotient (GQ dc ) specifies a value (e.g., 100) indicating that the data center is energy efficient (e.g., “green”). If the energy quotient (E Q ) is greater than the green energy percentage (g) of the data center (e.g., EQ >g), the energy efficiency module 32 is the green quotient (GQ dc ) specifies a value indicating that the data center is not energy efficient (e.g., “not green”) (e.g., ).

[0120] If the Green Quotient of the data center indicates that the data center is not energy efficient (“NO” of step 510 ), energy efficiency module 32 may cause edge service controller 28 to invoke one or more actions to improve energy usage of the data center ( 512 ). For example, the energy efficiency module 32 can, in response to determining a green quotient indicating that the data center is not a green data center, cause the edge service controller 28 to modify a network protocol implemented by devices within the data center to another network protocol that is more energy efficient (e.g., uses less power), modify a tunneling protocol implemented in the underlying network of the data center to another tunneling protocol that is more energy efficient, automatically scale the port speeds of high-speed interfaces of devices within the data center, reduce the number of duplicate and / or equal cost multi-path (ECMP) paths used to provide high availability, modify an equal cost multi-path (ECMP) algorithm or a weighted ECMP (wECMP) algorithm to consider energy costs of different paths (e.g., using a target distance and / or number of hops) when classifying packet flows to corresponding paths of different paths, shut down one or more network devices (e.g., half) that are configured to achieve high availability when high availability is disabled, proactively activate a cooling system of the data center before the temperature of the data center exceeds a configured temperature threshold, automatically scale the MTU size of packets, or other actions for improving energy usage of the data center.

[0121] If the Green Quotient of the data center indicates that the data center is energy efficient ("YES" at step 510), energy efficiency module 32 may restart the process (eg, at step 502) to determine whether the data center is energy efficient, or stop the process.

[0122] Figure 6 FIG. 2 is a flow chart of an exemplary operation of an edge service controller 28 for providing a self-learning application workload. Figure 6 In the example of , the energy efficiency module 32 of the edge service controller 28 can determine the green quotient of the application and invoke one or more actions based on the green quotient of the application to improve energy usage of the data center where the application is deployed.

[0123] For example, the energy efficiency module 32 of the edge service controller 28 may obtain energy usage data of a data center (602). The energy usage data of the data center may include the current energy usage (E C ) and the total energy usage of the data center when the data center is running at full capacity (E TAs an example, the edge service controller 28 can be configured to receive the data from a computing device (e.g., Figure 2 The edge service platform (ESP) agent (e.g., Figure 2 In some examples, the edge service controller 28 may obtain total energy usage data from information provided by the manufacturer of the computing devices in the data center, such as a manufacturer-provided file that specifies the maximum energy capacity of the computing devices.

[0124] The energy efficiency module 32 of the edge service controller 28 may determine a first energy quotient (E quotient ) indicating a percentage of the total energy usage of the data center consumed by the current energy usage of the data center running the application based on the energy usage data of the data center when the application is running. Q )(604). The energy efficiency module 32 of the edge service controller 28 can compare the first energy quotient of the data center that is not running the application with the percentage of energy provided to the data center by one or more renewable energy sources (e.g., green energy percentage (g)) (606), and based on the energy quotient (E Q ) and the green energy percentage (g) of the data center, the energy efficiency module 32 of the edge service controller 28 can be the first green quotient (GQ) of the data center running the application dc2 ) specifies a value indicating the energy usage of the data center when the application is running (608). For example, if the energy quotient (E Q ) is less than or equal to the green energy percentage (g) of the data center (e.g., E Q ≤g), the energy efficiency module 32 is the first green quotient (GQ) of the data center running the application dc2 ) specifies a value (e.g., 100) indicating that the data center is energy efficient (e.g., “green”). If the energy quotient (E Q ) is greater than the green energy percentage (g) of the data center (e.g., E Q >g), the energy efficiency module 32 is the green quotient (GQ) of the data center running the application dc2 ) specifies a value (e.g., ). In some examples, the first green quotient is calculated based on an average value of green quotients of data centers when the application is running, determined over a period of time.

[0125] The energy efficiency module 32 of the edge service controller 28 can also determine a second energy quotient (E quotient ) of the data center running the application, indicating a percentage of the current energy usage of the data center not running the application to the total energy usage of the data center, based on the energy usage data of the data center when the application is not running. Q The energy efficiency module 32 of the edge service controller 28 can compare the energy quotient of the data center where the application is not running with the percentage of energy provided to the data center by one or more renewable energy sources (e.g., green energy percentage (g)) (612), and based on the energy quotient (E quotient) of the data center where the application is not running, Q ) and the green energy percentage (g) of the data center, the energy efficiency module 32 of the edge service controller 28 can be a second green quotient (GQ) of the data center that is not running the application dc1 ) specifies a value (614) that indicates energy usage of a data center when an application is not running.

[0126] The energy efficiency module 32 of the edge service controller 28 can calculate the first green quotient (GQ) of the data center running the application dc2 ) and the second green quotient (GQ) of the data center where no application is running dc1 ) to calculate the energy usage of the application (GQ Δ )(618). For example, the energy efficiency module 32 of the edge service controller 28 may calculate the difference (eg, GQ) between the first green quotient of the data center running the application and the second green quotient of the data center not running the application. dc2 –GQ dc1 The energy efficiency module 32 of the edge service controller 28 may compare the increment between the first green quotient and the second green quotient with the percentage (g) of energy provided to the data center by one or more renewable energy sources, and may determine, based on the comparison, a green quotient (GQ) of the application specifying a value indicating whether the data center where the application is deployed is energy efficient. app )(620). For example, if the energy usage of the application (GQ Δ ) is less than or equal to the green energy percentage (g) of the data center (e.g., GQ Δ ≤g), the energy efficiency module 32 is the green quotient (GQ app ) specifies a value (e.g., 100) indicating that the data center where the application is deployed is energy efficient (e.g., "green"). If the energy usage (GQ Δ ) is greater than the percentage of energy provided to the data center by one or more renewable energy sources (e.g., GQ Δ >g), the energy efficiency module 32 is the green quotient (GQ app ) specifies a value (e.g., other than 100) indicating that the data center where the application is deployed is not energy efficient.

[0127] If the Green Quotient of the application indicates that the data center where the application is deployed is not energy efficient ("No" of step 622), the energy efficiency module 32 may cause the edge service controller 28 to invoke one or more actions to improve the energy usage of the data center (624). For example, the energy efficiency module 32 may cause the edge service controller 28 to automatically deploy application workloads, migrate application workloads to fewer servers and shut down remaining unused servers, proactively activate the cooling system of the data center 10 before the temperature of the data center 10 exceeds a configured temperature threshold, disable the expansion of the application if the energy usage of the application exceeds a configured threshold, or other actions for improving the energy usage of the data center running the application in response to determining that the Green Quotient of the application is a value indicating that the data center 10 is not green (e.g., not 100).

[0128] If the Green Quotient of the application indicates that the data center where the application is deployed is energy efficient ("Yes" at step 622), energy efficiency module 32 may restart the process (eg, at step 626) to determine whether the data center where the application is deployed is energy efficient, or stop the process (626).

[0129] Figure 7 is a block diagram of an exemplary network system 700 according to the technology of the present disclosure. More specifically, Figure 7 An exemplary network system is depicted that computes wECMP paths using the green bit cost metric as described herein.

[0130] Equal Cost Multipath (ECMP) and Weighted Equal Cost Multipath (wECMP, also known as Unequal Cost Multipath) are two common load balancing techniques applied in network traffic engineering. The implementation of these features covers both the control protocol and data path forwarding of network nodes. Network control protocols such as BGP, OSPF, ISIS, LDP use path metric costs to arrive at multiple feasible paths. The data path then load balances the active traffic flows using various packet hashing schemes. In the case of ECMP, the traffic is equally distributed across multiple paths / links, while for wECMP, the traffic is unequally distributed based on the weight / balance specified by the control protocol.

[0131] Reducing the energy footprint of telecommunication networks is a key step in addressing global climate change. The present disclosure enhances load balancing mechanisms to achieve optimal energy efficient forwarding schemes in telecommunication networks.

[0132] Weighted ECMP / Unequal Cost Load Balancing

[0133] Network system 700 includes customer edge ("CE") devices 720 and 721, which are interconnected by network devices 701 to 709. In some examples, network devices 701 to 709 are Figure 1Examples of routers and switches 16, 18. Figure 7 As shown in the example, network devices 701 to 709 provide four multipaths between CE device 720 and device 721:

[0134] Path 730 701 → 703 → 705 → 708 → 709 (path cost: 40)

[0135] Path 732 701 → 702 → 704 → 707 → 709 (path cost: 80)

[0136] Path 740 701 → 702 → 704 → 706 → 708 → 709 (path cost: 90)

[0137] Path 742 701 → 702 → 704 → 705 → 708 → 709 (path cost: 100)

[0138] Conventionally, network devices not using the techniques of the present disclosure may install wECMP routing that may produce an approximate traffic distribution ratio based on the metric cost, which may result in a distribution of approximately (43%, 22%, 18%, 17%) across paths 730, 732, 740, and 742, respectively. Note that the metric cost is derived from the reference bandwidth / interface bandwidth.

[0139] However, if Figure 7 As shown in the example of , while paths 730 and 732 may be the shortest paths with a traditional metric, paths 740 and 742 may be more energy efficient. Fig.12 As explained in more detail below, network devices that include the energy efficiency of each path can modify the wECMP cost of each path to take into account the energy efficiency of each path. Fig.12 Explained in more detail, when considering the energy efficiency of each path, a more energy efficient distribution of network traffic calculated using the techniques of the present disclosure would be approximately (21%, 19%, 35.2%, 24%) across paths 730, 732, 740, and 742, respectively. Thus, because paths 740 and 742 have lower energy costs, even though paths 730, 732 may have smaller metric costs (e.g., shorter path lengths, etc.), the techniques of the present disclosure may enable a network device to load balance a greater proportion of packets on paths 740 and 742 than forwarding on paths 730, 732.

[0140] Figure 8 is a block diagram of an exemplary aggregated link bundle according to the techniques of the present disclosure. An aggregated link bundle can use the techniques described herein to achieve a traffic ratio. For example, Figure 7In the exemplary system, some flows are aggregated over the Ethernet link between network devices 702 and 704. Figure 8 As shown, the aggregated bundle can also achieve a wECMP traffic ratio relative to the link bandwidth. The aggregated bundle includes 5 links of unequal capacity {2×10G, 2×40G, 1×100G}. Therefore, according to traditional wECMP technology, two 10G links will have a metric cost of 10, two 40G links will have a metric cost of 5, and the 100G link will have a metric cost of 2. Using traditional wECMP technology, network devices that do not use the technology of the present disclosure will distribute traffic at a ratio of {2×1G:10%, 2×40G:40%, 1×100G:50%}.

[0141] Data path forwarding.

[0142] Data path processing of a wECMP flow includes the following steps: classifying incoming frames based on n-tuples; and constructing a hash key based on the n-tuples and looking up a hash table. The hash table result provides the appropriate distributed weight / balance for packet forwarding. Although there are multiple algorithms / optimizations for link selection with weights / balances programmed by the control protocol, this disclosure is not concerned with this aspect. Instead, the techniques described herein retain any existing mechanism for forwarding packets, but use the energy cost of each path to modify such weights / balances.

[0143] Fig. 9 is a block diagram of an exemplary network system that performs network traffic routing according to conventional ECMP techniques, i.e., for a system that does not employ the techniques of the present disclosure. In current routing protocols, ECMP distributes traffic equally among participating links, such as Fig. 9 As shown. Fig. 9 , network device 901 distributes traffic equally between network devices 902, 908, and 910. A typical router uses 5 to 10 kilowatts of power. Therefore, since packets traversing network device 902 additionally traverse network devices 903, 904, 905, and 906 to reach network device 907 (a total of 5 network devices), packets traversing network device 908 (a total of 2 network devices also traverse network device 909), and packets traversing network device 910 (which traverses only a single network device), packets traversing network device 902 may consume more power than other paths.

[0144] The traditional wECMP algorithm is modified using the hop count.

[0145] Fig.10 is a block diagram of an exemplary network system that performs network traffic routing according to energy-aware ECMP routing using hop counts in accordance with the techniques of the present disclosure. Fig.10The exemplary network system modifies the conventional wECMP traffic distribution algorithm to incorporate the number of hops to achieve more energy efficient packet routing, which results in Fig.10 The traffic distribution shown in Fig. 9 In the example of the traditional ECMP technology shown, traffic is forwarded to network devices 902, 908, and 910 in a uniform distribution of (33%, 33%, 33%), but the network device operating according to the technology of the present disclosure can distribute network traffic to network devices 1002, 1008, and 1010 while considering the energy consumption of each path, so that the distribution to network devices 1002, 1008, and 1010 is (17%, 34%, 49%). Fig.11 Describe in more detail how network devices use the hop count to determine the energy consumption of each path and modify the wECMP cost of each path.

[0146] Fig.11 is a block diagram of an exemplary computing device 1100 that performs network traffic routing according to energy-aware ECMP routing using hop counts in accordance with techniques of this disclosure. In some examples, Fig.11 Depicted is a computing device 1100 implementing a software model for overriding or modifying wECMP weights / balances, where the energy cost of each path is determined based on the number of hops of the corresponding path.

[0147] The computing device 1100 includes a routing protocol 1102, a routing information base (RIB) 1106, a hop count calculator 1110, a packet forwarding engine (also referred to herein as a "packet forwarding component"), and an application specific integrated circuit (ASIC) application 1104, each of which can be stored in a storage medium ( Fig.11 ) and stored by the processing circuit (not shown) of the computing device 1100 Fig.11 Software components executed by (not shown).

[0148] The routing protocol 1102 executes software instructions to implement one or more control plane networking protocols, such as a routing protocol for exchanging routing information with other routing devices and for updating the RIB 1106. The RIB 1106 may describe the topology of the computer network in which the computing device 1100 is located, and the routes through the shared trees in the computer network. The RIB 1106 may also describe the various routes within the computer network and the appropriate next hop for each route, for example, the adjacent routing devices along each route. The PFE ASIC application 1104 may provide interconnection between the RIB 1106 and the FIB 1108, such as programming the various routes learned by the RIB 1106 into the FIB 1108.

[0149] The computing device 1100 also includes a forwarding information base (FIB) 1108, a flow classification 1112, a forwarding (IP) lookup 1114, a wECMP path selection 1116, and an egress processing 1118, each of which can be part of or implemented by a hardware-based packet forwarding ASIC. The flow classification 1112 receives an incoming packet and determines the classification of a packet flow associated with the incoming packet. The forwarding (IP) lookup 1114 performs a lookup via the FIB 1108 to obtain an IP address to which a packet associated with the packet flow should be forwarded. In some examples, the forwarding (IP) lookup 1114 can use the classification of the associated packet flow to obtain an IP address via the FIB 1108. The FIB 1108 stores a forwarding data structure that associates a network destination with a next hop and an outgoing interface. The wECMP path selection 1116 performs path selection to determine one or more paths on which the incoming packet can be forwarded. In addition, the wECMP path selection 1116 determines the cost of each path by applying wECMP techniques to the determined one or more paths. Egress processing 1118 performs egress processing of incoming packets and forwards the packets to the next hop toward the packet's destination.

[0150] According to the techniques of this disclosure, and as described in more detail below, hop count calculator 1110 may determine an energy cost for each of the plurality of paths at least in part by determining a hop count for each corresponding path. Hop count calculator 1110 may communicate with PFE ASIC application 1104 to modify the wECMP cost of each path as determined by wECMP path selection 1116 and stored in FIB 1108 using the determined energy cost, thereby incorporating the energy cost of each path into the path selection performed by the packet forwarding ASIC hardware.

[0151] Deriving the number of hops in a network

[0152] Traceroute is an operational tool that allows discovery of packet paths and the delays experienced across those paths. In some examples, hop calculator 1110 can employ a modification of the traceout tool to identify the number of hops along each path in an ECMP scheme. As described herein, hop calculator 1110 can use the number of hops for each path as a weight in a wECMP scheme in order to incorporate the energy consumption of each path into the wECMP weighted scheme.

[0153] Green Path Selection Solutions

[0154] about Fig.10 For an exemplary network system, the green bit cost path calculated via the number of hops of each path will be:

[0155] Path 1 1001→1002→1003→1004→1005→1006→1007 (number of hops: 6, weight = 17)

[0156] Path 2 1001→1008→1009→1007 (number of hops: 3, weight = 34)

[0157] Path 3 1001→1010→1007 (number of hops: 2, weight = 49)

[0158] By programming the new weights / balance into Fig.11 The new traffic distribution of the ASIC hardware of the computing device 1100 is {17%, 34%, 49%}, while the traffic distribution calculated using the traditional ECMP technology is approximately {33%, 33%, 33%}. In this example, due to the reprogramming of the weights, the computing device 1100 can significantly improve the traffic pattern to select energy-efficient links by implementing the techniques disclosed herein.

[0159] Fig.12 is a block diagram of an example computing device 1200 that performs network traffic routing according to energy-aware ECMP routing using green bit costs in accordance with the techniques of this disclosure. In some examples, Fig.12 A computing device implementing a software model is depicted for overlaying wECMP weights / balances, wherein an energy cost for each path is calculated based on a green bit cost for the corresponding path. In some examples, the green bit cost is based on a geographic distance for each path. In some examples, instead of or in addition to the geographic distance, the green bit cost is based on one or more factors, including a watt per bit cost for each path, physical parameters for each path, and / or device characteristics of one or more network devices implementing each path.

[0160] The computing device 1200 may be used with Fig.11 The computing device 1100 of the embodiment of the present invention operates in a substantially similar manner. The computing device 1200 includes a routing protocol 1102, a routing information base (RIB) 1106, a bit cost tracker application 1210, and a packet forwarding engine (also referred to herein as a "packet forwarding component") application specific integrated circuit (ASIC) application 1104, each of which can be stored in a storage medium ( Fig.11 ) and stored by the processing circuit (not shown) of the computing device 1100 Fig.11 Software components executed by (not shown).

[0161] The computing device 1200 also includes a forwarding information base (FIB) 1108, flow classification 1112, forwarding (IP) lookup 1114, wECMP path selection 1116, and egress processing 1118, each of which may be part of or implemented by a hardware-based packet forwarding ASIC.

[0162] In accordance with the techniques of this disclosure, and as described in more detail below, a bit cost tracker application 1210 may determine an energy cost for each of a plurality of paths at least in part by determining a green bit cost for each corresponding path. In some examples, the green bit cost is based on one or more factors, including a cost per bit in watts for each path, a physical parameter of each path, and / or device characteristics of one or more network devices implementing each path, instead of or in addition to geographic distance. The bit cost tracker application 1210 may communicate with the PFE ASIC application 1104 to modify the wECMP cost of each path as determined by the wECMP path select 1116 and stored in the FIB 1108 using the determined energy cost, thereby incorporating the energy cost of each path into the path selection performed by the packet forwarding ASIC hardware.

[0163] Modification of the traditional wECMP algorithm using green bit cost

[0164] In this example, the bit cost tracker application 1210 modifies the wECMP traffic distribution algorithm based on a new metric called green bit cost. Once the green bit cost metric is available, multipath selection and distribution will be controlled based on this metric. The present disclosure describes techniques for deriving green bit cost based on simple operational and management tools (such as route trace / ping), and then using this derived information to improve traffic distribution logic.

[0165] Calculate the green bit cost of the considered link

[0166] The energy consumption of the link under consideration depends on many factors. It is usually a function of at least the physical parameters of the link and the device characteristics of the network equipment (such as a switch or router). Physical parameters include the distance between connected nodes (determines the laser power required to drive the signal), the link speed, or the type of optics. Device characteristics can include the type of switch / router and its internal design, as well as the components involved in packet forwarding, which affect the energy consumption of the device, for example, including various ASIC devices such as forwarding ASICs, structural ASICs, gearboxes, etc. As a general rule of thumb, modern ASICs manufactured with lower nanometer (nm) technology tend to be more energy efficient than devices from ten years ago. Physical parameters are the main factors contributing to the cost of green bits, and this implementation of the technology disclosed in the present invention addresses this aspect.

[0167] Deriving the green bit cost of links in the network

[0168] Traceroute is an operational tool that allows discovery of packet paths and the delays experienced across those paths. In some examples, the bit cost tracker application 1210 uses a modification of the traceroute tool to identify multiple paths and derive the path delays and geographic locations of the network devices under consideration. One such tool is nMap: Network Mapper. Based on traceroute and utilities such as ping, physical topological distances and propagation delays can be identified.

[0169] The techniques of this disclosure use the capabilities of such tools bundled in a "bit cost tracker" application (bit cost tracker application 1210) that collects the information needed to derive the green bit cost based on the multiple paths being optimized. The bit cost tracker application 1210 actively listens to the FIB telemetry data generated by the routing protocol. The FIB telemetry data allows the bit cost tracker application 1210 to learn the multiple paths installed by the routing protocol in the forwarding ASIC.

[0170] Green Path Selection Solutions

[0171] In some examples, the bit cost tracker application 1210 monitors the active paths being programmed in the system. The bit cost tracker application 1210 uses the geographic location and the latency calculated using the route tracing protocol to calculate a green bit cost metric associated with the FIB database. The bit cost tracker application 1210 updates the weights and pushes the newly calculated weights to the ASIC directly or via a broker application that interfaces with the ASIC.

[0172] continue Figure 7 For the example shown, the green bit cost path would be:

[0173] Path 730 701 → 703 → 705 → 708 → 709 (path cost: 40, green bit cost metric = 31)

[0174] Path 732 701 → 702 → 704 → 707 → 709 (path cost: 80, green bit cost metric = 35)

[0175] Path 740 701 → 702 → 704 → 706 → 708 → 709 (path cost: 90, green bit cost metric = 19)

[0176] Path 742 701 → 702 → 704 → 705 → 708 → 709 (path cost: 100, green bit cost metric = 27)

[0177] By programming the new weights / balance into the ASIC hardware, the new traffic distribution would be {21%, 19%, 35.2%, 24%}, whereas the traffic distribution using conventional wECMP techniques would be approximately {43%, 22%, 18%, 17%}. In this example, since the bit cost tracker application 1210 reprograms the weights, the techniques described herein significantly improve the traffic pattern for selecting energy efficient links. With path reservations in place, the green bit cost metric is limited by the maximum reserved bandwidth for that path. This approach can also be applied to ECMP paths.

[0178] In some examples, the application of the green bit cost metric can be controlled by network policy. As an example, under non-peak load time conditions, the green bit cost wECMP scheme can be applied, while at peak network utilization times, the conventional path cost wECMP weighting / balancing can be applied. In non-peak loads, an "eco-profile" is applied using the green bit cost techniques described herein, while in peak loads, a "performance-profile" is applied using traditional ECMP or wECMP techniques.

[0179] Aggregated Ethernet

[0180] A similar approach can be utilized on aggregated interfaces with unequal interface speeds and combined with adaptive load balancing. Here, the weights on the aggregated interface are adjusted based on the green bit cost metric. For example, a 400G short-haul optic may consume between 13W and 20W, while a 1G optic may consume 3W. With adaptive load balancing, when the traffic load is low, the 400G optical path is preferred because the watts / bit is minimized.

[0181] Modifications to this method

[0182] In some examples, a centralized controller can push new weights instead of out-of-the-box applications controlling wECMP weights like bit cost tracker application 1210, which can further improve efficiency. Simple hop counts can also be used as a simple metric to calculate weights.

[0183] Further enhancements can be made to introduce software policies for applying green bit cost weights. For example, a policy that uses green bit cost weights during learning network usage can be used, while during peak network times, a policy based on traditional weights based on path costs can be applied.

[0184] Advantages of the method described in this article

[0185] The techniques of the present disclosure provide significant advantages. For example, the techniques of the present disclosure do not require changes to existing routing protocols to calculate green bit costs. In addition, the bit cost tracker application 1210 described herein can be installed on any network device (such as a router or switch, or a server in the cloud). Utilizing standardized data path programming models such as FIB telemetry and SAI availability, the bit cost tracker application 1210 described herein can program the ability to update the weights of other vendors' devices. This allows any service provider to optimize the network immediately. In addition, this approach can be applied to single-hop aggregated Ethernet interfaces or multi-hop paths.

[0186] The technology of the present disclosure describes a green bit cost metric, which is a newly proposed metric that is superior to traditional technologies. The green bit cost metric can be applied to both single-hop (such as aggregated Ethernet interfaces) and multi-hop paths. The system as described herein can derive the green bit cost from platform data (such as optical type-LR, ER, etc.) and platform models. The green bit cost setting can be configuration driven. In addition, the technology of the present disclosure provides a method for deriving green bit costs based on existing network tools (such as route tracking, ping and / or geolocation data). In addition, the technology of the present disclosure provides a method for overlaying ECMP and wECMP weights based on the energy-efficient green quotient. In addition, the technology of the present disclosure enables a network system to apply a green hashing scheme based on network-wide attributes of an "eco mode" or a "performance mode".

[0187] Fig.13 is a block diagram of an exemplary computing device 1300 according to the techniques of this disclosure. In general, computing device 1300 may be Figure 1 The computing device 1300 includes a processing circuit 1302 for executing any one or more of the application 1322, the routing component 1350, or any other computing device described herein. Other examples of the computing device 1300 can be used in other instances to implement the techniques of the present disclosure. Although in Fig.13 1300 is shown in the examples as a single device, but in some examples, computing device 1300 may be implemented as a computing system distributed across multiple devices, such as in a distributed computing or cloud computing environment, for example.

[0188] Although for the purposes of this example Fig.13 1300, but a computing device operating according to the techniques of the present disclosure may be any component or system including one or more processors or other suitable computing environment for executing software instructions, and does not necessarily need to include, for example, Fig.131306; and in some examples, components such as one or more storage devices 1308 may not be co-located with other components or in the same chassis. In some examples, computing device 1300 may be implemented as a virtualized network function (VNF). In some examples, one or more aspects of computing device 1300 may run as one or more containers or as one or more applications within a virtual machine of a network function virtualization (NFV) platform, such as virtual input and output (VirtIO) and single root input / output virtualization (SR-IOV) network virtualization technologies, or on a bare metal server. In some examples, computing device 1300 is a physical network device, such as a switch, router, gateway, or other device that sends and receives network traffic.

[0189] like Fig.13 As shown in the example of , the computing device 1300 includes a processing circuit 1302, one or more input devices 1304, one or more communication units 1306, one or more output devices 1312, one or more storage devices 1308, and one or more user interface (UI) devices 1310. In one example, the computing device 1300 also includes one or more applications 1322 and an operating system 1316 that can be executed by the computing device 1300. Each of the components 1302, 1304, 1306, 1308, 1310, and 1312 is coupled (physically, communicatively, and / or operationally) for inter-component communication. In some examples, one or more communication channels 1314 may include a system bus, a network connection, an inter-process communication data structure, or any other method for transmitting data. As an example, the components 1302, 1304, 1306, 1308, 1310, and 1312 may be coupled via one or more communication channels 1314.

[0190] In one example, processing circuit 1302 is configured to implement functions and / or process instructions for execution within computing device 1300. In some examples, processing circuit 1302 includes one or more hardware-based processors. For example, processing circuit 1302 is capable of processing instructions stored in storage device 1308. Examples of processing circuit 1302 may include any one or more of a microprocessor, a controller, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or equivalent discrete or integrated logic circuits.

[0191] One or more storage devices 1308 may be configured to store information within computing device 1300 during operation. In some examples, one or more storage devices 1308 are described as computer-readable storage media. In some examples, one or more storage devices 1308 include temporary storage, which means that the primary purpose of storage device 1308 is not long-term storage. In some examples, one or more storage devices 1308 include volatile memory, which means that one or more storage devices 1308 do not maintain the stored content when the computer is turned off. Examples of volatile memory include random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), and other forms of volatile memory. In some examples, one or more storage devices 1308 store program instructions for execution by processing circuit 1302. In one example, one or more storage devices 1308 are used by software or applications running on computing device 1300 to temporarily store information during program execution.

[0192] In some examples, one or more storage devices 1308 also include one or more computer-readable storage media. One or more storage devices 1308 can be configured to store a larger amount of information than volatile memory. One or more storage devices 1308 can also be configured for long-term storage of information. In some examples, one or more storage devices 1308 include non-volatile storage elements. Examples of such non-volatile storage elements include magnetic hard disks, optical disks, floppy disks, flash memory, or electrically programmable memory (EPROM) or electrically erasable programmable (EEPROM) memory.

[0193] In some examples, computing device 1300 also includes one or more communication units 1306. In one example, computing device 1300 utilizes one or more communication units 1306 to communicate with external devices via one or more networks (such as one or more wired / wireless / mobile networks). One or more communication units 1306 may include a network interface, such as an Ethernet card, an optical transceiver, a radio frequency transceiver, or any other type of device that can send and receive information. Other examples of such network interfaces may include 3G / 4G / 5G and WiFi radios. In some examples, one or more communication units 1306 may include multiple high-speed network interface cards. In some examples, computing device 1300 uses one or more communication units 1306 to communicate with external devices. For example, in some examples, computing device 1300 uses communication unit 1306 to communicate with Figure 1The communication unit 1306 communicates with one of the network devices (such as chassis switch 18, tor switch 16, or one or more of the servers 12). In some examples, the communication unit 1306 includes a set of interface cards 1348A to 1348N (collectively referred to as "IFC 1348") (depicted separately for clarity).

[0194] In one example, computing device 1300 also includes one or more user interface devices 1310. In some examples, user interface device 1310 is configured to receive input from a user through tactile, audio, or video feedback. Examples of one or more user interface devices 1310 include a presence-sensitive display, a mouse, a keyboard, a voice response system, a camera, a microphone, or any other type of device for detecting commands from a user. In some examples, the presence-sensitive display includes a touch-sensitive screen. In some examples, a user such as an administrator of service provider network 150 can enter configuration data for computing device 1300.

[0195] One or more output devices 1312 may also be included in computing device 1300. In some examples, one or more output devices 1312 are configured to provide output to a user using tactile, audio, or video stimulation. In one example, one or more output devices 1312 include a presence-sensitive display, a sound card, a video graphics adapter card, or any other type of device for converting signals into an appropriate form understandable to a human or machine. Additional examples of one or more output devices 1312 include speakers, cathode ray tube (CRT) monitors, liquid crystal displays (LCDs), or any other type of device that can generate understandable output to a user.

[0196] Computing device 1300 may include operating system 1316. In some examples, operating system 1316 controls the operation of components of computing device 1300. For instance, in one example, operating system 1316 facilitates communication of one or more applications 1322 with processing circuit 1302, one or more communication units 1306, one or more storage devices 1308, one or more input devices 1304, one or more user interface devices 1310, and one or more output devices 1312. Applications 1322 may also include program instructions and / or data that may be executed by computing device 1300.

[0197] In some examples, processing circuit 1302 executes routing component 1350 and forwarding component 1370, where the routing component provides control plane functions, such as determining the route of received packets, and the forwarding component provides forwarding or data plane functions for computing device 1300 to send and receive traffic through a set of interface cards 1348 that typically have one or more physical network interface ports.

[0198] Routing component 1350 can communicate with other network devices (e.g., such as Figure 1 18, tor switch 16, or one or more of the servers 12) to establish and maintain a computer network (such as Figure 1 The computer network system 8 of the present invention is used to transmit network traffic between one or more customer sites 11. The routing protocol daemon (RPD) 1354 of the routing component 1350 can execute software instructions to implement one or more control plane networking protocols 1356. For example, the protocol 1356 can include one or more routing protocols, such as IGMP 1321 and / or Border Gateway Protocol (BGP) 1320, for exchanging routing information with other routing devices and for updating routing information 1352, Multi-Protocol Label Switching (MPLS) protocol 1315, PIM 1325, and other routing protocols. The protocol 1356 can also include one or more communication session protocols 1323, such as TCP, UDP, TLS, or ICMP. The protocol 1356 additionally includes BFD 1324.

[0199] Routing information 1352 may describe the topology of the computer network in which computing device 1300 resides, and may also include routes through shared trees in the computer network. Routing information 1352 may describe various routes within the computer network and the appropriate next hop for each route, e.g., adjacent routing devices along each route. Routing information 1352 may be programmed into a dedicated forwarding chip, a series of tables, a complex database, a linked list, a radix tree, a database, a flat file, or a variety of other data structures.

[0200] The forwarding component 1370 performs packet switching and forwarding of incoming data packets for transmission across the network. The forwarding component 1370 includes a forwarding information base (FIB) 1378 that stores forwarding data structures that associate network destinations with next hops and outgoing interfaces. Fig.131370, but forwarding component 1370 may include a central processing unit (CPU), a memory, and one or more programmable packet forwarding application specific integrated circuits (ASICs). In some examples, forwarding component 1370 may perform path selection when forwarding packets through a network. In some examples, forwarding component 1370 may apply path selection techniques, such as ECMP 1372 or wECMP 1374, to select one or more paths for forwarding packets on the network. In some examples, forwarding component 1370 may use ECMP 1372 or wECMP 1374 techniques to select one or more paths with similar performance metrics or "equal" costs in order to load balance network traffic across such one or more paths. In some examples, forwarding component 1370 may forward equal portions of network traffic across one or more paths. In some examples, forwarding component 1370 may forward unequal portions of network traffic across one or more paths.

[0201] According to the techniques of the present disclosure, routing component 1350 of computing device 1300 includes path energy analyzer 1380, which can help forwarding component 1370 consider the energy cost of each potential path when performing path selection. In some examples, path energy analyzer 1380 is Fig.11 In some examples, the path energy analyzer 1380 is Fig.12 Implementation of the bit cost tracker application 1210.

[0202] For example, forwarding component 1370 receives packets of multiple packet flows via IFC 1348 to forward to another network device. In some examples, the other network device is Figure 1 The forwarding component 1370 determines a plurality of paths on which the plurality of packet flows may be forwarded. In addition, the forwarding component 1370 calculates a wECMP cost of each of the plurality of paths on which packets of the plurality of packet flows are forwarded via wECMP 1374.

[0203] The path energy analyzer 1380 determines an energy cost for each of a plurality of paths on which packets of the plurality of packet flows may be forwarded. In some examples, the path energy analyzer 1380 approximates or estimates an energy cost for each path based at least in part on a number of hops of the corresponding path. For example, assuming that each network device corresponding to a hop in a path uses approximately the same amount of energy, a path with a relatively large number of hops may use more energy than a path with a relatively small number of hops.

[0204] In some examples, path energy analyzer 1380 approximates or estimates the energy cost of each path based at least in part on the geographic distance of the corresponding path. For example, the disclosed techniques may assume that a path with a relatively long length may use more energy than a path with a relatively short length due to inefficiencies in transmission or retransmission, a large number of devices required to form a path, inefficiencies in conversion between different data types or formats, etc.

[0205] In some examples, the path energy analyzer 1380 performs a route trace of the path for each of the plurality of paths to obtain a count of one or more hops of the path and an IP address of each of the one or more hops of the path. The path energy analyzer 1380 determines the geographic location of each of the one or more hops of each path based on the corresponding IP address of the corresponding hop. In addition, the path energy analyzer 1380 determines the geographic distance of the path based on the geographic location of each of the one or more hops of the path.

[0206] In some examples, the path energy analyzer 1380 approximates or estimates the energy cost of each path based at least in part on the physical parameters of each path in the multiple paths or the device characteristics of the network device that implements each path in the multiple paths. For example, different physical parameters and / or attributes of the path may consume different amounts of energy. The path energy analyzer 1380 may determine the energy cost of the path based at least in part on the distance of the link of the path, the speed of the link of the path, or the optical type of the link of the path. In addition, different device characteristics of the network devices that form the path may consume different amounts of energy. For example, due to the implementation of more energy-efficient technologies, newer network devices may generally (but not always) consume less power than older network devices. The path energy analyzer 1380 may determine the energy cost of the path based at least in part on the brand or model of the network device or the type of one or more components of the network device (e.g., the number of radios, antennas, interfaces, CPU models, the number of CPU or processor cores, the number of RAMs, GPU models, CPU / RAM / GPU utilization, power supplies, etc.). In some examples, path energy analyzer 1380 may query a database storing information describing one or more physical parameters of each path and / or device characteristics of the network devices forming each path to determine the energy cost of each path. In some examples, path energy analyzer 1380 may directly query the network devices forming part of the path to obtain the device characteristics of the network devices.

[0207] In some cases, the first path may have a relatively high energy cost, and the second path may have a relatively low energy cost. However, in this example, the first path may have a relatively high bandwidth, and the second path may have a low bandwidth. In this case, the first path can be used more energy efficiently than the second path, wherein the higher bandwidth of the first path offsets its higher energy consumption. In some examples, the technology of the present disclosure can explain the priority of the first path over the second path. For example, the path energy analyzer 1380 can determine the energy cost of each path based at least in part on the watt per bit cost of each path in a plurality of paths, wherein the watt per bit cost is the energy cost of the path (in watts) divided by the bandwidth of the path (in bits). In this way, the technology of the present disclosure can explain the priority of the first path over the second path, wherein the bit cost of the first path offsets the energy consumption of the first path.

[0208] The path energy analyzer 1380 modifies the wECMP cost of each of the multiple paths based at least in part on the energy cost of the corresponding path in the multiple paths to obtain a modified wECMP cost for each of the multiple paths. The forwarding component 1370 load balances the multiple packet flows over the multiple paths based on the modified wECMP cost. For example, the forwarding component 1370 can allocate a specific packet flow in the multiple packet flows to be forwarded along different paths in the multiple paths in proportion to the modified wECMP cost of each corresponding wECMP path. As another example, the forwarding component 1370 can forward packets of the specific packet flow along different paths in the multiple paths in proportion to the modified wECMP cost of each corresponding wECMP path.

[0209] Fig.14 is a flow chart illustrating an exemplary operation of the technology according to the present disclosure. More specifically, Fig.14 The operation of describes an exemplary operation of performing path selection while considering the energy cost of each potential path. Fig.13 The computing device 1300 describes Fig.14 However, according to the techniques described in this article, Fig.14 The operations may be performed by other devices described herein, such as by Figure 1 The process is performed by one of the network devices, such as a chassis switch 18, a tor switch 16, or one or more of the servers 12).

[0210] Forwarding component 1370 receives packets of multiple packet flows via IFC 1348 to forward to another network device. In some examples, the other network device is Figure 1The forwarding component 1370 determines a plurality of paths on which the plurality of packet flows may be forwarded. In addition, the forwarding component 1370 calculates a wECMP cost (1402) of each of the plurality of paths on which packets of the plurality of packet flows are forwarded via wECMP 1374.

[0211] Path energy analyzer 1380 determines an energy cost for each of a plurality of paths over which packets of the plurality of packet flows may be forwarded (1404). In some examples, path energy analyzer 1380 approximates or estimates an energy cost for each path based at least in part on a number of hops for the corresponding path. In some examples, path energy analyzer 1380 approximates or estimates an energy cost for each path based at least in part on a geographic distance for the corresponding path.

[0212] In some examples, the path energy analyzer 1380 performs a route trace of the path for each of the plurality of paths to obtain a count of one or more hops of the path and an IP address of each of the one or more hops of the path. The path energy analyzer 1380 determines the geographic location of each of the one or more hops of each path based on the corresponding IP address of the corresponding hop. In addition, the path energy analyzer 1380 determines the geographic distance of the path based on the geographic location of each of the one or more hops of the path.

[0213] The path energy analyzer 1380 modifies the wECMP cost of each of the plurality of paths based at least in part on the energy cost of the corresponding path in the plurality of paths to obtain a modified wECMP cost for each of the plurality of paths (1406). The forwarding component 1370 load balances the plurality of packet flows over the plurality of paths based on the modified wECMP cost (1408). For example, the forwarding component 1370 may allocate a particular packet flow in the plurality of packet flows to be forwarded along a different path in the plurality of paths in proportion to the modified wECMP cost of each corresponding wECMP path. As another example, the forwarding component 1370 may forward packets of the particular packet flow along a different path in the plurality of paths in proportion to the modified wECMP cost of each corresponding wECMP path.

[0214] The techniques described herein may be implemented in hardware, software, firmware, or any combination thereof. The various features described as modules, units, or components may be implemented together in an integrated logic device, or individually as discrete but interoperable logic devices or other hardware devices. In some cases, the various features of the electronic circuit may be implemented as one or more integrated circuit devices, such as an integrated circuit chip or chipset.

[0215] If implemented in hardware, the present disclosure may be directed to an apparatus such as a processor or an integrated circuit device such as an integrated circuit chip or chipset. Alternatively or additionally, if implemented in software or firmware, the techniques may be implemented at least in part by a computer-readable data storage medium that includes instructions that, when executed, cause a processor to perform one or more of the methods described above. For example, a computer-readable data storage medium may store such instructions for execution by a processor.

[0216] The computer-readable medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include computer data storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. In some examples, an article of manufacture may include one or more computer-readable storage media.

[0217] In some examples, computer-readable storage media may include non-transitory media. The term "non-transitory" may indicate that the storage medium is not incorporated into a carrier wave or propagating signal. In some examples, non-transitory storage media may store data that changes over time (e.g., stored in RAM or cache).

[0218] The code or instructions may be software and / or firmware executed by a processing circuit including one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Thus, the term "processor" as used herein may refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described in the present disclosure may be provided within a software module or a hardware module.

Claims

1. A network device, comprising: The processing circuit is in communication with the storage medium and is configured to: determining a weighted equal cost multipath (WECMP) cost for each of a plurality of paths over which packets of the plurality of packet flows are forwarded; determining an energy cost for each of the plurality of paths; modifying the wECMP cost for each of the plurality of paths based at least in part on an energy cost of a corresponding path of the plurality of paths to obtain a modified wECMP cost; as well as The plurality of packets are load balanced on the plurality of paths according to the modified wECMP cost.

2. The network device according to claim 1, wherein: The processing circuit is configured to determine an energy cost for each of the plurality of paths based at least in part on a watts per bit cost for each of the plurality of paths.

3. The network device according to claim 1, wherein: The processing circuit is configured to determine an energy cost for each of the plurality of paths based at least in part on a physical parameter of each of the plurality of paths or a device characteristic of a network device implementing each of the plurality of paths.

4. The network device according to claim 3, wherein: The physical parameter of each path of the plurality of paths includes a distance of a link of the path, a speed of the link of the path, or an optical type of the link of the path.

5. The network device according to claim 3, wherein: The device characteristics of the network device include a model of the network device or a type of one or more components of the network device.

6. The network device according to claim 1, in, The plurality of paths include a first path and a second path, wherein the length of the first path is shorter than the length of the second path, wherein the energy cost of the second path is less than the energy cost of the first path, and In order to load balance the plurality of packets on the plurality of paths according to the modified wECMP cost, the processing circuit is configured to forward a greater proportion of the plurality of packets on the second path than on the first path.

7. The network device according to any one of claims 1 to 6, wherein: The processing circuit is configured to determine an energy cost for each of the plurality of paths based at least in part on a hop count for each of the plurality of paths.

8. The network device according to claim 7, wherein: The processing circuit is configured to perform a route trace on each of the plurality of paths to obtain a hop count of each of the plurality of paths.

9. The network device according to any one of claims 1 to 6, wherein: The processing circuit is configured to determine an energy cost for each of the plurality of paths based at least in part on a geographic distance of each of the plurality of paths.

10. The network device according to claim 9, wherein: For each path in the plurality of paths, the processing circuit is configured to: performing a traceroute of the path to obtain an Internet Protocol (IP) address of each of the one or more hops of the path; determining a geographic location of each of the one or more hops of the path based on the IP address of each of the one or more hops of the path; as well as The geographic distance of the path is determined based at least in part on the geographic location of each of the one or more hops of the path.

11. A computer networking method, comprising: determining, by the network device, a weighted equal cost multipath (WECMP) cost for each of a plurality of paths over which packets of the plurality of packet flows are forwarded; determining, by the network device, an energy cost for each of the plurality of paths; modifying, by the network device, the wECMP cost of each of the plurality of paths based at least in part on an energy cost of a corresponding path of the plurality of paths to obtain a modified wECMP cost; as well as The network device load balances the plurality of packets on the plurality of paths according to the modified wECMP cost.

12. The computer networking method according to claim 11, wherein: The processing circuit is configured to determine an energy cost for each of the plurality of paths based at least in part on a watts per bit cost for each of the plurality of paths.

13. The computer networking method according to claim 11, wherein: The processing circuit is configured to determine an energy cost for each of the plurality of paths based at least in part on a physical parameter of each of the plurality of paths or a device characteristic of a network device implementing each of the plurality of paths.

14. The computer networking method according to claim 13, wherein: The physical parameter of each path of the plurality of paths includes a distance of a link of the path, a speed of the link of the path, or an optical type of the link of the path.

15. The computer networking method according to claim 13, wherein: The device characteristics of the network device include a model of the network device or a type of one or more components of the network device.

16. The computer networking method according to any one of claims 11 to 15, wherein: The processing circuit is configured to determine an energy cost for each of the plurality of paths based at least in part on a hop count for each of the plurality of paths.

17. The computer networking method according to claim 16, wherein: The processing circuit is configured to perform a route trace on each of the plurality of paths to obtain a hop count of each of the plurality of paths.

18. The computer networking method according to any one of claims 11 to 15, wherein: The processing circuit is configured to determine an energy cost for each of the plurality of paths based at least in part on a geographic path of each of the plurality of paths.

19. The computer networking method according to claim 18, wherein: For each path in the plurality of paths, the processing circuit is configured to: performing a traceroute of the path to obtain an Internet Protocol (IP) address of each of the one or more hops of the path; determining a geographic location of each of the one or more hops of the path based on the IP address of each of the one or more hops of the path; as well as The geographic distance of the path is determined based at least in part on the geographic location of each of the one or more hops of the path.

20. A computer-readable storage medium encoded with instructions for causing one or more programmable processors to be configured as a network device according to any one of claims 1 to 10, or to be configured to perform a computer networking method according to any one of claims 11 to 19.

Images