Network-internal calculation and control of network overloads based on delays in network-internal calculations

Abstract

Systems, devices, manufactured articles, and methods for network-internal computation and network congestion control based on delays in network-internal computations are disclosed. An exemplary device includes an interface circuit for accessing a data packet containing a header that has: (1) a destination address field for identifying a first network address of a target device capable of performing an action, and (2) a workload class field for identifying a workload class associated with the action. The exemplary device also includes a programmable circuit for using machine-readable instructions to perform the action in the device based on the workload class field, wherein the device has a second network address that is different from the first network address.Additionally, the exemplary programmable circuit is designed to modify a display field of the data packet to indicate that the action has been performed and to cause the interface circuit to forward the data packet with the modified display field to the target device.

Description

TECHNICAL FIELD OF DISCLOSURE

[0001] This disclosure relates generally to congestion control, and more particularly to in-network computation and control of network congestion based on delays in in-network computations. BACKGROUND

[0002] Edge environments (e.g., an edge, fog, multi-access edge computing (MEC), or Internet of Things (IoT) network) enable execution of a workload (e.g., execution of one or more computational tasks, execution of a machine learning model using input data, etc.) near endpoint devices requesting execution of the workload. Edge environments may include infrastructure, such as an edge platform, connected via networks, such as the internet, to cloud infrastructure, endpoint devices, and / or additional edge infrastructure. Edge platforms may be located closer to endpoint devices than cloud infrastructure, such as centralized servers. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is a block diagram of an example multi-layer network including an example edge layer, an example aggregation layer, and an example core layer. Fig. 2 is a block diagram of an exemplary implementation of an exemplary layered application device of the multi-layer network of the Fig. 1. Fig. 3 is a block diagram of an exemplary implementation of an exemplary feedback-capable device of the multi-layer network of the Fig. 1. Fig. 4A is an illustration of an example data packet including an example header according to examples disclosed herein. Fig. 4B is another illustration of an example packet of an example data packet including an example header according to examples disclosed herein. Fig. Figure 5 illustrates an overview of an edge cloud configuration for edge computing. Fig. Figure 6 illustrates operational layers between endpoints, an edge cloud, and cloud computing environments. Fig. Figure 7 illustrates an example approach for networking and services in an edge computing system. Fig. 8 is a flowchart representing example machine-readable instructions and / or example operations that may be executed, instantiated, and / or performed by an example programmable circuit to implement the layered application device of the Fig. 2 to control network congestion based on delays in in-network computation. Fig. 9 is a flowchart representing example machine-readable instructions and / or example operations that may be executed, instantiated, and / or performed by an example programmable circuit to implement the feedback-capable device of the Fig. 3 to perform intra-network processing. Fig. 10 is a flowchart representing example machine-readable instructions and / or example operations that may be executed, instantiated, and / or performed by an example programmable circuit to implement the feedback-capable device of the Fig. 3 to calculate a bit rate used to transmit a data packet flow based on delays in the network's internal computation. Fig. 11 is a flowchart representing example machine-readable instructions and / or example operations that may be executed, instantiated, and / or performed by an example programmable circuit to implement the feedback-capable device of the Fig. 3 for calculating control parameters of the feedback device based on delays in the network-internal calculation. Fig. 12 is a flowchart representing example machine-readable instructions and / or example operations that may be executed, instantiated, and / or performed by an example programmable circuit to implement the layered application device of the Fig. 2 to process a data packet for a congestion notification. Fig. 13 is a flowchart representing example machine-readable instructions and / or example operations that may be executed, instantiated, and / or performed by an example programmable circuit to implement the layered application device of the Fig. 2 to perform bitrate recovery after network congestion has subsided. Fig. 14 is a block diagram of an example processing platform including programmable circuitry structured to execute, instantiate, and / or perform the example machine-readable instructions and / or to perform the example operations of the Fig. 8, Fig. 9, Fig. 10, Fig. 11, Fig. 12 and / or 13 to the layered application device of the Fig. 2 and / or the feedback device of the Fig. 3 to be implemented. Fig. 15 is a block diagram of an exemplary implementation of the programmable circuit of the Fig. 14. Fig. 16 is a block diagram of another exemplary implementation of the programmable circuit of the Fig. 14. Fig. 17 is a block diagram of an example software / firmware / instruction distribution platform (e.g., one or more servers) for distributing software, instructions, and / or firmware (e.g., corresponding to the example machine-readable instructions of the Fig. 8, Fig. 9, Fig. 10, Fig. 11, Fig. 12 and / or 13) to client devices associated with end users and / or consumers (e.g., for licensing, selling and / or using), retailers (e.g., for selling, reselling, licensing and / or sublicensing), and / or original equipment manufacturers (OEMs) (e.g., for inclusion in products to be distributed, for example, to retailers and / or other end users, such as direct customers).

[0003] In general, the same reference numerals are used throughout the drawings and accompanying written descriptions to refer to the same or similar parts. The figures are not necessarily to scale. DETAILED DESCRIPTION

[0004] Edge environments and / or cloud environments may include in-network computing capable devices (INC-capable devices). In some examples, the characteristics of an INC-capable device include (i) performing line-rate processing with minimal latency, (ii) balancing network bandwidth with available computation on a per-service packet basis, and (iii) ensuring that the Quality of Service (QoS) and / or other parameters of Service Level Agreements (SLAs) are met despite limited compute, storage, and / or memory capabilities of the INC-capable device. In some examples, an INC-capable device has additional or alternative characteristics.As used here, “line speed” refers to a data transmission rate at which one or more bits are sent on a transmission medium such as a wire, a wireless communication channel, a wired network, etc.

[0005] INC-enabled devices (sometimes referred to as INC-enabled network nodes) include switches (e.g., programmable switches), switch chips, multiple switch systems on a chip (SoCs), routers, router chips, router SoCs, infrastructure processing units (IPUs), data processing units (DPUs), edge processing units (EPUs), and network-attached accelerator circuitry (e.g., FPGAs, ASICs, GPUs, XPUs, etc.). For example, network-attached accelerator circuitry can be implemented by a network device (e.g., a network interconnect, a network switch, etc.) including accelerator circuitry (e.g., an FPGA-based accelerator circuit) to accelerate demanding workloads such as distributed machine learning.Other examples of a network-attached accelerator circuit include off-the-shelf network devices that are modified to incorporate an accelerator circuit. Furthermore, an INC-enabled device can be a programmable network device (e.g., a programmable network switch) that provides reliable network transport with flexible congestion control to meet stringent latency requirements imposed by workloads running at the edge and / or the cloud. Example programmable switches can be programmed using one or more programming languages, such as the P4 programming language, the Software for Open Network in the Cloud (SONiC) programming language, and the Network Programming Language (NPL), among others.

[0006] In some examples, to achieve example characteristics of an INC-capable device, a device may (i) offload and reuse basic operations (e.g., a vector summation operation, a counting operation, a cache operation, a store operation, an acquisition operation, a hash table operation, etc.), ii) be resilient to failures (e.g., packet loss, link failure, etc.), and / or iii) trade off between a) preserving and / or sharing a portion of the network traffic for workload scalability and b) providing support for maintaining compatibility between legacy networking functionalities and policies. In some examples, a device may perform additional or alternative operations to achieve the characteristics of an INC-capable device.The transport layer of the OSI model (Open Systems Interconnection model) and the ability to handle congestion and flow control (CFC) of a network within the transport layer can be used to implement INC-capable devices.

[0007] Examples provided herein include an INC-informed CFC in a network fabric (e.g., in an end host device in a network, as well as in other devices in the network that are different from the end host). The disclosed example methods, apparatus, and articles of manufacture incorporate transport-layer CFC techniques for making a network (e.g., an edge network, a cloud network, etc.) lossless, efficient, and scalable by jointly performing CFC tasks across the network fabric and the end host devices. As used herein, "end host device" refers to a device from which a data packet flow originates in a network (e.g., a source computer) or in which a data packet flow terminates in a network (e.g., a destination computer).As used herein, “data packet flow” refers to a sequence of data packets transmitted from an end host device to a destination, which may be another end host device, a multicast group of devices, or a broadcast domain.

[0008] Additionally or alternatively, a data packet flow refers to a sequence of data packets sent from a particular end host device to a particular unicast, anycast, or multicast destination, and referred to by the end host device as a data packet flow. In some examples, a data packet flow may consist of all data packets in a particular transport connection or media stream. However, a data packet flow may not be mapped one-to-one to a transport connection (e.g., a transport connection may contain a first number of data packets that is greater than a second number of data packets of the data packet flow being transmitted over the transport connection). In still further examples, a data packet flow refers to a group of Internet Protocol (IP) data packets that pass through an observation point in a network during a particular time interval.

[0009] Fig. 1 is a block diagram of an example multi-layer network 100 including an example edge layer 102, an example aggregation layer 104, and an example core layer 106. In the example of Fig. 1, the multi-layer network 100 is an edge network. In additional or alternative examples, the multi-layer network 100 may be a cloud network. In the example of Fig. 1, the edge layer 102 includes devices that are closer to an endpoint device (e.g., a consumer device, a producer device, etc.) (e.g., an autonomous vehicle, user equipment, commercial and industrial equipment, a video capture device, a drone, a smart city and / or building device, a sensor, an IoT device, etc.) than the aggregation layer 104 and the core layer 106.

[0010] In the example of Fig. 1, computing, memory, and data security resources offered in the edge layer 102 are advantageous for providing extremely low latency response times for services and functions used by endpoint data sources, as well as for reducing network backhaul traffic from the edge layer 102 to the core layer 106, thereby improving energy consumption and overall network utilization, among other benefits. In the multi-layer network 100 of the Fig. 1, computing, memory, and storage resources may be scarce and generally decrease by location (e.g., fewer processing resources are available in a consumer's endpoint devices than in devices in the edge layer 102 and in devices in the core layer 106). However, the closer a location is to an endpoint (e.g., a user equipment (UE)), the more space and performance may be constrained.

[0011] In the illustrated example of Fig. 1, the edge layer 102 includes exemplary edge computing devices 108 and exemplary edge network devices 110. The edge layer 102 includes, for example, first edge computing devices 108 1a up to 108 n (which is known as W 1a to W n referred to) in communication with a first edge network device 110 1A (which is known as S 1AIn addition, the edge layer 102 includes second edge computing devices 108 1a up to 108 k (which is known as W 1b to W k referred to) in communication with a second edge network device 110 K (which is known as S K In the example of the Fig. 1, the edge layer 102 contains third edge computing devices 108 1c up to 108 l (which is known as W 1c to W l referred to) in communication with a third network device 110 1B (which is known as S 1B In addition, the edge layer 102 includes fourth edge computing devices 108 1d up to 108 m (which is known as W 1d to W m referred to) in communication with a fourth edge network device 110 L (which is known as S L referred to as).

[0012] In the illustrated example of Fig. 1, one or more of the edge computing devices 108 may be implemented by a gateway, a server (e.g., an on-premises server), or network devices that are physically located near one or more endpoint devices. In some examples, at least one of the edge computing devices 108 includes an accelerator circuit. In the example of Fig. 1, one or more edge computing devices 108 implement one or more services and / or one or more functions that can be used by endpoint data sources.

[0013] In the illustrated example of Fig. 1, one or more of the edge network devices 110 may be implemented by, among others, an FPGA with a network interface control unit (sometimes referred to as a network interface circuit), an XPU, a programmable network switch, a middle box. In some examples, at least one of the edge network devices 110 includes an accelerator circuit. In the example of Fig. 1, one or more of the edge network devices 110 facilitate communications between one or more edge layer devices 102 and one or more aggregation layer devices 104.

[0014] In the illustrated example of Fig. 1, the devices offered in the aggregation layer 104 facilitate the transfer of data between the edge layer 102 and the core layer 106. For example, the devices offered in the aggregation layer 104 forward the network traffic from the core layer 106 to the edge layer 102. Furthermore, the devices offered in the aggregation layer 104 collect the network traffic from the edge layer 102 and forward the network traffic to the core layer 106.

[0015] In the illustrated example of Fig. 1, the aggregation layer 104 includes exemplary aggregation network devices 112. The aggregation layer 104 includes, for example, a first aggregation network device 1121 (denoted by A1) in communication with the first edge network device 110 1A and the second edge network device 110 KIn addition, the aggregation layer 104 includes a second aggregation network device 112 N (which is known as A N referred to) in communication with the third edge network device 110 1B and the fourth edge network device 110 L . In the example of Fig. 1, one or more of the aggregation network devices 112 may be implemented by, among others, a base station, a radio processing unit, a network hub, a regional data center (DC), or local area network equipment. Furthermore, one or more of the aggregation network devices 112 may include or be implemented by an FPGA with a network interface controller, an XPU, a programmable network switch, or a middle box. In some examples, at least one of the aggregation network devices 112 includes an accelerator circuit.

[0016] In the illustrated example of Fig. 1, one or more of the aggregation network devices 112 bundle multiple network connections to combine them into a single communication link. Therefore, the aggregation network devices 112 provide increased bandwidth and better network performance. In the example of Fig. 1, the aggregation network devices 112 use link aggregation protocols, such as a Link Aggregation Control Protocol (LACP) and Ethernet aggregation, to combine multiple communication links into a single logical connection. As such, the aggregation network devices 112 provide flexibility and scalability, allowing for network expansion or reconfiguration. Generally, the aggregation network devices 112 include greater computing, memory, and storage resources than the edge network devices 110. Furthermore, the aggregation network devices 112 provide faster switching speeds (e.g., greater bandwidth) than the edge network devices 110. Furthermore, the aggregation network devices 112 include fewer interfaces than the edge network devices 110.

[0017] In the illustrated example of Fig. 1, the devices offered in the core layer 106 facilitate the transfer of data between devices in the aggregation layer 104. Therefore, the devices offered in the core layer 106 perform a large part of the data transfers and routing in the multi-layer network 100. In the example of Fig. 1, the core layer 106 includes exemplary core network devices 114. The core layer 106 includes, for example, a first core network device 1141 (denoted by C1) in communication with the first aggregation network device 1121 and the second aggregation network device 112 N .

[0018] In the illustrated example of Fig. 1, one or more of the core network devices 114 may be implemented by a data center of a core network. In some examples, one or more of the core network devices 114 may include or be implemented by an FPGA with a network interface controller, an XPU, a programmable network switch, a mid-box, etc. In some examples, one or more of the core network devices 114 include an accelerator circuit.

[0019] In the illustrated example of Fig. 1, the core network devices 114 generally include greater computing, memory, and storage resources than the aggregation network devices 112. In addition, the core network devices 114 provide faster switching speeds (e.g., greater bandwidth) than the aggregation network devices 112. Furthermore, the core network devices 114 include fewer interfaces than the aggregation network devices 112. In the example of Fig. 1, network communication within the multi-layer network 100 and between the various layers may occur over any number of wired and / or wireless media, including connectivity architectures and technologies not shown.

[0020] In the illustrated example of Fig. 1, one or more of the edge computing devices 108 may operate as an end host device (EH device) or an in-network device (IN device). Additionally, one or more of the edge network devices 110, one or more of the aggregation network devices 112, and / or one or more of the core network devices 114 may operate as an IN device or an EH device. As described above, an EH device refers to a device from which (e.g., a source computer) a data packet flow in a network originates or in which (e.g., a destination computer) a data packet flow in a network terminates. Furthermore, an in-network device refers to a device that may be located along a network path of an end-to-end (E2E) data packet flow from a source device to the destination device.In the context of an in-network device, the in-network device refers to a device type or classification based on the characteristics and / or capabilities of the device, but does not refer to the operational state of that device (e.g., does not require the device to be included in a network and / or inserted into a network or to be operating in a network to be considered an in-network device). Examples of in-network devices include, but are not limited to, switches (e.g., programmable switches), integrated circuit chips, switch SoCs, routers, router chips, router SoCs, IPUs, DPUs, EPUs, programmable accelerator circuitry (e.g., FPGAs, ASICs, GPUs, XPUs, etc.). A device can function as an IN device if the device is an intermediate device (e.g.,on a communication path between a source device and a destination device) that is responsible for routing, switching, and / or relaying incoming data packets.

[0021] In some examples, the IN devices provide computing capabilities for processing incoming data packets (e.g., data associated with a workload with which the data packets are associated). For example, a computing circuit of an IN device may range from a circuit with a low computing capability (e.g., an Atom® processor offered by Intel Corporation) to a circuit with a full computing capability (e.g., a Xeon® processor offered by Intel Corporation). If two IN devices are on the same network path (e.g., sharing a source device and a destination device), the computing capabilities of a first IN device may be less than the computing capabilities of a second IN device that is one or more hops closer to a source device and / or a destination device on the network path. In the example of the Fig. 1, the EH devices generally provide greater computing capacity than the IN devices. However, in some examples, an IN device may provide greater computing capacity than an EH device.

[0022] In the illustrated example of Fig. 1, network congestion may occur in the EH devices and / or in the IN devices. Advantageously, the examples described herein implement congestion control (CC) and flow control (FC). The disclosed methods, apparatus, and articles of manufacture implement advanced CC and FC techniques to provide INC-aware network functionality. The INC-aware network functionality may be implemented in both EH devices and IN devices (e.g., programmable IPUs, programmable DPUs, programmable EPUs, an FPGA-based circuit of intermediary accelerators, etc.).

[0023] The exemplary EH devices and / or IN devices disclosed herein use the transport layer of the OSI model to communicate information between devices in a communication path. For example, the disclosed EH devices and / or IN devices attach contextual information to data packets transmitted in the transport layer to insert and / or retain various information in data packets communicated in a network. Furthermore, the disclosed IN devices determine the contextual information to determine how received data should be interpreted when processed by an IN device. Contextual information includes, for example, information for identifying queue utilization, a bit rate used to transmit a data packet flow, a traffic class, etc.

[0024] Examples disclosed herein include adapting and correlating a data packet flow granularity level (e.g., a data packet level, a message level, and a byte stream level) with the INC-aware application context to partition the execution of transport functionalities in both the EH devices and the IN devices. As such, the methods, apparatus, and articles of manufacture disclosed herein enhance network reliability, traffic flow, and congestion control features to improve the performance of INC-aware applications. For example, the disclosed methods, apparatus, and articles of manufacture jointly consider a network structure (e.g., an IN device) and congestion control at EH devices with INC awareness to allow a network to manage large incoming congestion at IN devices and / or EH devices.

[0025] Therefore, the disclosed methods, devices, and articles of manufacture provide reliable and rapid congestion detection, signaling, and rate adaptation, and allow heterogeneous traffic classes to coexist in a network with requirements for varying data flow volumes. Furthermore, the examples disclosed herein reduce tail latency and improve fairness for coexisting data packet flows because a network (e.g., an edge network, a cloud network, etc.) can execute multi-tenant workloads concurrently (e.g., simultaneously, concurrently, etc.). Although in the example of the Fig. 1 illustrates a particular network topology, the disclosed methods, apparatus, and articles of manufacture are applicable to additional or alternative network topologies.

[0026] Fig. 2 is a block diagram of an exemplary implementation of an exemplary layered application device 200 of the multi-layer network 100 of the Fig. 1. In some examples, the layered application device 200 implements an EH device in the multi-layer network 100 of the Fig. 1. In some examples, the layered application device 200 implements an IN device in the multi-layer network 100 of the Fig. 1. In the example of Fig. 2, the layered application device 200 is operated in an exemplary application layer 202, an exemplary middle layer 204, and an exemplary transport layer 206. In the example of Fig. 2, the application layer 202 corresponds to the application layer of the OSI model. In addition, the middle layer 204 and the transport layer 206 correspond to the Fig. 2 of the middle layer or transport layer of the OSI model.

[0027] In the illustrated example of Fig. 2, the layered application device 200 includes an example application execution circuit 208, example command and completion queues 210, an example operation scheduling circuit 212, an example operation-to-dataflow mapping circuit 214, an example dataflow-to-operation mapping circuit 216, an example congestion control circuit 218, and an example network interface controller (NIC) 220. In the example of Fig. 2, the congestion control circuit 218 includes an exemplary delay determination circuit 222, an exemplary queue analysis circuit 224, an exemplary rate calculation circuit 226, and an exemplary rate adaptation circuit 228. In the example of Fig. 2, the NIC 220 includes exemplary NIC queues 230 and exemplary transmitter and / or receiver circuitry 232.

[0028] In the illustrated example of Fig. 2, the layered application device 200 can be instantiated by programmable circuits, such as a CPU (e.g., by creating an instance for insertion for an arbitrary period of time, materialization, implementation, etc.), with the CPU (Central Processor Unit) executing initial instructions. Additionally or alternatively, the layered application device 200 of the Fig. 2 may be instantiated (e.g., by creating an instance for insertion for any period of time, materialization, implementation, etc.) by (i) an application-specific integrated circuit (ASIC) and / or (ii) a field-programmable gate array (FPGA), which are structured and / or configured in response to execution of second instructions to perform operations corresponding to the first instructions. It is understood that thus, some or all of the circuitry of Fig. 2 can be instantiated at the same or different times. Some or all of the circuits of the Fig. 2 may, for example, be instantiated in one or more threads that execute concurrently in hardware and / or sequentially in hardware. Furthermore, in some examples, some or all of the circuitry of the Fig. 2 be implemented by a microprocessor circuit that executes instructions and / or an FPGA circuit that performs operations to implement one or more virtual machines and / or containers.

[0029] In the illustrated example of Fig. 2, the layered application device 200 implements application context- and / or semantics-informed joint queue-based and delay-based congestion detection and control. For example, the application execution circuitry 208 adds an application context and / or semantics to one or more application-layer data packets generated based on a workload performed by the application execution circuitry 208. To add an application context and / or semantics to one or more application-layer data packets, the application execution circuitry 208 may set a flag in the metadata of a header of the one or more application-layer data packets.

[0030] In the illustrated example of Fig. 2, the application execution circuit 208 is coupled to the instruction and completion queues 210. In some examples, the application execution circuit 208 is instantiated by a programmable circuit that executes the application instructions and / or is configured to perform operations, such as those illustrated by the flowchart of the Fig. 8. In an exemplary operation of the layered application device 200, the application execution circuit 208 executes a workload associated with an application. The application executed by the application execution circuit 208 initiates and / or triggers, for example, a bit stream (e.g., a byte stream), and the application execution circuit 208 begins filling a sending queue (SQ) of the command and completion queues 210 with the bit stream. To perform an operation in a network (e.g., the multi-layer network 100 of the Fig. 1) the application executed by the application layer 208 writes, in the example of Fig. 2 a command (in the form of one or more data packets of an application layer) into the SQ of the command and completion queues 210. In the example of Fig. 2, the command and completion queues 210 contain the SQ and a receiving queue (RQ).

[0031] In some examples, application execution circuitry 208 assigns a class identifier (ID), a workload ID, and / or an application ID to one or more application layer data packets based on the type and / or class of the application and / or workload. Therefore, when the one or more application layer data packets (sometimes referred to as application data packets) are accessed via operation scheduling circuitry 212, the ID (e.g., the class ID, the workload ID, and / or the application ID) is maintained for access by circuitry operating in middle layer 204 and may be propagated to lower layers of the OSI model (e.g., transport layer 206) and / or communicated between devices in multilayer network 100.By adding a class ID, a workload ID, and / or an application ID to the one or more application layer data packets, the application execution circuitry 208 may request that an IN device process data associated with the workload and / or application corresponding to the ID (e.g., a request to process data associated with the workload corresponding to the workload type).

[0032] In this way, assigning a class ID, a workload ID, and / or an application ID to the one or more application-layer data packets serves as setting a flag in the metadata of a data packet header, as described above. By including (e.g., inserting) an ID into a data packet, the application execution circuitry 208 facilitates a decision for an IN device (e.g., an intra-network switch) receiving the data packet as to whether the data packet needs to be processed locally. For example, the IN device may perform processing of data packets of a data packet flow in the application layer if the IN device has processing capacity to perform such processing. For example, an IN device first processes the payload data included in a data packet to generate a second payload data.In at least some such examples, the IN device updates the payload of the data packet based on the second payload to generate an updated payload. For example, the IN device may replace the first payload of the data packet with the second payload. In some examples, the IN device adds the second payload to the first payload. In some examples, the IN device adjusts the first payload based on the second payload.

[0033] In the illustrated example of Fig. 2, application execution circuitry 208 generates an ID (e.g., the class ID, the workload ID, and / or the application ID) based on a type and / or class of an application and / or workload that induces a data flow of application layer data packets. For example, application execution circuitry 208 generates an ID to uniquely associate one or more application layer data packets with the type and / or class of the application and / or workload. As such, an ID may be maintained with the one or more application layer data packets as the one or more application layer data packets are processed by a network. In this manner, the exemplary IDs disclosed herein maintain application context and / or semantic awareness throughout all lower layers of the OSI model (e.g., middle layer 204, transport layer 206, etc.).) and / or between devices in the multi-layer network 100.

[0034] In some examples, the layered application device 200 includes elements for generating one or more application data packets. The elements for generating one or more application data packets may be implemented, for example, by the application execution circuit 208. In some examples, the application execution circuit 208 may be instantiated by a programmable circuit, such as the example programmable circuit 1412 of Fig. 14. The application execution circuit 208 may be implemented, for example, by the exemplary microprocessor 1500 of the Fig. 15 which executes machine-executable instructions, such as those executed by at least blocks 802, 804 and 832 in Fig. 8. In some examples, the application execution circuit 208 may be instantiated by a hardware logic circuit implemented by an ASIC, an XPU, or the FPGA circuit 1600 in Fig. 16, which is configured and / or structured to perform operations corresponding to the machine-readable instructions. Additionally or alternatively, the application execution circuit 208 may be instantiated by any other combination of hardware, software, and / or firmware. The application execution circuit 208 may, for example, be implemented by at least one or more hardware circuits (e.g., a processor circuit, a discrete and / or integrated analog and / or digital circuit, an FPGA, an ASIC, an XPU, a comparator, an operational amplifier (op-amp), a logic circuit, etc.) configured and / or structured to execute some or all of the machine-readable instructions and / or to perform some or all of the operations corresponding to the machine-readable instructions without executing any software or firmware, but other structures are equally suitable.

[0035] In the illustrated example of Fig. 2, the command and completion queues 210 are coupled to the application execution circuit 208 and the operation scheduling circuit 212. In the example of Fig. 2, the layered application device 200 includes command and completion queues 210 for recording data (e.g., one or more application layer data packets, etc.) that is transmitted over the multi-layer network 100 and / or processed by the application execution circuitry 208 before being transmitted over the multi-layer network 100. The command and completion queues 210 may be implemented by volatile memory (e.g., synchronous dynamic random access memory (SDRAM), dynamic random access memory (DRAM), RAMBUS dynamic random access memory (RDRAM), etc.) and / or non-volatile memory (e.g., flash memory).The command and completion queues 210 may additionally or alternatively be implemented by one or more double data rate memories (DDR memories), such as DDR, DDR2, DDR3, DDR4, DDR5, mobile DDR (mDDR), DDR-SDRAM, etc. The command and completion queues 210 may additionally or alternatively be implemented by one or more mass storage devices such as hard disk drives, compact disk drives (CD drives), digital versatile disk drives (DVD drives), solid-state drives, secure digital cards (SD cards), compact flash cards (CF cards), etc. Although the illustrated example illustrates the command and completion queues 210 as two queues, the command and completion queues 210 may be implemented by any number and / or types of queues.In addition, the data stored in the command and completion queues 210 may be in any data format, such as binary data, comma-separated data, tab-separated data, structured SQL (Query Language) structures, etc.

[0036] In the illustrated example of Fig. 2, the operation scheduling circuit 212 is coupled to the instruction and completion queues 210, the operation-to-dataflow mapping circuit 214, and the dataflow-to-operation mapping circuit 216. In some examples, the operation scheduling circuit 212 is instantiated by a programmable circuit that executes operation scheduling instructions and / or is configured to perform operations such as those represented by the flowchart of the Fig. 8. In an exemplary operation of the layered application device 200, the operation scheduling circuit 212 accesses one or more application layer data packets from the command and completion queues 210 and schedules the one or more application layer data packets. For example, the operation scheduling circuit 212 determines one or more operations according to which the application layer data packets are processed.

[0037] In the illustrated example of Fig. 2, example operations include reordering the data packets, reassembling the data packets, and duplicating the data packets. In the example of Fig. 2, one or more operations scheduled by operation scheduling circuitry 212 are scheduled to improve the resilience and / or reliability of one or more data packets forwarded to transport layer 206. As described above, application context and / or semantic awareness is maintained in the one or more application layer data packets via an ID identified in the header metadata of the one or more application layer data packets.

[0038] In some examples, the layered application device 200 includes elements for determining one or more operations. The elements for determining one or more operations may be implemented, for example, by the operation scheduling circuit 212. In some examples, the operation scheduling circuit 212 may be implemented by a programmable circuit, such as the exemplary programmable circuit 1412 of Fig. 14. For example, the operation planning circuit 212 may be instantiated by the exemplary microprocessor 1500 of Fig. 15 which executes machine-executable instructions such as those specified by at least block 806 of the Fig. 8. In some examples, the operation scheduling circuit 212 may be instantiated by a hardware logic circuit implemented by an ASIC, an XPU, or the FPGA circuit 1600 of the Fig. 16 that are configured and / or structured to perform operations corresponding to the machine-readable instructions. Additionally or alternatively, the operation scheduling circuit 212 may be instantiated by any other combination of hardware, software, and / or firmware. The operation scheduling circuit 212 may, for example, be implemented by at least one or more hardware circuits (e.g., a processor circuit, a discrete and / or integrated analog and / or digital circuit, an FPGA, an ASIC, an XPU, a comparator, an operational amplifier (op-amp), a logic circuit, etc.) that are configured and / or structured to execute some or all of the machine-readable instructions and / or to perform some or all of the operations corresponding to the machine-readable instructions without executing any software or firmware, but other structures are equally suitable.

[0039] In the illustrated example of Fig. 2, the operation-to-dataflow mapping circuit 214 is coupled to the operation scheduling circuit 212, the dataflow-to-operation mapping circuit 216, the delay determination circuit 222, and the queue analysis circuit 224. In some examples, the operation-to-dataflow mapping circuit 214 is instantiated by a programmable circuit that executes operation mapping instructions and / or is configured to perform operations such as those shown in the flowchart of Fig. 8. In an exemplary operation of the layered application device 200, the operation-to-dataflow mapping circuit 214 converts the one or more application layer data packets into a data packet flow based on the one or more operations.

[0040] In the illustrated example of Fig. 2, to convert the one or more application-layer data packets into a data packet flow, the operation-to-dataflow mapping circuit 214 maps the one or more operations to a transport-layer data packet flow (e.g., the data packet flow). In some examples, the operation-to-dataflow mapping circuit 214 also converts the one or more application-layer data packets into a data packet flow based on the priority of a class of workload with which the one or more application data packets are associated. The operation-to-dataflow mapping circuit 214, for example, analyzes and / or decodes the control plane of application-layer data packets to release the contents of the application-layer data packets and identify a priority of associated commands.As such, the operation-to-flow mapping circuit 214 may classify and prioritize data packet flows based on context and / or urgency to better manage network congestion by inserting application context and / or semantic insight IDs into the application layer data packets. In the example of FIG. Fig. 2, the layered application device 200 also includes the data flow to operation mapping circuit 216 configured to convert transport layer data packet flows into one or more application layer data packets.

[0041] In some examples, the layered application device 200 includes elements for converting one or more application data packets. The elements for converting one or more application data packets may be implemented, for example, by the operation-to-dataflow mapping circuit 214. In some examples, the operation-to-dataflow mapping circuit 214 may be implemented by a programmable circuit, such as the example programmable circuit 1412 of Fig. 14. The operation-to-dataflow mapping circuit 214 may be implemented, for example, by the exemplary microprocessor 1500 of the Fig. 15 which executes machine-executable instructions such as those specified by at least block 808 of the Fig. 8. In some examples, the operation-to-dataflow mapping circuit 214 may be instantiated by a hardware logic circuit implemented by an ASIC, an XPU, or the FPGA circuit 1600 of the Fig. 16 that are configured and / or structured to perform operations corresponding to the machine-readable instructions. Additionally or alternatively, the operation-to-dataflow mapping circuit 214 may be instantiated by any other combination of hardware, software, and / or firmware. The operation-to-dataflow mapping circuit 214 may, for example, be implemented by at least one or more hardware circuits (e.g., a processor circuit, a discrete and / or integrated analog and / or digital circuit, an FPGA, an ASIC, an XPU, a comparator, an operational amplifier (op-amp), a logic circuit, etc.) that are configured and / or structured to execute some or all machine-readable instructions and / or to perform some or all operations corresponding to the machine-readable instructions without executing any software or firmware, but other structures are equally suitable.

[0042] In the illustrated example of Fig. 2, the congestion control circuit 218 is coupled to the operation-to-dataflow mapping circuit 214 and the NIC queues 230. In some examples, the congestion control circuit 218 is instantiated by a programmable circuit that executes congestion control instructions and / or is configured to perform operations such as those illustrated by the flowcharts of the Fig. 8, Fig. 12 and Fig. 13. In an exemplary operation of the layered application device 200, the congestion control circuit 218 implements joint queue-based and delay-based congestion detection and control. As described above, the layered application device 200 may implement an EH device and / or an IN device. As such, the layered application device 200 facilitates joint queue-based and delay-based congestion detection and control in both the EH devices and the IN devices and decouples a network structure from EH congestion.

[0043] In some examples, the layered application device 200 includes elements for controlling congestion. The elements for controlling congestion may be implemented by the congestion control circuit 218. In some examples, the congestion control circuit 218 may be implemented by a programmable circuit, such as the exemplary programmable circuit 1412 of Fig. 14. For example, the overload control circuit 218 may be instantiated by the exemplary microprocessor 1500 of Fig. 15 that executes machine-executable instructions, such as those executed by at least blocks 810, 812, 814, 816, 818, 820, 822, 824, 826, 828, and 830 of the Fig. 8, at least blocks 1202, 1204, 1206, 1208, 1210, 1212 and 1214 of the Fig. 12 and / or at least blocks 1302, 1304, 1306, 1308, 1310, 1312, 1314 and 1316 of the Fig. 13. In some examples, the congestion control circuit 218 may be instantiated by a hardware logic circuit implemented by an ASIC, an XPU, or the FPGA circuit 1600 of the Fig. 16 that are configured and / or structured to perform operations corresponding to the machine-readable instructions. Additionally or alternatively, the overload control circuit 218 may be instantiated by any other combination of hardware, software, and / or firmware. The overload control circuit 218 may, for example, be implemented by at least one or more hardware circuits (e.g., a processor circuit, a discrete and / or integrated analog and / or digital circuit, an FPGA, an ASIC, an XPU, a comparator, an operational amplifier (op-amp), a logic circuit, etc.) that are configured and / or structured to execute some or all machine-readable instructions and / or to perform some or all operations corresponding to the machine-readable instructions without executing any software or firmware, but other structures are equally suitable.

[0044] As described above, the congestion control circuit 218 includes the delay detection circuit 222, the queue analysis circuit 224, the frequency calculation circuit 226, and the rate adaptation circuit 228. In the example of Fig. 2, the delay determination circuit 222 is coupled to the dataflow-to-operation mapping circuit 214, the rate calculation circuit 226, and the NIC queues 230. In some examples, the delay determination circuit 222 is instantiated by a programmable circuit that executes delay determination instructions and / or is configured to perform operations such as those illustrated by the flowchart of the Fig. 8 are represented.

[0045] In an exemplary operation of the layered application device 200, the delay determination circuit 222 identifies one or more historical data packet flows having second IDs that match a first ID of a data packet flow being processed by the delay determination circuit 222. Based on identifying a historical data packet flow having an ID matching a current data packet flow, the delay determination circuit 222 determines a delay measurement of the second data packet flow. For example, the delay determination circuit 222 determines a round-trip time (RTT) of the second data packet flow.

[0046] In the illustrated example of Fig. 2, the RTT of a historical data packet flow can be captured using techniques such as In-Network Telemetry (INT). In the example of Fig. 2, the INT allows the congestion control circuit 218 to infer network states and / or collect metadata from the network across single and / or multiple hops. For example, timestamps (TSS) of data packets during dequeueing, queue length measurements (QLen measurements) at the time a data packet is dequeued, transmission size measurements (TxBytes measurements) (in bytes), and a speed class of output ports in an IN device of the congestion control circuit 218 can assist in dynamically scaling a target delay based on network utilization and / or network topology.Furthermore, telemetry data collected in an EH device can be fed back to a packet flow generator to signal that the EH packet processing stack has limited resources (which adds latency to the packet flow end time). Therefore, the packet flow generator can adjust the packet flow (e.g., by temporarily stopping the packet flow, reducing the packet flow bit rate, etc.).

[0047] In the illustrated example of Fig. 2, the delay measurement may be determined as an E2E measurement (e.g., the time it takes for a data packet (or data packet flow) to travel from a source device to a destination device and until the source device receives an acknowledgment from the destination device). Additionally or alternatively, the delay measurement may be determined as a single-hop measurement (e.g., the time it takes for a data packet (or data packet flow) to travel from a first device to a next device and until the first device receives an acknowledgment from the next device). In some examples, the delay measurement may be determined as a hop-by-hop measurement (e.g., the sum of the single-hop delay measurements measured at each IN device along an E2E path).

[0048] In exemplary operation of the layered application device 200, the delay determination circuit 222 predicts a delay associated with the transmission of the current data packet flow through the network based on the delay measurement of a historical data packet flow. Furthermore, the delay determination circuit 222 determines whether the predicted delay meets (e.g., is greater than) a delay threshold for the layered application device 200. If the layered application device 200 implements an EH device, in some examples, the delay threshold is determined based on a maximum allowable E2E delay such that the delay of the EH device does not cause the E2E delay to exceed a maximum allowable E2E delay for a communication path including the EH device.If the layered application device 200 implements an IN device, the delay threshold is additionally or alternatively determined based on a maximum allowable delay in the IN device such that the IN device delay does not cause the E2E delay to exceed a maximum allowable E2E delay for a communication path including the IN device. Based on the delay determination circuit 222 determining that the predicted delay satisfies the delay threshold, the delay determination circuit 222 sets a first Explicit Congestion Notification (ECN) flag for data packets of the current data packet flow. For example, the delay determination circuit 222 sets one or more bits in the headers of the data packets of the current data packet flow.

[0049] In some examples, the layered application device 200 includes elements for predicting a delay. The elements for predicting a delay may be implemented, for example, by the delay determination circuit 222. In some examples, the delay determination circuit 222 may be implemented by a programmable circuit, such as the example programmable circuit 1412 of Fig. 14. The delay detection circuit 222 may be implemented, for example, by the exemplary microprocessor 1500 of Fig. 15 which executes machine-executable instructions, such as those executed by at least blocks 810, 812, 814 and 816 in Fig. 8. In some examples, the delay determination circuit 222 may be instantiated by a hardware logic circuit implemented by an ASIC, an XPU, or the FPGA circuit 1600 of the Fig. 16 that are configured and / or structured to perform operations corresponding to the machine-readable instructions. Additionally or alternatively, the delay determination circuit 222 may be instantiated by any other combination of hardware, software, and / or firmware. The delay determination circuit 222 may, for example, be implemented by at least one or more hardware circuits (e.g., a processor circuit, a discrete and / or integrated analog and / or digital circuit, an FPGA, an ASIC, an XPU, a comparator, an operational amplifier (op-amp), a logic circuit, etc.) that are configured and / or structured to execute some or all of the machine-readable instructions and / or to perform some or all of the operations corresponding to the machine-readable instructions without executing any software or firmware, but other structures are equally suitable.

[0050] In the illustrated example of Fig. 2, the queue analysis circuit 224 is coupled to the operation-to-dataflow mapping circuit 214, the rate calculation circuit 226, and the NIC queues 230. In some examples, the queue analysis circuit 224 is instantiated by a programmable circuit that executes queue analysis instructions and / or is configured to perform operations such as those illustrated by the flowchart of the Fig. 8. In an exemplary operation of the layered application device 200, the queue analysis circuit 224 determines the state information of a queue (e.g., an SQ) of the NIC queues 230.

[0051] In the illustrated example of Fig. 2, the queue state information may be the queue state information for the current data packet flow. Additionally or alternatively, the queue state information may be the queue state information for a historical data packet flow. For example, the queue analysis circuit 224 receives an identification of the one or more historical data packet flows having matching IDs from the delay determination circuit 222. In some examples, the queue analysis circuit 224 identifies one or more historical data packet flows having second IDs that match a first ID of a data packet flow being processed by the queue analysis circuit 224 and forwards an identification of the one or more historical data packet flows to the delay determination circuit 222.

[0052] In the illustrated example of Fig. 2, the queue state information corresponds to a queue utilization. An example queue utilization may be determined as an E2E measurement (e.g., the queue utilization by a data packet (or data packet flow) transmitted from a source device to a destination device). Additionally or alternatively, the queue utilization may be determined as a single-hop measurement (e.g., the queue utilization by a data packet (or data packet flow) transmitted from a first device in a communication path to a next device in the communication path). In some examples, the queue utilization may be determined as a hop-by-hop measurement (e.g., the sum of the queue utilization measurements measured at each IN device along an E2E path).

[0053] In an exemplary operation of the layered application device 200, the queue analysis circuit 224 predicts a second queue utilization by the current data packet flow based on the queue utilization by a historical data packet flow. Furthermore, the queue analysis circuit 224 determines whether the predicted utilization meets (e.g., is greater than) a utilization threshold for the layered application device 200. In the example of Fig. 2, the utilization threshold is configurable and depends on the memory and processing latency parameters of an EH device (or an IN device if the layered application device 200 implements the IN device). Based on the queue analysis circuit 224 determining that the predicted utilization meets the utilization threshold, the queue analysis circuit 224 sets a second ECN flag for data packets of the current data packet flow. For example, the queue analysis circuit 224 sets one or more bits in the headers of data packets of the current data packet flow.

[0054] In some examples, the layered application device 200 includes elements for predicting the utilization of a queue. The elements for predicting the utilization of a queue may be implemented, for example, by the queue analysis circuit 224. In some examples, the queue analysis circuit 224 may be implemented by a programmable circuit, such as the exemplary programmable circuit 1412 of Fig. 14. The queue analysis circuit 224 may be implemented, for example, by the exemplary microprocessor 1500 of the Fig. 15 which executes machine-executable instructions, such as those executed by at least blocks 818, 820, 822 and 824 in Fig. 8. In some examples, the queue analysis circuit 224 may be instantiated by a hardware logic circuit implemented by an ASIC, an XPU, or the FPGA circuit 1600 of the Fig. 16 that are configured and / or structured to perform operations corresponding to the machine-readable instructions. Additionally or alternatively, the queue analysis circuit 224 may be instantiated by any other combination of hardware, software, and / or firmware. For example, the queue analysis circuit 224 may be implemented by at least one or more hardware circuits (e.g., a processor circuit, a discrete and / or integrated analog and / or digital circuit, an FPGA, an ASIC, an XPU, a comparator, an operational amplifier (op-amp), a logic circuit, etc.) that are configured and / or structured to execute some or all of the machine-readable instructions and / or to perform some or all of the operations corresponding to the machine-readable instructions without executing any software or firmware, but other structures are equally suitable.

[0055] In the illustrated example of Fig. 2, the rate calculation circuit 226 is coupled to the delay determination circuit 222, the queue analysis circuit 224, and the NIC queues 228. In some examples, the rate calculation circuit 226 is instantiated by a programmable circuit that executes rate calculation instructions and / or is configured to perform operations such as those illustrated by the flowcharts of the Fig. 8, Fig. 12 and Fig. 13. In an exemplary operation of the layered application device 200, the rate calculation circuit 226 determines a bit rate that must be used to transmit the current data packet flow to a device in the network. For example, the rate calculation circuit 226 determines the bit rate based on the predicted delay for the current data packet flow and the predicted congestion by the current data packet flow. In some examples, the rate calculation circuit 226 determines the bit rate based on whether the first and / or second ECN flags are set.

[0056] In the illustrated example of Fig. 2, the bit rate determined by the rate calculation circuit 226 refers to the allowable bit rate for the data packet flow based on the predicted structural congestion (e.g., the predicted queue utilization) and the EH device congestion (e.g., the predicted delay). As described above, by analyzing and / or decoding the control plane to extract the data flow ID and / or other metadata in the congestion control circuit 218, the congestion control circuit 218 can compare the current data packet flow with historical data packet flows to derive conditions (e.g., delay, queue utilization, etc.) that have affected network congestion during the historical data packet flows in order to adjust transmission parameters (e.g., the bit rate) of the current data packet flow accordingly.

[0057] In some examples, the layered application device 200 includes elements for calculating a bit rate. The elements for calculating a bit rate may be implemented, for example, by the rate calculation circuit 226. In some examples, the rate 226 may be determined by a programmable circuit, such as the exemplary programmable circuit 1412 of Fig. 14. For example, the rate calculation circuit 226 may be instantiated by the exemplary microprocessor 1500 of the Fig. 15 that executes machine-executable instructions, such as those executed by at least block 826 of the Fig. 8, at least blocks 1202, 1204, 1206, 1208, 1210, 1212 and 1214 of the Fig. 12 and / or at least blocks 1302, 1304, 1306, 1308, 1310, 1312, 1314 and 1316 of the Fig. 13. In some examples, the rate calculation circuit 226 may be instantiated by a hardware logic circuit implemented by an ASIC, an XPU, or the FPGA circuit 1600 of the Fig. 16 that are configured and / or structured to perform operations corresponding to the machine-readable instructions. Additionally or alternatively, the rate calculation circuit 226 may be instantiated by any other combination of hardware, software, and / or firmware. The rate calculation circuit 226 may, for example, be implemented by at least one or more hardware circuits (e.g., a processor circuit, a discrete and / or integrated analog and / or digital circuit, an FPGA, an ASIC, an XPU, a comparator, an operational amplifier (op-amp), a logic circuit, etc.) that are configured and / or structured to execute some or all of the machine-readable instructions and / or to perform some or all of the operations corresponding to the machine-readable instructions without executing any software or firmware, but other structures are equally suitable.

[0058] In the illustrated example of Fig. 2, the rate adjustment circuit 228 is coupled to the rate calculation circuit 226 and the NIC queues 230. In some examples, the rate adjustment circuit 228 is instantiated by a programmable circuit that executes the rate adjustment instructions and / or is configured to perform operations such as those illustrated by the flowchart of Fig. 8. In an exemplary operation of the layered application device 200, the rate adaptation circuit 228 determines whether the allowable bit rate determined for the current data packet flow exceeds a size of the data packet flow (e.g., a first data packet of the data packet flow). Based on the rate adaptation circuit 228 determining that the size of the data packet flow exceeds the allowable bit rate, the rate adaptation circuit 228 segments the data packet flow into two or more segments.

[0059] As such, rate adjustment circuitry 228 may adjust (e.g., slow down) a rate of a data packet flow proportional to the size of the data packet flow. As described above, the data packet header of the data packet flow identifies one or more ECN flags, queue state information, and / or other metadata related to the parameters of the data packet flow. Rate adjustment circuitry 228 subsequently loads the one or more segments of the data packet flow into a queue (e.g., an SQ) of the NIC queues 230.

[0060] In some examples, the layered application device 200 includes elements for splitting a data packet flow. The elements for splitting a data packet flow may be implemented, for example, by the rate adaptation circuit 228. In some examples, the rate adaptation circuit 228 may be implemented by a programmable circuit, such as the exemplary programmable circuit 1412 of Fig. 14. The rate matching circuit 228 may be implemented, for example, by the exemplary microprocessor 1500 of the Fig. 15 which executes machine-executable instructions, such as those executed by at least blocks 828 and 830 in Fig. 8. In some examples, the rate adaptation circuit 228 may be instantiated by a hardware logic circuit implemented by an ASIC, an XPU, or the FPGA circuit 1600 of the Fig. 16 that are configured and / or structured to perform operations corresponding to the machine-readable instructions. Additionally or alternatively, the rate adaptation circuit 228 may be instantiated by any other combination of hardware, software, and / or firmware. The rate adaptation circuit 228 may be implemented, for example, by at least one or more hardware circuits (e.g., a processor circuit, a discrete and / or integrated analog and / or digital circuit, an FPGA, an ASIC, an XPU, a comparator, an operational amplifier (op-amp), a logic circuit, etc.) that are configured and / or structured to execute some or all of the machine-readable instructions and / or to perform some or all of the operations corresponding to the machine-readable instructions without executing any software or firmware, but other structures are equally suitable.

[0061] In the illustrated example of Fig. 2, the NIC queues 230 are coupled to the delay detection circuit 222, the queue analysis circuit 224, and the rate adaptation circuit 228. In the example of Fig. 2, the layered application device 200 includes the NIC queues 230 for recording data (e.g., from one or more transport layer data packets, from one or more data packet flows, etc.) transmitted through the multi-layer network 100 and / or received from the multi-layer network 100. The NIC queues 230 may be implemented by a volatile memory (e.g., SDRAM, DRAM, RDRAM, etc.) and / or a non-volatile memory (e.g., flash memory). The NIC queues 230 may additionally or alternatively be implemented by one or more DDR memory, such as DDR, DDR2, DDR3, DDR4, DDR5, mDDR, etc. The NIC queues 230 may additionally or alternatively be implemented by one or more mass storage devices, such as HDDs, CD drives, DVD drives, SSD drives, etc.Although the illustrated example illustrates NIC queues 230 as two queues, NIC queues 230 may be implemented by any number and / or types of queues. Furthermore, the data stored in NIC queues 230 may be in any data format, such as binary data, comma-separated data, tab-separated data, structured SQL, etc.

[0062] In the illustrated example of Fig. 2, the transmitter and / or receiver circuitry 232 is coupled to the NIC queues 230. In exemplary operation of the layered application device 200, the transmitter and / or receiver circuitry 232 transmits a first segment (e.g., a first data packet) of the data packet flow at the bit rate. As described above, the updated headers of data packets of the data packet flow maintain application context and / or semantic awareness from the application layer 202 through the NIC 220 (operating in the data link layer of the OSI model). As such, the application context and / or semantic awareness is communicated to a destination device via the updated header of the data packets of the data packet flow. In the examples disclosed herein, a destination device may be one or more hops away from a source device.

[0063] In the illustrated example of Fig. 2, the operation of the layered application device 200 was described from the perspective of transmitting a data packet flow. In the illustrated example, the exemplary layered application device 200 functions to receive a data packet flow in the reverse direction with respect to the above description of Fig. 2. For example, the transmitter and / or receiver circuit 232 receives a data packet flow, and the rate calculation circuit 226 determines a bit rate at which the data packet flow must be transmitted by the transmitter and / or receiver circuit 232. The exemplary bit rate is based on a predicted delay for the data packet flow (and / or a delay measurement for all previous hops by the data packet flow) and on a predicted utilization of a queue of the receiving device by the data packet flow (and / or a current utilization of the queue by other data packet flows).

[0064] For example, if the device additionally includes processing capability (e.g., an integrated accelerator circuit) after determining the bit rate, the data flow-to-operation mapping circuit 216 may convert the data packet flow into one or more application-layer data packets. Furthermore, the operation scheduling circuit 212 may determine one or more operations according to which the one or more application-layer data packets are to be processed and store the one or more application-layer data packets in an RQ of the command and completion queues 210. Subsequently, the application execution circuit 208 may process the application-layer data packets and thereafter return the data packets to the NIC 220 for transmission to the next device in a communication path of the data packet flow, as described above.

[0065] In some examples, the layered application device 200 includes circuitry for discovering one or more INC capabilities of an IN device and / or for selecting a communication path for a data packet flow through a network based on the one or more INC capabilities of an IN device and / or one or more workload parameters of a data packet flow. Such circuitry operates, for example, in the middle layer 204. To discover one or more INC capabilities of an IN device, the circuitry of the layered application device 200 operating in the middle layer 204, in the example of Fig. 2, one or more messages containing contextual information of the application or a communication history with the neighboring devices in the network. Therefore, the circuitry of the layered application device 200 operating in the middle layer 204 can discover potential communication paths via neighboring devices.

[0066] Furthermore, the circuitry of the layered application device 200 operating in the middle layer 204 may implement an artificial intelligence and machine learning (AI / ML) model to predict the communication loss in advance (e.g., current or alternative) in order to select or switch to an alternative communication path earlier. If the AI / ML model predicts a loss of all and / or part of the communication that is greater than a threshold, the circuitry of the layered application device 200 operating in the middle layer 204 may buffer latency-insensitive data packets and / or data traffic during a period of connection loss. Furthermore, the circuitry of the layered application device 200 operating in the middle layer 204 mayAn original, a duplicate, or repeat data packets) can be assigned to a primary and / or an alternate communication path and metadata can be added to the data packets to enable application-informed handling of data packets at the lower layers (e.g., transport layer 206, etc.). Circuitry operating at transport layer 206 can, for example, use this information to decide on duplication / retransmission, coordinated flow control (e.g., treating a number of data packets as a group when an end-to-end service experience depends jointly on a group of data packets), self-decodable segmentation, and a congestion control strategy. Furthermore, the network layers can use this information to select a next-hop device.

[0067] Fig. 3 is a block diagram of an exemplary implementation of an exemplary feedback-capable application device 300 of the multi-layer network 100 of the Fig. 1. In some examples, the feedback-capable device 300 implements an IN device in the multi-layer network 100 of the Fig. 1. In some examples, the feedback-capable application device 300 implements an EH device in the multi-layer network 100 of the Fig. 1. The example of Fig. 3 illustrates a block diagram of the feedback-capable device 300 for performing rate calculation and feedback (e.g., immediate feedback) for single-hop, multi-hop, and / or E2E data packet flows to support congestion control in the multi-layer network 100 of the Fig. 1.

[0068] In the illustrated example of Fig. 3, the feedback-capable device 300 includes an exemplary receiver circuit 302, an exemplary queue of data packet flows 304, an exemplary application processing circuit 306, an exemplary rate calculation circuit 308, an exemplary data storage for metadata of a data packet flow 310, an exemplary circuit for generating feedback data traffic 312, and an exemplary transmitter circuit 314. In the example of Fig. 3, the feedback-capable device 300 can be instantiated by programmable circuits such as a CPU (e.g., by creating an instance for creation for an arbitrary period of time, for materialization, for implementation, etc.), with the CPU (Central Processor Unit) executing initial instructions. Additionally or alternatively, the feedback-capable device 300 can be Fig. 3 be instantiated (e.g., by creating an instance for creation for any duration of time, materializing, implementing, etc.) by (i) an application-specific integrated circuit (ASIC) and / or (ii) a field-programmable gate array (FPGA) structured and / or configured in response to execution of second instructions to perform operations corresponding to the first instructions.

[0069] It is self-evident that some or all of the circuits of the Fig. 3 can be instantiated at the same or different times. Some or all of the circuits of the Fig. 3 may, for example, be instantiated in one or more threads that execute concurrently in hardware and / or sequentially in hardware. Furthermore, in some examples, some or all of the circuitry of the Fig. 3 be implemented by a microprocessor circuit that executes instructions and / or an FPGA circuit that performs operations to implement one or more virtual machines and / or containers.

[0070] In the illustrated example of Fig. 3, the receiver circuit 302 is coupled to the queues of data packet flows 304 and the data storage for metadata of a data packet flow 310. In some examples, the receiver circuit 302 is instantiated by a programmable circuit that executes the rate calculation instructions and / or is configured to perform operations such as those illustrated by the flowchart of Fig. 9. In an exemplary operation of the feedback-capable device 300, the receiver circuit 302 receives one or more data packets and loads the one or more data packets into the queue of data packet flows 304. In the example of Fig. 3, the receiver circuit 302 adds a first timestamp to an initial data packet of a data packet flow when the initial data packet is received by the receiver circuit 302. Furthermore, the receiver circuit 302 determines metadata included in one or more data packets and stores the metadata in the data packet flow metadata store 310 with an associated data packet flow ID to identify an associated data packet flow with which the metadata is associated.

[0071] In some examples, the feedback-capable device 300 includes elements for receiving a data packet. The elements for receiving a data packet may be implemented, for example, by the receiver circuit 302. In some examples, the receiver circuit 302 may be implemented by a programmable circuit, such as the exemplary programmable circuit 1412 of Fig. 14. The receiver circuit 302 can be implemented, for example, by the exemplary microprocessor 1500 of Fig. 15 which executes machine-executable instructions, such as those described at least by blocks 902 and 910 in Fig. 9. In some examples, the receiver circuit 302 may be instantiated by a hardware logic circuit implemented by an ASIC, an XPU, or the FPGA circuit 1600 of the Fig. 16 that are configured and / or structured to perform operations corresponding to the machine-readable instructions. Additionally or alternatively, the receiver circuit 302 may be instantiated by any other combination of hardware, software, and / or firmware. The receiver circuit 302 may, for example, be implemented by at least one or more hardware circuits (e.g., a processor circuit, a discrete and / or integrated analog and / or digital circuit, an FPGA, an ASIC, an XPU, a comparator, an operational amplifier (op-amp), a logic circuit, etc.) that are configured and / or structured to execute some or all of the machine-readable instructions and / or to perform some or all of the operations corresponding to the machine-readable instructions without executing any software or firmware, but other structures are equally suitable.

[0072] In the illustrated example of Fig. 3, the queue of data packet flows 304 is coupled to the receiver circuit 302, the application processing circuit 306, the rate calculation circuit 308, and the transmitter circuit 314. In the example of Fig. 3, the feedback-capable device 300 includes the data packet flow queue 304 to record data (e.g., one or more transport layer data packets, one or more data packet flows, etc.) that is transmitted over the multi-layer network 100 and / or processed by a computer circuit (not shown) of the feedback-capable device 300 before being transmitted over the multi-layer network 100. In the example of Fig. 3, the data packet flow queue 304 adds a second timestamp to an initial data packet of a data packet flow when the initial data packet is enqueued in the data packet flow queue 304. Furthermore, the data packet flow queue 304 adds a third timestamp to a final data packet of a data packet flow when the final data packet is dequeued from the data packet flow queue 304.

[0073] In the illustrated example of Fig. 3, the queue of data packet flows 304 may be implemented by a volatile memory (e.g., SDRAM, DRAM, RDRAM, etc.) and / or a non-volatile memory (e.g., flash memory). The queue of data packet flows 304 may additionally or alternatively be implemented by one or more DDR memory types, such as DDR, DDR2, DDR3, DDR4, DDR5, mDDR, etc. The queue of data packet flows 304 may additionally or alternatively be implemented by one or more mass storage devices, such as HDDs, CD drives, DVD drives, SSD drives, etc. Although the illustrated example illustrates the queue of data packet flows 304 as one queue, the queue of data packet flows 304 may be implemented by any number and / or types of queues.In addition, the data stored in the queue of data packet flows 304 may be in any data format, such as binary data, comma-separated data, tab-separated data, structured SQL structures, etc.

[0074] In the illustrated example of Fig. 3, the application processing circuit 306 is coupled to the queue of data packet flows 304. In some examples, the application processing circuit 306 is instantiated by a programmable circuit that executes the application processing instructions and / or is configured to perform operations such as those illustrated by the flowchart of the Fig. 9. In an exemplary operation of the feedback-capable device 300, the application processing circuitry 306 analyzes one or more data packets in the queue of data packet flows 304 to determine the corresponding IDs (e.g., class IDs, workload IDs, application IDs, etc.) contained in the one or more data packets. By analyzing a data packet for a workload ID, the application processing circuitry 306 determines, for example, a workload type of a workload associated with the data packet. Examples of workload types include workloads related to low-latency applications (e.g., online meetings, online gaming, high-frequency trading, etc.), a workload related to long-tail applications (e.g., applications with a long-tail latency, e.g.,Applications that introduce latency in the 98th percentile compared to average latency) and workloads associated with and / or originating from an application implemented in a terminal device and an EH device and / or an IN device (e.g., location service applications, social networking applications, etc.). An example workload type additionally or alternatively includes, but is not limited to, a compression and / or decompression workload, an encryption workload, a distribution / gathering workload, a load balancing workload, a network address translation workload, a domain name system (DNS) server workload, a caching workload, a data reduction workload (e.g.,a workload that relates to reducing the amount of data transferred from one in-network device to another device along a communication path) and a data aggregation workload (e.g., a workload that relates to aggregating a batch of data in an in-network device).

[0075] In the illustrated example of Fig. 3, by determining the workload type of a workload associated with a data packet or a data packet flow, the application processing circuit 306 determines how one or more data packets of the data packet flow are to be interpreted when processed by the application processing circuit 306. Accordingly, the application processing circuit 306 processes data contained in a received data packet based on the workload type of a workload associated with the data packet. For example, the application processing circuit 306 performs a portion of the workload associated with the data packet flow, performs the entirety of the workload associated with the data packet flow, and / or performs some additional or alternative operations (e.g.,a compression and / or decompression operation, encryption operation, distribution / gathering operation, load balancing operation, network address translation operation, DNS server operation, caching operation, data reduction operation, data aggregation operation, etc.) on the data packet flow.

[0076] After processing data contained in a data packet of a data packet flow, application processing circuitry 306 loads the result into the data packet flow queue 304 for forwarding to another device in a communication path of the data packet flow or for sending back to an EH device that has requested IN processing. In this way, application processing circuitry 306 causes transmitter circuitry 314 to transmit the result to another device in multilayer network 100. Furthermore, after processing the data contained in a data packet of a data packet flow, application circuitry 306 modifies the data packet to indicate that an action has been performed on the data packet.For example, there may only be a preset list of operations, functions, primitives, and / or processing that the IN devices can perform on a data packet. By modifying a display field of the data packet based on the operation performed on the data packet, application processing circuitry 306 can identify the operation to be performed on the data packet for the next device in the data packet's communication path.

[0077] In some examples, the indicator field may be implemented by adapting the field to include a class ID, a workload ID, and / or an application ID of a data packet. For example, application processing circuitry 306 may modify a data packet after performing an action on the data packet by removing the information in the field containing a class ID, a workload ID, and / or an application ID of the data packet. In this way, a subsequent device in the communication path of the data packet will not repeat the action performed on the data packet by application processing circuitry 306.In additional or alternative examples, the indicator field may be implemented as an explicit field containing information mapping to the preset list of operations, functions, primitives, and / or processing that can be performed by IN devices on a data packet. For example, if the preset list of operations, functions, primitives, and / or processing that can be performed by IN devices on a data packet is mapped to a lookup table (LUT), application processing circuitry 306 sets a value in the indicator field that maps to the corresponding action in the LUT performed on the data packet.Additionally or alternatively, the feedback-capable device 300 may transmit an additional data packet in a communication path of a data packet to the next device to identify the action performed by the feedback-capable device 300 on the data packet.

[0078] In such examples, a subsequent device in a communication path of a data packet may also implement the LUT and / or signaling to identify an action performed on a data packet, such as a hop count or a forwarding device ID. After a device (e.g., an IN device and / or an EH device) receives a data packet that was processed by a previous device in a communication path of the data packet, the device takes over a workload from where the previous device left off. In addition to the workload-related operations, the device performs additional operations on a per-workload basis. These operations include latency-related operations, queuing-related and / or update-related operations, and / or scaling-related operations to meet the workload requirements (e.g.,to meet QoS requirements specified by an SLA and / or a Service Level Objective (SLO) associated with the workload).

[0079] In some examples, the feedback-capable device 300 includes elements for processing a data packet. The elements for processing a data packet may be implemented, for example, by the application processing circuit 306. In some examples, the application processing circuit 306 may be instantiated by a programmable circuit, such as the exemplary programmable circuit 1412 of Fig. 14. The application processing circuit 306 may be implemented, for example, by the exemplary microprocessor 1500 of the Fig. 15, which executes machine-executable instructions, such as those executed by at least blocks 904, 906 and 908 of the Fig. 9. In some examples, the application processing circuit 306 may be instantiated by a hardware logic circuit implemented by an ASIC, an XPU, or the FPGA circuit 1600 of the Fig. 16 that is configured and / or structured to perform operations corresponding to the machine-readable instructions. Additionally or alternatively, the application processing circuit 306 may be instantiated by any other combination of hardware, software, and / or firmware. The application processing circuit 306 may, for example, be implemented by at least one or more hardware circuits (e.g., a processor circuit, a discrete and / or integrated analog and / or digital circuit, an FPGA, an ASIC, an XPU, a comparator, an operational amplifier (op-amp), a logic circuit, etc.).) that are configured and / or structured to execute some or all machine-readable instructions and / or to perform some or all operations corresponding to the machine-readable instructions without executing any software or firmware, but other structures are equally suitable.

[0080] In the illustrated example of Fig. 3, the rate calculation circuit 308 is coupled to the queue of data packet flows 304 and the feedback traffic generation circuit 312. In some examples, the rate calculation circuit 308 is instantiated by a programmable circuit that executes rate calculation instructions and / or is configured to perform operations such as those illustrated by the flowcharts of the Fig. 10 and Fig. 11. In exemplary operation of the feedback-capable device 300, the rate calculation circuit 308 calculates (e.g., periodically initiated by a trigger (e.g., a trigger from a source device), etc.) a bit rate used by the feedback-capable device 300 to transmit a data packet flow and sends a feedback data packet over one or more hops to a source device to identify the bit rate for the source device.

[0081] In the illustrated example of Fig. 3, as described above, a data packet flow received and transmitted by the feedback-capable device 300 is associated with a workload. Furthermore, as described above, the feedback-capable device 300 includes the application processing circuit 306, which is capable of performing application-level processing on data contained in the data packet flow. In the example of Fig. 3, the rate calculation circuit 308 determines a delay associated with the processing of data associated with the workload in the feedback-capable device 300. The delay may, for example, be calculated on a per data packet flow basis, with the respective delays corresponding to a delay associated with the processing of data associated with a workload with which a data packet flow is associated. Additionally or alternatively, the delay may be calculated as a composite measure of the delay for processing data in the feedback-capable device 300 over a certain period of time. In some examples, the delay may be calculated on a per data packet basis. In the example of Fig. 3, to determine a delay associated with processing data associated with the workload in the feedback-capable device 300, the rate calculation circuit 308 determines a difference between a total delay in the feedback-capable device 300 for a data packet flow and a queuing delay in the queue of data packet flows 304 for the data packet flow.

[0082] For example, when the last data packet of a data packet flow is transmitted by the transmitter circuit 314, the rate calculation circuit 308 determines a fourth timestamp for the last data packet. To determine the total delay in the feedback-capable device 300 for a data packet flow, the rate calculation circuit 308 determines a difference between the fourth timestamp (e.g., the transmission time) of a last data packet of the data packet flow and the first timestamp (e.g., the reception time) of an initial data packet of the data packet flow. Additionally or alternatively, the rate calculation circuit 308 may determine the total delay in the feedback-capable device 300 by starting a timer (e.g., a counter) at the reception time and stopping the timer at the transmission time. In the example of the Fig. 3, to determine the queuing delay, the rate calculation circuit 308 determines the difference between the third timestamp (e.g., the dequeue time) of the last data packet of the data packet flow and the second timestamp (e.g., the enqueue time) of the initial data packet of the data packet flow. Additionally or alternatively, the rate calculation circuit 308 may determine the queuing delay associated with the queue of data packet flows 304 by starting a timer (e.g., a counter) at the enqueue time and stopping the timer at the dequeue time.

[0083] In the illustrated example of Fig. 3, the rate calculation circuit 308 determines a bit rate (e.g., a fair rate) that the transmitter circuit 314 will use to transmit a data packet flow based on a utilization of the queue of data packet flows 304 by the data packet flow (e.g., a queue size for the data packet flow) and on a delay associated with processing data of the data packet flow in the feedback-capable device 300. As such, the rate calculation circuit 308 considers the queuing delay and the processing delay (e.g., a calculation delay) in the feedback-capable device 300 while calculating the bit rate for a data packet flow. In some examples, the rate calculation circuit 308 compares the bit rate to an upper bit rate threshold and a lower bit rate threshold for the feedback-capable device 300. Fig. 10 shows example machine-readable instructions and / or example operations for calculating a bit rate used to transmit a data packet flow based on an intra-network computation delay.

[0084] In some examples, rate calculation circuitry 308 implements approximately fair dropping (AFD) based on queue utilization by data packet flows and processing delays for the data packet flows. For example, it evaluates a first delay associated with processing data of a first data packet flow and a second delay associated with processing data of a second data packet flow. Due to the first delay being greater than the second delay, rate calculation circuitry 308 decreases a bit rate used to transmit the second data packet flow so that a bit rate used to transmit the first data packet flow may be increased.In general, rate calculation circuitry 308 may prioritize bit rates assigned to data packet flows in ascending order of the data tuple, the utilization of the data packet flow queue 304 by a data packet flow, and a processing delay associated with processing the data of the data packet flow (data packet flow queue utilization, data packet flow processing delay). In the example of FIG. Fig. 3, the rate calculation circuit 308 forwards the bit rate to the feedback traffic generation circuit 312.

[0085] In some examples, feedback-capable device 300 includes elements for calculating a bit rate. The elements for calculating a bit rate may be implemented, for example, by rate calculation circuitry 308. In some examples, rate 308 may be implemented by a programmable circuit, such as exemplary programmable circuitry 1412 of Fig. 14. For example, the rate calculation circuit 308 may be implemented by the exemplary microprocessor 1500 of the Fig. 15 that executes machine-executable instructions, such as those specified by at least blocks 1002, 1004, 1006, 1008, 1010, 1012, 1014, 1016, 1018, 1020, 1022, 1024, 1026 and 1030 of the Fig. 10 and / or at least blocks 1102, 1104, 1106, 1108, 1110, 1112, 1114 and 1116 of the Fig. 11. In some examples, the rate calculation circuit 308 may be instantiated by a hardware logic circuit implemented by an ASIC, an XPU, or the FPGA circuit 1600 of the Fig. 16 that are configured and / or structured to perform operations corresponding to the machine-readable instructions. Additionally or alternatively, the rate calculation circuit 308 may be instantiated by any other combination of hardware, software, and / or firmware. The rate calculation circuit 308 may, for example, be implemented by at least one or more hardware circuits (e.g., a processor circuit, a discrete and / or integrated analog and / or digital circuit, an FPGA, an ASIC, an XPU, a comparator, an operational amplifier (op-amp), a logic circuit, etc.) that are configured and / or structured to execute some or all of the machine-readable instructions and / or to perform some or all of the operations corresponding to the machine-readable instructions without executing any software or firmware, but other structures are equally suitable.

[0086] In the illustrated example of Fig. 3, the data storage for metadata of a data packet flow 310 is coupled to the receiver circuit 302 and the transmitter circuit 314. In the example of Fig. 3, the feedback-capable device 300 includes the data packet flow metadata store 310 for recording data associated with one or more data packets received at the feedback-capable device 300. The data packet flow metadata store 310 maintains a data flow table that identifies the respective data packets associated with the data packets received at the feedback-capable device 300 and other metadata indicating the characteristics of the respective data packet flows. Example metadata includes a data flow ID of a data packet flow, a feedback flag for the data packet flow, a bit rate used by a source device to transmit the data packet flow, a utilization of a queue of the source device by the data packet flow (e.g.,a queue size for the data packet flow), a delay measurement for transmitting the data packet flow through a network, an ID of the source device, and an ID of a destination device of the data packet flow.

[0087] In the illustrated example of Fig. 3, an acknowledgment flag indicates whether a source device of a data packet flow has requested acknowledgment associated with the data packet flow. In the examples disclosed herein, an acknowledgment flag may be implemented as a binary variable. An example dataflow table is illustrated below in Table 1. As illustrated below, the dataflow table maintained by the data packet flow metadata store 310 allows the acknowledgment-capable device 300 to keep track of which devices should receive acknowledgment messages. Table 1 Date nflus s-ID Rückmel deflag Bitrate Warteschlang engröße Übertragungsverz ögerung Quellvorri chtungs-ID Zielvorrich tungs-ID A 0 250 Megabit pro Sekunde (Mbit / s) 50 Megabytes (MBs) 100 Mikrosekunden (µs) A1 A2 B 1 500 Mbit / S 100 MBs 200 µs B1 B2

[0088] In the example of Fig. 3, the size of the data flow table maintained by the data store for metadata of a data packet flow 310 is limited by the data tuple of the queue of data packet flows 304 and the processing delay associated with the feedback-capable device 300 (e.g., [size of the queue of data packet flows 304, processing delay associated with the feedback-capable device 300]). As described above, the delay may be calculated on a per data packet basis, per data packet flow basis, and / or as a composite measure for processing data in the feedback-capable device 300. In the example of Fig. 3, the feedback-capable device 300 sizes the data flow table based on the size of the queue of data packet flows 304 and the delay associated with processing data in the feedback-capable device 300.

[0089] For example, the dataflow table maintained in the data packet flow metadata store 310 may store metadata for a limited number of data packet flows based on the size of the data packet flow queue 304. By implementing a lower bit rate threshold for the rate calculation circuit 308 as described above, the dataflow table maintained in the data packet flow metadata store 310 has an upper limit on the number of concurrent data packet flows that can be supported by the feedback-capable device 300. Furthermore, the data packet flow metadata store 310 may take data packet flow delay into account when sizing the dataflow table.As such, the data packet flow metadata store 310 may dequeue data packet flows from the data flow table based on the data tuple of the queue size of data packet flows 304 and the processing delay associated with the feedback-capable device 300 (e.g., [the queue size of data packet flows 304, the processing delay associated with the feedback-capable device 300]). In additional or alternative examples, the data flow table may be implemented differently, provided the additional or alternative implementations jointly include delay and queue state information that can be used to filter the data flows.

[0090] In the illustrated example of Fig. 3, the data storage for metadata of a data packet flow 310 may be implemented by a volatile memory (e.g., an SDRAM, a DRAM, an RDRAM, etc.) and / or a non-volatile memory (e.g., a flash memory). The data storage for metadata of a data packet flow 310 may additionally or alternatively be implemented by one or more DDR memory, such as a DDR, DDR2, DDR3, DDR4, DDR5, mDDR, etc. The data storage for metadata of a data packet flow 310 may additionally or alternatively be implemented by one or more mass storage devices, such as HDDs, CD drives, DVD drives, SSD drives, etc.Although the illustrated example illustrates the data packet flow metadata data store 310 as a single data store, the data packet flow metadata data store 310 may be implemented using any number and / or types of data stores. Furthermore, the data stored in the data packet flow metadata data store 310 may be in any data format, such as binary data, comma-separated data, tab-separated data, structured SQL structures, etc.

[0091] In the illustrated example of Fig. 3, the feedback traffic generation circuit 312 is coupled to the rate calculation circuit 308 and the transmitter circuit 314. In some examples, the feedback traffic generation circuit 312 is instantiated by a programmable circuit that executes the feedback generation instructions and / or is configured to perform operations such as those illustrated by the flowchart of Fig. 10. In exemplary operation of the feedback-capable device 300, the feedback traffic generation circuitry 312 generates a feedback message (e.g., one or more feedback data packets) and causes the feedback message to be transmitted to the source device of a data packet flow or the previous immediate neighbor device in the communication path so that the feedback message can be forwarded to the source device.

[0092] For example, if the previous immediate neighbor device is not the source device, the feedback traffic generation circuit 312 causes a transmission of the feedback message to the previous immediate neighbor device in the communication path so that the feedback message can be forwarded to the source device. In some examples, the feedback message is referred to as a Congestion Notification Packet (CNP). As such, the source device can use the CNP to reduce (e.g., minimize) the queue growth in the feedback-capable device 300 through rate control in the source device. In the example of Fig. 3, the CNP encapsulates the fair rate that must be used by the feedback-capable device 300 to transmit a data packet flow and a data flow ID for the data packet flow. For example, the CNP may include a data tuple indicating: (a) the bit rate that must be used by the feedback-capable device 300 to transmit a data packet flow, (b) the use of the queue of data packet flows 304 by the data packet flow, c) a processing delay associated with processing the data of the data packet flow in the feedback-capable device 300, d) a data flow ID for the data packet flow, e) an ID of a source device of the data packet flow, and (f) an ID of a destination device of the data packet flow (e.g., [a bit rate of a data packet flow, a queue utilization by the data packet flows, a processing delay for the data packet flows, a data flow ID, a source ID, a destination ID]).

[0093] In some examples, feedback-capable device 300 includes feedback generating elements. For example, the feedback generating elements may be implemented by feedback traffic generating circuitry 312. In some examples, feedback traffic generating circuitry 312 may be instantiated by a programmable circuit, such as the exemplary programmable circuitry 1412 of Fig. 14. The feedback traffic generation circuit 312 may be implemented, for example, by the exemplary microprocessor 1500 of the Fig. 15 that executes machine-executable instructions such as those specified by at least block 1028 of the Fig. 10. In some examples, the feedback traffic generation circuit 312 may be instantiated by hardware logic circuitry implemented by an ASIC, an XPU, or the FPGA circuitry 1600 of the Fig. 16 that are configured and / or structured to perform operations corresponding to the machine-readable instructions. Additionally or alternatively, the feedback traffic generation circuit 312 may be instantiated by any other combination of hardware, software, and / or firmware. The feedback traffic generation circuit 312 may, for example, be implemented by at least one or more hardware circuits (e.g., a processor circuit, a discrete and / or integrated analog and / or digital circuit, an FPGA, an ASIC, an XPU, a comparator, an operational amplifier (op-amp), a logic circuit, etc.) that are configured and / or structured to execute some or all machine-readable instructions and / or to perform some or all operations corresponding to the machine-readable instructions without executing any software or firmware, but other structures are equally suitable.

[0094] As in Fig. 3, the data flow table implemented by the data packet flow metadata store 310 simplifies sending feedback messages (CNPs) for selected data packet flows (e.g., elephant flows). For example, the rate calculation circuit 308 changes the fair rate for a congested data packet flow until the arrival rate for the congested data packet flow matches the dequeue rate 304 for the congested data packet flow. Therefore, the fair rate for the congested data packet flow will change at F=Cl−BRmiceN stabilize, where C l represents the bandwidth of the feedback-capable device 300, BW micerepresents the bandwidth of the feedback-capable device 300 allocated to data packet flows that do not contribute to congestion (e.g., mouse flow), and N is the number of data packet flows that do contribute to congestion. Accordingly, the rate calculation circuit 308 tracks the data packet flows that contribute to congestion and queue buildup in the feedback-capable device 300.

[0095] In the illustrated example of Fig. 3, the transmitter circuit 314 is coupled to the data packet flow queue 304, the data packet flow metadata storage 310, and the feedback traffic generation circuit 312. In exemplary operation of the feedback-capable device 300, the transmitter circuit 314 transmits data packets of a data packet flow at the bit rate specified by the rate calculation circuit 308. For example, the transmitter circuit 314 transmits the data packets of the data packet flow to the next device in a communication path of the data packet flow. In addition, the transmitter circuit 314 also transmits a CNP to the previous device in the communication path.

[0096] In some examples, the feedback-capable device 300 includes circuitry for communicating one or more INC capabilities of the feedback-capable device 300 to an EH device. Such circuitry operates, for example, in the middle layer of the feedback-capable device 300. In the example of Fig. 3, the circuitry of the feedback-capable device 300 operating in the middle layer exchanges one or more messages containing contextual information of an application or a communication history with the neighboring devices in the network. Therefore, the circuitry of the feedback-capable device 300 operating in the middle layer facilitates the discovery of potential communication paths by an EH device through the neighboring devices.

[0097] Fig. 4A is an illustration of an example data packet 400 including an example header 402 according to examples disclosed herein. Data packet 400 corresponds, for example, to a data packet traversing the communication path from a source device through one or more IN devices to a destination device and / or vice versa. In the example of Fig. 4A, the header 402 includes an example workload class (WL) field 404, an example explicit congestion notification (ECN) field 406, an example transport type field 408, an example delay level field 410, an example queue information field 412, an example fair rate field 414, and an example hop counter field 416.

[0098] In the illustrated example of Fig. 4A, the workload class field 404 (e.g., a header field) is a 6-bit field indicating a WL class with which the data packet 400 is associated. The WL class field 404 functions as a flag that allows an IN device to opportunistically perform one or more operations (e.g., primitives, transport type functions, etc.) related to the workload class. In some examples, a device may modify the WL class field 404 after an action has been performed on the data packet 400 by removing the information in the WL class field 404. In this way, a subsequent device in the communication path of the data packet 400 will not repeat the action performed on the data packet 400. In the example of Fig. 4A, the ECN field 406 is a 4-bit field that indicates whether one or more of the first ECN flag or the second ECN flag is set for the data packet 400. For example, if both the first ECN flag and the second ECN flag are set, one bit of the ECN field 406 (e.g., a binary value) may be used to indicate the state of the first and second ECN flags. If one or both of the first ECN flag and the second ECN flag are set, additionally or alternatively, multiple bits of the ECN field 406 may be used to indicate the state of the first and second ECN flags.

[0099] In the illustrated example of Fig. 4A, the transport type field 408 is an 8-bit field indicating a transport type with which the data packet 400 is associated. In some examples, the transport type field 408 (e.g., a transport type header field and / or an application type header field) depends on the WL class identified in the WL class field 404 (e.g., a workload class header field). Therefore, when a device receives the data packet 400, the WL class field 404 and / or the transport type field 408 of the data packet 400 can trigger a specific type of transport protocol corresponding to various types of application-layer WL classes, ranging from network-delay applications and / or congestion-sensitive applications to processing-time-sensitive data flows.Example transport types include a Transmission Control Protocol (TCP), a User Datagram Protocol (UDP), a Stream Control Transmission Protocol (SCTP), a Datagram Congestion Control Protocol (DCCP), a Remote Desktop Protocol (RDP), and a Reliable User Datagram Protocol (RUDP). Other examples of transport types include custom transports such as those provided by InfiniBand and Google.

[0100] In the illustrated example of Fig. 4A, the delay level field 410 is an 8-bit field indicating the processing delay associated with a data packet flow with which the data packet 400 is associated. The processing delay represented in the delay level field 410 corresponds, for example, to the delay associated with processing data associated with a workload in a device transmitting the data packet 400, with the data packet 400 associated with the workload. Based on the processed delay communicated in the delay level field 410, an E2E delay in EH devices can be calculated (e.g., an RTT delay). To implement the delay level field 410, the processing delay in the device is available once the data packet 400 is transferred from an inlet to an outlet of a device (e.g.,from a receiver of a device to a transmitter of a device). Therefore, the device adds the delay stage field 410 to the header 402. In the example of the . Fig. 4A, the delay stage field 410 is quantified in bits. In the example of Fig. 4A, the queue information field 412 is a 6-bit field representing the utilization of a queue of a source device of the data packet 400.

[0101] In the illustrated example of Fig. 4A, the fair rate field 414 is a 4-bit field that indicates the bit rate used by the source device to transmit the data packet 400. In the example of Fig. 4A, the fair rate field 414 may be updated after the source device receives a CNP. Furthermore, the example hop counter field 416 is a 4-bit field indicating a forwarding and / or routing history of the path traversed by the data packet 400. As such, the hop counter field 416 maintains network topology knowledge. The example of Fig. 4A illustrates example sizes of the data fields in the header 402. In additional or alternative examples, each of the data fields of the header 402 may have different sizes.

[0102] As in Fig. 4A, the header 402 includes first header fields (e.g., the WL class field 404, the ECN field 406, the transport type field 408, the delay level field 410, the queue information field 412, the fair rate field 414, and the hop counter field 416). In addition, the header 402 contains second header fields (e.g., a version field, an Internet Header Length (IHL) field, a total length field, a packet ID field, a fragment flag field, a fragment offset field, a validity period field, a header checksum field, a source port address field, a destination port address field, a timestamp field, a retransmission (RTX) field, an acknowledgment (ACK) field, an options field, a NIC receiver (RX) timestamp field, etc.). In the example of the Fig. 4A, the first header fields refer to network-internal calculations and network congestion control based on delays in network-internal calculations. In the example of Fig. 4A, one or more of the second header fields also refer to a protocol for communication of the data packet 400.

[0103] Fig. 4B is a further illustration of an example data packet 418 including an example header 420 according to examples disclosed herein. Data packet 418 corresponds, for example, to a data packet traversing the communication path from a source device through one or more IN devices to a destination device and / or vice versa. In the example of Fig. 4B, the header 420 includes the example WL class field 404, the example ECN field 406, the example transport type field 408, the example delay level field 410, the example queue information field 412, the example fair rate field 414, and the example hop counter field 416. In the example of Fig. 4B, header 420 includes an exemplary display field 422 that may be implemented in addition to or instead of the options field described above. Display field 422 may, for example, be between one and 32 bits long.

[0104] As in Fig. 4B, the header 420 includes first header fields (e.g., the WL class field 404, the ECN field 406, the transport type field 408, the delay level field 410, the queue information field 412, the fair rate field 414, the hop counter field 416, and the indicator field 422). In addition, the header 420 includes second header fields (e.g., a version field, an IHL field, a total length field, a data packet ID field, a frag flag field, a fragment offset field, a validity period field, a header checksum field, a source port address field, a destination port address field, a timestamp field, an RTX field, an ACK field, an options field, a NIC RX timestamp field, etc.). In the example of Fig. 4B, the first header fields refer to intra-network calculations and network congestion control based on delays in intra-network calculations. In the example of Fig. 4B, one or more of the second header fields also refer to a protocol for communication of the data packet 418.

[0105] In the example of Fig. 4B, the indicator field 422 is implemented as an explicit field containing information mapping to a preset list of operations, functions, primitives, and / or processing that can be performed by IN devices on a data packet. For example, the device transmitting the data packet 418 sets a value in the indicator field 422 associated with a corresponding action in a LUT of the device that was performed on the data packet 418. Additionally or alternatively, the indicator field 422 is implemented as an explicit completion flag. For example, the device transmitting the data packet 418 sets an explicit completion flag (e.g., a 1-bit field) in the indicator field 422 that specifies a state of an associated workload (e.g., whether the workload was completed during transmission or processed at all).

[0106] Although an exemplary manner of implementing the layered application device 200 of the Fig. 2 in Fig. 2, one or more of the elements, one or more of the processes and / or one or more of the devices shown in Fig. 2 may be combined, divided, rearranged, omitted, eliminated, and / or implemented in other ways. Although an exemplary manner of implementing the feedback-capable device 300 of Fig. 3 in Fig. 3, one or more of the elements, one or more of the processes and / or one or more of the devices shown in Fig. 3 may be illustrated, combined, divided, rearranged, omitted, eliminated and / or implemented in any other way. Furthermore, the example application execution circuitry 208, the example command and completion queues 210, the example operation scheduling circuitry 212, the example operation-to-stream mapping circuitry 214, the example dataflow-to-operation mapping circuitry 216 and / or the example delay determination circuitry 222, the example queue analysis circuitry 224, the example rate calculation circuitry 226, the example rate adaptation circuitry 228 and / or more generally the example congestion control circuitry 218 and / or the example NIC queues 230, the example transmitter and / or receiver circuitry 232, and / or more generally the example NIC 220 and / or more generally the layered application device 200 of the Fig. 2, and / or the exemplary receiver circuit 302, the exemplary queue of data packet flows 304, the exemplary application processing circuit 306, the exemplary rate calculation circuit 308, the exemplary data store for metadata of a data packet flow 310, the exemplary circuit for generating a feedback data traffic 312, the exemplary transmitter circuit 314, and / or more generally, the exemplary feedback-capable device 300 of Fig. 3 be implemented by hardware alone or by hardware in combination with software and / or firmware. In this way, each of the example application execution circuitry 208, the example command and completion queues 210, the example operation scheduling circuitry 212, the example operation-to-stream mapping circuitry 214, the example dataflow-to-operation mapping circuitry 216, and / or the example delay determination circuitry 222, the example queue analysis circuitry 224, the example rate calculation circuitry 226, the example rate adaptation circuitry 228, and / or more generally the example congestion control circuitry 218, and / or the example NIC queues 230, the example transmitter and / or receiver circuitry 232, and / or more generally the example NIC 220, and / or more generally the layered application device 200 of the Fig. 2, and / or the exemplary receiver circuit 302, the exemplary queue of data packet flows 304, the exemplary application processing circuit 306, the exemplary rate calculation circuit 308, the exemplary data store for metadata of a data packet flow 310, the exemplary circuit for generating a feedback data traffic 312, the exemplary transmitter circuit 314, and / or, more generally, the exemplary feedback-capable device 300 of the Fig. 3 may be implemented by a programmable circuit in combination with machine-readable instructions (e.g., firmware or software), a processor circuit, analog circuits, digital circuits, logic circuits, programmable processors, programmable microcontrollers, graphics processing units (GPUs), digital signal processors (DSPs), ASICs, programmable logic devices (PLDs), and / or field-programmable logic devices (FPLDs), such as FPGAs. Furthermore, the exemplary layered application device 200 of the Fig. 2 contain one or more elements, one or more processes and / or one or more devices in addition to or instead of those described in Fig. 2, and / or may include more than one or all of the illustrated elements, processes, and devices. In addition, the exemplary feedback-capable device 300 of Fig. 3 contain one or more elements, one or more processes and / or one or more devices in addition to or instead of those described in Fig. 3, and / or it may include more than one or all of the illustrated elements, processes, and devices.

[0107] Fig. 5 is a block diagram 500 illustrating an overview of a configuration for edge computing that includes a processing layer, which will be referred to as an "edge cloud" in many of the following examples. As shown, the edge cloud 510 is located at an edge location, such as an access point or base station 540, a local processing hub 550, or a central office 520, and may therefore include multiple entities, devices, and equipment instances. The edge cloud 510 is located much closer to the endpoint data sources (consumers and manufacturers) 560 (e.g., in autonomous vehicles 561, user devices 562, commercial and industrial equipment 563, video capture devices 564, drones 565, smart city and building devices 566, sensors and IoT devices 567, etc.) than the cloud data center 530.Compute, memory, and storage resources offered at the edges of the edge cloud 510 are critical for providing ultra-low latency response times for services and functions used by the endpoint data sources 560, as well as for reducing network backhaul traffic from the edge cloud 510 to a cloud data center 530, thereby improving power consumption and overall network utilization, among other benefits.

[0108] Computing, memory, and storage resources are scarce resources that typically decrease depending on the edge location (e.g., fewer processing resources are available in consumer endpoint devices than in a base station or central office). However, the closer the edge location is to the endpoint (e.g., the user equipment (UE)), the more space and performance are often constrained. Therefore, edge computing attempts to reduce the amount of resources required for network services by distributing more resources that are closer to the endpoints, both geographically and in terms of network access times. In this way, edge computing attempts to bring computing resources to the workload data where appropriate, or to bring the workload data to computing resources.

[0109] The following describes aspects of an edge cloud architecture that covers several potential deployment areas and addresses limitations that some network operators or service providers may have in their own infrastructures. These include a variation of configurations based on edge location (since edges at a base station level, for example, in a multi-tenant scenario, may have limited capabilities and capacity); configurations based on the type of compute, memory, storage, fabric, and acceleration resources, or similar resources available at edge locations; site tiers or site groups; service, security, and management and orchestration capabilities; and the associated objectives for achieving usability and performance of the end services.These deployments can perform processing in network layers that can be considered as "near-edge", "proximal-edge", "local-edge", "mid-edge", or "far-edge" layers depending on latency, distance, and timing characteristics.

[0110] Edge computing is an emerging paradigm in which data processing is performed at or closer to the edge of a network, typically through the use of a computing platform (e.g., an x86 or ARM computer hardware architecture) implemented in base stations, gateways, network routers, or other devices located much closer to the endpoint devices that produce and consume the data. Edge gateway servers, for example, can be equipped with memory pools and data storage resources to perform real-time computations for low-latency use cases (e.g., autonomous driving or video surveillance) for connected client devices.Or, as one example, base stations can be enhanced with computing and acceleration resources to directly process service workloads for connected user devices, without the need to additionally communicate data over backhaul networks. Or, as another example, a central office's network management hardware can be replaced with standardized computing hardware that performs virtualized network functions and provides computing resources for running services and consumer functions for connected devices. Within edge computing networks, there may be scenarios for services where the computing resource is "offset" to the data, as well as scenarios where the data is "offset" to the computing resource.Or, as an example, the base station's compute, acceleration, and network resources can provide services to scale to workload needs on an as-needed basis, by activating dormant capacity (contribution, on-demand capacity) to manage exceptions or emergencies, or to provide longevity for deployed resources over a significantly longer implemented lifecycle.

[0111] Fig. Figure 6 illustrates operational layers between endpoints, an edge cloud, and cloud computing environments. In particular, Fig. 6 illustrates examples of computing use cases 605 in which the edge cloud 510 is used in several illustrative layers of network computing. The layers begin in an endpoint layer (devices and things) 600, which accesses the edge cloud 510 to perform data creation, analysis, and data consumption activities. The edge cloud 510 may span multiple network layers, such as an edge device layer 610, which includes gateways, on-premises servers, or a network device (node ​​615) located in physically adjacent edge systems; a network access layer 620, which includes base stations, radio processing units, network hubs, regional data centers (DCs), or local network devices (device 625); and any devices, devices, or nodes located in between (in layer 612, not shown in detail).The network communications within the edge cloud 510 and between the various layers may occur over any number of wired or wireless media, including through-hole connectivity architectures and technologies not shown.

[0112] Examples of latencies resulting from constraints imposed by network communication distance and processing time can range from less than one millisecond (ms) between devices in the endpoint layer 600, less than 5 ms in the edge device layer 610, and even between 10 and 40 ms when communicating with nodes in the network access layer 620. Outside of the edge cloud 510 are the core network 630 and cloud data center 640 layers, each with decreasing latency (e.g., between 50 and 60 ms in the core network layer 630 and 100 ms or more in the cloud data center layer). Therefore, operations in a core network 635 or cloud data center 645 data center with latencies of at least 50 to 100 ms or more are unable to perform many time-critical functions of use cases 605.Each of these latency values ​​is provided for illustrative and comparative purposes; it is understood that the use of other access network media and technologies can further reduce latencies. In some examples, corresponding sections of the network may be categorized as "near-edge," "local-edge," "proximity-edge," "mid-edge," or "far-edge" layers relative to a network source and destination.For example, from the perspective of the core network data center 635 or a cloud data center 645, a central office or content data network may be considered to be within a "near-edge" layer ("close" to the cloud, which has high latency values ​​when communicating with the devices and endpoints of the use cases 605), whereas an access point, base station, on-premises server, or network gateway may be considered to be within a "far-edge" layer ("far" from the cloud, which has low latency values ​​when communicating with the devices and endpoints of the use cases 605).It is understood that other categorizations of a particular network layer as a "near edge," "local edge," "proximate edge," "mid edge," or "far edge" may be formed based on latency, distance, number of network hops, or other measurable characteristics when measuring from a source in any of the network layers 600 to 640.

[0113] The various use cases 605 may access resources under pressure from incoming data streams utilizing the edge cloud for multiple services. To achieve low-latency results, the services executing within the edge cloud 510 balance varying requirements in the following sense: (a) priority (throughput or latency) and quality of service (QoS) (e.g., road traffic may have a higher priority for an autonomous vehicle than a temperature sensor in terms of a response time requirement; or, depending on the application, there may be a performance sensitivity / bottleneck in a computer / accelerator, memory, storage, or network resource); (b) reliability and resiliency (e.g.,some input data streams must be processed and the traffic must be forwarded with mission-critical reliability, while some other input data streams can tolerate occasional failure, depending on the application) and (c) physical constraints (e.g., power supply, cooling, and form factor).

[0114] The end-to-end service view for these use cases incorporates the concept of a service flow and is associated with a transaction. The transaction displays the details of the entire service request for the entity consuming the service, as well as the associated services for the resources, workloads, workflows, and requirements for the trading and trading layer functions. Services executed under the described "conditions" can be managed at each layer in a manner that ensures contractual compliance for the transaction throughout the service lifecycle, both in real time and at runtime.If a component in the transaction misses its agreed SLA, the system as a whole (components in the transaction) can provide the capability to (1) understand the impact of the SLA violation, (2) augment other components in the system to reinstate the overall SLA of the transaction, and (3) implement actions for remediation.

[0115] Taking these variations and service characteristics into account, edge computing within edge cloud 510 can thus provide the ability to serve and respond to multiple applications of use cases 605 (e.g., object tracking, video surveillance, connected vehicles, etc.) in real-time or near-real-time, and meet the ultra-low latency requirements for these diverse applications. These benefits enable an entirely new class of applications (Virtual Network Functions (VNFs), Function as a Service (FaaS), Edge as a Service (EaaS), standard processes, etc.) that have not been able to utilize traditional cloud computing due to latency or other limitations.

[0116] However, the benefits of edge computing come with the following caveats. Devices located at the edge are often resource-constrained, and therefore there is pressure to utilize edge resources. Typically, this is addressed by pooling memory and storage resources for use by multiple users (tenants) and devices. The edge may be power- and cooling-constrained, and therefore, energy consumption must be considered for the applications that consume the most energy. There may be inherent power-performance trade-offs with these pooled memory resources, as many are likely to use new memory technologies where more power requires greater memory bandwidth.Likewise, enhanced hardware security and trusted root of trust capabilities are also required, as edge locations may be unmanned and even require authorized access (e.g., if they are hosted at a third-party site). Such issues are amplified in the edge cloud 510 in a multi-tenant, owner-permit, or access-permit environment where services and applications are requested by many users, especially when network usage fluctuates dynamically and the composition of different stakeholders, use cases, and services changes.

[0117] At a more generic level, an edge computing system can be described as including any number of deployments in the previously discussed layers operating in the edge cloud 510 (network layers 600 through 640) that provide coordination of client and distributed computing devices. One or more edge gateway nodes, one or more edge aggregation nodes, and one or more core data centers may be distributed across layers of the network to provide an implementation of the edge computing system by or on behalf of a telecommunications service provider ("Telco," or "TSP"), an Internet of Things service provider, a cloud service provider (CSP), an enterprise entity, or any number of other entities.Different implementations and configurations of the edge computing system can be deployed dynamically, such as when orchestrated to meet service objectives.

[0118] Consistent with the examples provided herein, a client computing node may be used as any type of endpoint component, device, appliance, or other component capable of communicating as a data producer or consumer of data. As used in the edge computing system, the term 'node' or 'device' does not necessarily imply that that node or device functions in a client or agent / companion / follower role; rather, all nodes or devices in the edge computing system refer to individual entities, nodes, or subsystems that include discrete or connected hardware or software configurations to facilitate or utilize the edge cloud 510.

[0119] As such, the edge cloud 510 is comprised of network components and functional features operated by and within edge gateway nodes, edge aggregation nodes, or other edge computing nodes between the network layers 610 to 630. The edge cloud 510 may thus be referred to as any type of network that provides edge computing and / or data storage resources located proximate to Radio Access Network-capable endpoint devices (RAN-capable endpoint devices) (e.g., mobile computing devices, IoT devices, smart devices, etc.) discussed herein.In other words, the edge cloud 510 can be conceived as an "edge" connecting endpoint devices and traditional network access points that serve as an entry point into the service providers' core networks. The core networks include mobile bearer networks (e.g., GSM (Global System for Mobile Communications) networks, LTE (Long-Term Evolution) networks, 5G / 6G networks, etc.), while also providing data storage and / or computing capabilities. Other types and forms of network access (e.g., Wi-Fi, wireless wide area networks, wired networks, including optical networks) can also be used instead of or in combination with these 3GPP bearer networks.

[0120] The network components of edge cloud 510 may be servers, multi-tenant servers, application computing devices, and / or any other type of computing device. For example, edge cloud 510 may include an application computing device that is a standalone electronic device having a housing, chassis, shell, or enclosure. Under certain circumstances, the housing may be dimensioned for portability so that it can be worn and / or transported by a human. Example enclosures may include materials that form one or more exterior surfaces that partially or fully protect the contents of the device, where the protection may include weather protection, protection from hazardous environmental conditions (e.g., EMI, vibration, extreme temperatures), and / or allow for submersibility.Example enclosures may include power circuitry for providing power to stationary and / or portable implementations, such as AC inputs, DC inputs, AC / DC or DC / AC converters, power regulators, transformers, charging circuitry, batteries, wired inputs, and / or wireless power inputs. Example enclosures and / or surfaces thereof may include or be coupled to mounting hardware to enable attachment to structures such as buildings, telecommunications structures (e.g., poles, antenna structures, etc.), and / or racks (e.g., server racks, blade holders, etc.). Example enclosures and / or surfaces thereof may support one or more sensors (e.g., temperature sensors, vibration sensors, light sensors, acoustic sensors, capacitive sensors, proximity sensors, etc.).One or more such sensors may be contained within, carried by, or otherwise embedded in the surface of the device and / or mounted on the surface of the device. Example housings and / or surfaces thereof may support mechanical connectivity elements, such as drive hardware (e.g., wheels, propellers, etc.) and / or articulated hardware (e.g., robotic arms, pivoting accessories, etc.). In certain circumstances, the sensors may include any type of input device, such as user interface hardware (e.g., buttons, switches, dials, sliders, etc.). In certain circumstances, example housings include output devices contained within, carried by, embedded in, and / or attached thereto. The output devices may include displays, touchscreens, lights, LEDs, speakers, I / O ports (e.g., USB), etc.In certain circumstances, edge devices are devices that are presented in the network for a specific purpose (e.g., a traffic light), but may also have processing capacity and / or other capabilities that can be used for other purposes. Such edge devices may be independent of other networked devices and may be provided with a housing that has a form factor suitable for their primary purpose; however, they may also be available for other computing tasks that do not affect their primary task. Edge devices also include Internet of Things devices. The application computing device may include hardware and software components to manage local issues, such as device temperature, vibration, resource utilization, updates, power issues, physical and network security, etc.The edge cloud 510 may also include one or more servers and / or one or more multi-tenant servers. Such a server may include an operating system and implement a virtual computing environment. A virtual computing environment may include a hypervisor that manages (e.g., spawns, deploys, destroys, etc.) one or more virtual machines, one or more containers, etc. Such virtual computing environments provide an execution environment in which one or more applications and / or other software, code, or scripts may be executed while isolated from one or more other applications, software, code, or scripts.

[0121] In Fig. 7, various client endpoints 710 (in the form of mobile devices, computers, autonomous vehicles, enterprise computing devices, industrial processing equipment) exchange requests and responses specific to the type of endpoint network aggregation. For example, client endpoints 710 may obtain network access over a wired broadband network by exchanging requests and responses 722 through an on-premises network system 732. Some client endpoints 710, such as mobile computing devices, may obtain network access over a wireless broadband network by exchanging requests and responses 724 through an access point (e.g., cellular tower) 734. Some client endpoints 710, such as autonomous vehicles, may obtain network access for requests and responses 726 over a wireless vehicular network through a road-based network system 736.Regardless of the type of network access, however, the TSP may deploy aggregation points 742 and 744 within edge cloud 510 to aggregate traffic and requests. Thus, the TSP may deploy various computing and data storage resources within edge cloud 510, such as edge aggregation nodes 740, to deliver the requested content. Edge aggregation nodes 740 and other systems of edge cloud 510 are connected to a cloud or data center 760, which uses a backhaul network 750 to fulfill higher-latency requests from a cloud / data center for websites, applications, database servers, etc.Additional or consolidated instances of the edge aggregation nodes 740 and the aggregation points 742, 744, including those deployed in a single server framework, may also exist within the edge cloud 510 or other areas of the TSP infrastructure.

[0122] The flowcharts representing exemplary machine-readable instructions that can be executed by a programmable circuit to (or the instructions, e.g., cause a programmable circuit to) the layered application device 200 of the Fig. 2 to implement and / or instantiate and / or represent exemplary operations that can be performed by a programmable circuit to implement the layered application device 200 of the Fig. 2 to implement and / or instantiate are described in the Fig. 8, Fig. 12 and / or 13. In addition, the flowcharts representing exemplary machine-readable instructions that can be executed by a programmable circuit to implement the feedback-capable device 300 of the Fig. 3 to implement and / or instantiate and / or represent exemplary operations that can be performed by a programmable circuit to (or the instructions, e.g., cause a programmable circuit to) the feedback-capable device 300 of Fig. 3 to implement and / or instantiate are described in the Fig. 9, Fig. 10 and / or 11. The machine-readable instructions may be one or more executable programs or portions of one or more executable programs for execution by a programmable circuit, such as the programmable circuit 1412 shown in the example of the programmable circuit platform 1400 described below, as further described below in connection with the Fig. 14, and / or they may be one or more functions or parts of functions performed by the exemplary programmable circuit (e.g., an FPGA) and described below in connection with the Fig. 15 and / or 16. In some examples, the machine-readable instructions cause an operation, task, etc., to be performed and / or performed in an automated manner in the real world. As used herein, "automated" means without human intervention.

[0123] The program may be embodied by instructions (e.g., software and / or firmware) embodied in one or more non-volatile, computer-readable and / or machine-readable data storage media such as a cache memory, a magnetic storage device or drive (e.g., a floppy disk, a hard disk drive, etc.), an optical data storage device or drive (e.g., a Blu-ray disk, a CD (compact disk), a DVD (digital versatile disk), etc.), a RAID system (redundant array of independent disks), a register, a ROM, an SSD drive (solid-state drive), SSD memory, non-volatile memory (e.g., electrically erasable programmable read-only memory (EEPROM), flash memory, etc.), volatile memory (e.g.,a random access memory (RAM) of any type, etc.) and / or any other data storage device or data storage drive. The instructions of the non-transitory computer-readable and / or machine-readable medium may program and / or be executed by programmable circuitry residing in one or more hardware devices; but the entire program and / or portions thereof may alternatively be executed and / or instantiated by one or more hardware devices different from the programmable circuitry and / or embodied in dedicated hardware. The machine-readable instructions may be distributed across multiple hardware devices and / or may be executed by two or more hardware devices (e.g., a server and a client hardware device).The client hardware device may be implemented, for example, by an endpoint client hardware device (e.g., a hardware device associated with a human and / or machine user) or an intermediate client hardware device gateway (e.g., a radio access network (RAN)) that may facilitate communication between a server and an endpoint client hardware device. Likewise, the non-transitory, computer-readable data storage medium may include one or more media. Although the example program is described with reference to the flowcharts included in the . Fig. 8, Fig. 9, Fig. 10, Fig. 11, Fig. 12 and / or 13, many other methods for implementing the exemplary layered application device 200 of the Fig. 2 and / or the feedback device 300 of the Fig. 3 may be used alternatively. For example, the order of execution of the blocks of the flowcharts may be changed and / or some of the described blocks may be changed, removed, or combined. Additionally or alternatively, some or all of the blocks of the flowchart may be implemented by one or more hardware circuits (e.g., a processor circuit, a discrete and / or integrated analog and / or digital circuit, an FPGA, an ASIC, a comparator, an operational amplifier (op-amp), or a logic circuit, etc.) structured to perform the corresponding operation without executing any software or firmware. The programmable circuit may be distributed across different network locations and / or locally on one or more hardware devices (e.g., a single-core processor (e.g., a single-core CPU), a multi-core processor (e.g., a multi-core CPU, an XPU, etc.)).The programmable circuit may be, for example, a CPU and / or an FPGA located in the same package (e.g., in the same integrated circuit (IC) package or in two or more separate packages), one or more processors in a single machine, multiple processors distributed across multiple servers of a server rack, multiple processors distributed across one or more server racks, etc., and / or any combination thereof.

[0124] The machine-readable instructions described herein may be stored in one or more of a compressed format, an encrypted format, a fragmented format, a compiled format, an executable format, a packed format, etc. The machine-readable instructions described herein may be stored as data (e.g., computer-readable data, machine-readable data, one or more bits (e.g., one or more computer-readable bits, one or more machine-readable bits, etc.), a bit stream (e.g., a computer-readable bit stream, a machine-readable bit stream, etc.), or a data structure (e.g., as part(s) of instructions, a code, representations of a code, etc.) that can be used to generate, manufacture, and / or produce machine-executable instructions.For example, the machine-readable instructions may be fragmented and stored in one or more data storage devices, on hard disks, and / or in computing devices (e.g., servers) located at the same location or at different locations of a network or collection of networks (e.g., in the cloud, on edge devices, etc.). The machine-readable instructions may require one or more of installation, modification, adaptation, updating, combination, addition, configuration, decryption, decompression, unpacking, distribution, reassignment, compilation, etc., to make them directly readable, interpretable, and / or executable by a computing device and / or another machine.For example, the machine-readable instructions may be stored in multiple pieces that are individually compressed, encrypted, and / or stored in separate computing devices, which pieces, when decrypted, decompressed, and / or combined, form a set of computer-executable and / or machine-executable instructions that implement one or more functions and / or operations that together form a program such as the one described herein.

[0125] In another example, the machine-readable instructions may be stored in a state where they can be read by programmable circuitry, but may require the addition of a library (e.g., a dynamic link library (DLL)), a software development kit (SDK), an application programming interface (API), etc., to execute the machine-readable instructions in a particular computing device or other device. In another example, the machine-readable instructions may require configuration (e.g., stored settings, data entry, recorded network addresses, etc.) before the machine-readable instructions and / or the corresponding programs can be executed in whole or in part.In this way, machine-readable, computer-readable and / or machine-readable media as used herein may contain instructions and / or programs, regardless of the particular format or state of the machine-readable instructions and / or programs.

[0126] The machine-readable instructions described here may be represented by any past, present, or future command language, scripting language, programming language, etc. For example, the machine-readable instructions may be represented using any of the following languages: C, C++, Java, C#, Perl, Python, JavaScript, HTML (HyperText Markup Language), SQL (Structured Query Language), Swift, etc.

[0127] As mentioned above, the exemplary processes of Fig. 8, Fig. 9, Fig. 10, Fig. 11, Fig. 12 and / or 13 may be implemented using executable instructions (e.g., computer-readable and / or machine-readable instructions) stored on one or more non-transitory computer-readable and / or machine-readable media. As used herein, the terms "non-transitory computer-readable medium," "non-transitory computer-readable data storage medium," "non-transitory machine-readable medium," and / or "non-transitory machine-readable data storage medium" are expressly defined to include any type of computer-readable data storage device and / or computer-readable data storage drive, and to exclude propagating signals and transmission media.Examples of such a non-transitory computer-readable medium, non-transitory computer-readable data storage medium, non-transitory machine-readable medium, and / or non-transitory machine-readable data storage medium include optical data storage devices, magnetic data storage devices, an HDD, flash memory, read-only memory (ROM), a CD, a DVD, a cache, RAM of any type, a register, and / or any other data storage device or data storage drive in which information is stored for any period of time (e.g., for extended periods of time, permanently, for brief moments, for temporary caching, and / or to cache the information).As used herein, the terms "non-volatile computer-readable data storage devices" and "non-volatile machine-readable data storage devices" are defined to include any physical (mechanical, magnetic, and / or electrical) hardware that retains information for a period of time, but excludes propagating signals and excludes transmission media. Examples of non-volatile computer-readable data storage devices and / or non-volatile machine-readable data storage devices include random-access memory of any type, read-only memory of any type, solid-state memory, flash memory, optical disks, magnetic disks, hard disk drives, and / or RAID (Redundant Array of Independent Disks) systems.As used herein, the term “apparatus” refers to physical structures such as mechanical and / or electrical devices, hardware, and / or circuitry that may or may not be configured by and / or made to execute computer-readable instructions, machine-readable instructions, etc.

[0128] "Include," "contain," "have," and "comprising" (and all forms and tenses thereof) are used herein as open-ended terms. Thus, when any form of "include," "contain," "have," and "comprising" (e.g., includes, contains, has, comprises, comprising, etc.) is used in a claim as a preamble or within any claim language, it is understood that additional elements, terms, etc. may be present without departing from the scope of the corresponding claim or language. As the term "at least" is used as a transitional term herein, for example, in a claim preamble, it is open-ended in the same way that the terms "include," "contain," "have," and "comprising" are open-ended.The term "and / or", when used in a form such as A, B and / or C, refers to any combination or subset of A, B, C, such as (1) A alone, (2) B alone, (3) C alone, (4) A with B, (5) A with C, (6) B with C or (7) A with B and with C. As used in the context of describing structures, components, elements, objects and / or things, the phrase "at least one of A and B" is to be understood to refer to implementations that include each of (1) at least one A, (2) at least one B or (3) at least one A and at least one B.Just as the phrase "at least one of A or B" is used in the context of describing structures, components, elements, objects, and / or things, it is to be understood in a similar manner to refer to implementations that include each of (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. Just as the phrase "at least one of A and B" is used in the context of describing the performance or execution of processes, instructions, actions, activities, etc., it is to be understood to refer to implementations that include each of (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. Just as the phrase "at least one of A or B" is used in the context of describing the performance or execution of processes, instructions, actions, activities, etc.is used, it is to be understood in a similar manner to refer to implementations that include any of (1) at least one A, (2) at least one B, or (3) at least one A and at least one B.

[0129] As used herein, references to a singular (e.g., "a," "an," "first," "second," etc.) do not exclude a plurality. As used herein, the terms "a," "an," or "an" refer to one or more of an item. The terms "a," "an" or "another," "one or more," and "at least one" are used interchangeably. Moreover, although listed individually, a plurality of means, elements, or acts may be implemented by, for example, the same entity or object. Furthermore, although individual features may be included in different examples or claims, they may be combined, and inclusion in different examples or claims does not imply that a combination of features is not practical and / or advantageous.

[0130] As used herein, the connecting references (e.g., attached, coupled, connected, linked, and associated) may include intermediate elements between the elements to which the connecting reference refers, and / or may include relative movement between those elements, unless otherwise indicated. Therefore, the connecting references do not necessarily imply that two elements are directly connected and / or have a fixed relationship.

[0131] Unless specifically stated otherwise, terms such as "first," "second," "third," etc., are used herein without implying or otherwise indicating any significance of priority, physical order, arrangement in a list, and / or ordering, but are used merely as labels and / or arbitrary names to distinguish elements of the disclosed examples for ease of understanding. In some examples, the term "first" may be used to refer to an element in the detailed description, while the same element may be referred to in a claim by a different term such as "second" or "third." In these examples, it is understood that these terms are used merely to identify those elements in the context of discussion (e.g.,within a claim) as different, where the elements might otherwise, for example, share a common name.

[0132] As used herein, the term "in communication," including variations thereof, encompasses direct communication and / or indirect communication through one or more intermediate components and does not require direct physical (e.g., wired) communication, nor does it require constant communication, but instead additionally includes selective communication at regular intervals, scheduled intervals, irregular intervals, and / or upon one-time events.

[0133] As used herein, "programmable circuit" is defined to include: i) one or more dedicated, special-purpose electrical circuits (e.g., an application-specific integrated circuit (ASIC)) structured to perform specific operations and containing one or more semiconductor-based logic devices (e.g., electrical hardware implemented by one or more transistors); and / or ii) one or more general-purpose semiconductor-based electrical circuits programmable with instructions to perform specific functions and / or operations and containing one or more semiconductor-based logic devices (e.g., electrical hardware implemented by one or more transistors). Examples of a programmable circuit include: programmable microprocessors such as central processing units (CPUs),which can execute first instructions to perform one or more operations and / or functions, field-programmable gate arrays (FPGAs) which can be programmed with second instructions to cause a configuration and / or structuring of the FPGAs to instantiate one or more operations and / or functions corresponding to the first instructions, graphics processor units (GPUs) which can execute first instructions to perform one or more operations and / or functions; digital signal processors (DSPs) which can execute first instructions to perform one or more operations and / or functions; XPUs; network processing units (NPUs); one or more microcontrollers which can execute first instructions to perform one or more operations and / or functions,and / or integrated circuits such as application-specific integrated circuits (ASICs). For example, an XPU can be implemented by a heterogeneous computer system that includes multiple types of programmable circuits (e.g., one or more FPGAs, one or more CPUs, one or more GPUs, one or more NPUs, one or more DSPs, etc., and / or a combination thereof) and orchestration technology (e.g., application programming interfaces (APIs)) that can distribute computational tasks to each of the different types of programmable circuits suitable and available for executing the computational tasks.

[0134] As used herein, an integrated circuit is defined as one or more semiconductor packages that may contain one or more circuit elements such as transistors, capacitors, inductors, resistors, power lines, diodes, etc. An integrated circuit may be implemented, for example, as one or more of an ASIC, an FPGA, a chip, a microchip, a programmable circuit, a semiconductor substrate coupling multiple circuit elements, a system-on-chip (SoC), etc.

[0135] Fig. 8 is a flowchart representing example machine-readable instructions and / or example operations 800 that may be executed, instantiated, and / or performed by an example programmable circuit to implement the layered application device 200 of the Fig. 2 for controlling network congestion based on delays in network-internal computation. The example machine-readable instructions and / or the example operations 800 of Fig. 8 begin in block 802, where the application execution circuit 208 generates one or more application data packets based on a workload being executed by the application execution circuit 208.

[0136] In the illustrated example of Fig. 8, in block 804, the application execution circuit 208 assigns a first identifier to one or more application data packets, wherein the first identifier is associated with a class of workload. In block 804, the application execution circuit 208 includes a first identifier in the one or more application data packets. In block 806, the operation scheduling circuit 212 determines one or more operations according to which one or more application data packets are to be processed. In block 808, the operation-to-dataflow mapping circuit 214 converts the one or more application data packets into a first data packet flow based on one or more operations and / or a priority of the class of workload.

[0137] In the illustrated example of Fig. 8, at block 810, the delay determination circuit 222 identifies a second data packet flow based on a second identifier included in the data packets of the second data packet flow, wherein the second identifier is associated with the class of workload (e.g., a matching workload type, a same workload type, etc.). At block 812, the delay determination circuit 222 predicts a delay associated with a transmission of the first data packet flow through a network based on a delay measurement of the second data packet flow. At block 814, the delay determination circuit 222 determines whether the delay meets a delay threshold. Based on (e.g., in response to) the delay determination circuit 222 determining that the delay meets the delay threshold (block 814: YES), the machine-readable instructions and / or operations 800 proceed to block 816.Based on (e.g., in response to) the delay determination circuit 222 determining that the delay does not meet the delay threshold (block 814: NO), the machine-readable instructions and / or operations 800 proceed to block 818.

[0138] In the illustrated example of Fig. 8, the delay determination circuit 222 sets a first ECN flag for data packets of the first data packet flow in block 816. In block 818, a first utilization of a queue of the interface circuit (e.g., the transmitter and / or receiver circuit 232 of the Fig. 2) by the second data packet flow. In block 820, the queue analysis circuit 224 predicts a second queue utilization by the first data packet flow based on the first utilization. In block 822, the queue analysis circuit 224 determines whether the predicted utilization meets a utilization threshold.

[0139] Based on (e.g., in response to) the queue analysis circuit 224 in the illustrated example of Fig. 8 determines that the second load meets the load threshold (block 822: YES), the machine-readable instructions and / or operations 800 proceed to block 824. Based on (e.g., in response to) the queue analysis circuit 224 determining that the second load does not meet the load threshold (block 822: NO), the machine-readable instructions and / or operations 800 proceed to block 826. In block 824, the queue analysis circuit 224 sets a second ECN flag for data packets of the first data packet flow.

[0140] In the illustrated example of Fig. 8, in block 826, the rate calculation circuit 226 determines, based on the delay and the second load, a bit rate that must be used to transmit the first data packet flow to a device in the network. In block 828, the rate adaptation circuit 228 divides the first data packet flow into one or more segments based on a difference between the bit rate and the size of the first data packet flow. In some examples, block 828 may be omitted from the machine-readable instructions and / or operations 800. For example, if the size of the first data packet flow is less than the bit rate, block 828 may be omitted from the machine-readable instructions and / or operations 800.

[0141] In the illustrated example of Fig. 8, in block 830, the rate adaptation circuit 228 causes the interface circuit to transmit a first data packet of the first data packet flow to the device at the bit rate. In block 832, the application execution circuit 208 determines whether an additional application data packet is present. Based on (e.g., in response to) the application execution circuit 208 determining that an additional application data packet is present (block 832: YES), the machine-readable instructions and / or operations 800 return to block 804. Based on (e.g., in response to) the application execution circuit 208 determining that no additional application data packet is present (block 832: NO), the machine-readable instructions and / or operations 800 end.

[0142] Fig. 9 is a flowchart representing example machine-readable instructions and / or example operations 900 that may be executed, instantiated, and / or performed by an example programmable circuit to implement the feedback-capable device 300 of the Fig. 3 for performing network-internal processing. The exemplary machine-readable instructions and / or the exemplary operations 900 of Fig. 9 begin at block 902, where receiver circuitry 302 receives a data packet of a data packet flow from a device, the data packet containing data associated with a workload. At block 904, application processing circuitry 306 analyzes the data packet to determine a workload type of the workload associated with the data packet flow. For example, at block 904, application processing circuitry 306 analyzes the data packet to determine an ID (e.g., a class ID, a workload ID, an application ID, etc.) included in the data packet.

[0143] In the illustrated example of Fig. 9, in block 906, the application processing circuit 306 processes the data contained in the data packet based on the workload type. In block 908, the application processing circuit 306 causes an interface circuit (e.g., configured to cause an interface circuit (e.g., transmitter circuit 314)) to transmit a result of processing the data to the device. In block 910, the receiver circuit 302 determines whether an additional data packet was received. Based on (e.g., in response to) the receiver circuitry 302 determining that an additional data packet was received (block 910: YES), the machine-readable instructions and / or operations 900 return to block 904. Based on (e.g., in response to) the receiver circuitry 302 determining that no additional data packet was received (block 910: NO), the machine-readable instructions and / or operations 900 end.

[0144] Fig. 10 is a flowchart representing example machine-readable instructions and / or example operations 1000 that may be executed, instantiated, and / or performed by an example programmable circuit to implement the feedback-capable device 300 of the Fig. 3 for calculating a bit rate used to transmit a data packet flow based on delays in the network-internal computation. The example machine-readable instructions and / or the example operations 1000 of Fig. 10 begin in block 1002, where the rate calculation circuit 308 determines whether a current utilization of a queue of a first device satisfies (e.g., is greater than or equal to) a utilization threshold for the first device, where the current utilization corresponds to a first data packet flow associated with a workload. In the example of Fig. 10, the utilization threshold is an upper utilization threshold for the first data packet flow, which corresponds to an upper limit for a length of a queue assigned to the first data packet flow.

[0145] Based on (e.g., in response to) the rate calculation circuit 308 in the illustrated example of Fig. 10 determines that the current workload meets the workload threshold (block 1002: YES), the machine-readable instructions and / or operations 1000 proceed to block 1006. Based on (e.g., in response to) the rate calculation circuit 308 determining that the current workload does not meet the workload threshold (block 1002: NO), the machine-readable instructions and / or operations 1000 proceed to block 1004. In block 1004, the rate calculation circuit 308 determines whether a current delay associated with processing data associated with the workload in the first device meets (e.g., is greater than) a delay threshold. For example, the delay threshold is an upper delay threshold, which corresponds to an upper bound (e.g., an acceptable delay limit) for a delay when processing the workload.

[0146] Based on (e.g., in response to) the rate calculation circuit 308 in the illustrated example of Fig. 10 determines that the current delay meets the delay threshold (block 1004: YES), the machine-readable instructions and / or operations 1000 proceed to block 1006. Based on (e.g., in response to) the rate calculation circuit 308 determining that the current delay does not meet the delay threshold (block 1004: NO), the machine-readable instructions and / or operations 1000 proceed to block 1030. In block 1006, the rate calculation circuit 308 determines whether a current bit rate used to transmit the first data packet flow meets (e.g., is greater than) an average of an upper bit rate threshold and a number of second data packet flows to be transmitted by the first device (e.g., RCurr>RUpperThresholdNFlows ).

[0147] In the illustrated example of Fig. 10, the upper bit rate threshold may correspond to a bandwidth of the first device. Based on (e.g., in response to) the rate calculation circuit 308 determining that the current bit rate meets the average (block 1006: YES), the machine-readable instructions and / or operations 1000 proceed to block 1008. Based on (e.g., in response to) the rate calculation circuit 308 determining that the current bit rate does not meet the average (block 1006: NO), the machine-readable instructions and / or operations 1000 proceed to block 1010. In block 1008, the rate calculation circuit 308 sets the current bit rate to a lower bit rate threshold for the first device.

[0148] In the illustrated example of Fig. 10, the rate calculation circuit 308 determines in block 1010 whether a first difference between the current queue load and a previous queue load meets a load increase threshold (e.g., is greater than). In block 1010, the rate calculation circuit 308 determines whether the queue load is increasing too quickly due to the first data packet flow. In the example of Fig. 10, the threshold of the load increase can be any value between zero and the load threshold. In the example of Fig. 10, the previous load also corresponds to a load of the queue determined during a previous iteration of the machine-readable instructions and / or operations 1000.

[0149] Based on (e.g., in response to) the rate calculation 308 in the illustrated example of Fig. 10 determines that the first difference between the current queue utilization and the previous queue utilization meets the utilization growth threshold (block 1010: YES), the machine-readable instructions and / or operations 1000 proceed to block 1014. Based on (e.g., in response to) the rate calculation 308 determining that the first difference between the current queue utilization and the previous queue utilization does not meet the utilization growth threshold (block 1010: NO), the machine-readable instructions and / or operations 1000 proceed to block 1012. In block 1012, the rate calculation circuit 308 determines whether a second difference between the current delay and a previous delay meets a delay growth threshold (e.g.,is greater than), where the previous delay is associated with processing data associated with the workload.

[0150] For example, in block 1012, the rate calculation circuit 308 determines whether the delay associated with processing data associated with the workload is increasing too rapidly. In the example of Fig. 10, the delay increase threshold can be any value between zero and the delay threshold. In the example of Fig. 10, the previous delay also corresponds to a delay associated with processing data associated with the workload determined during a previous iteration of the machine-readable instructions and / or operations 1000.

[0151] Based on (e.g., in response to) the rate calculation circuit 308 in the illustrated example of Fig. 10 determines that the second difference between the current delay and the previous delay meets the delay increment threshold (block 1012: YES), the machine-readable instructions and / or operations 1000 proceed to block 1014. Based on (e.g., in response to) the rate calculation circuit 308 determining that the second difference between the current delay and the previous delay does not meet the delay increment threshold (block 1012: NO), the machine-readable instructions and / or operations 1000 proceed to block 1016. In block 1014, the rate calculation circuit 308 decreases the current bit rate.

[0152] In block 1014, the rate calculation circuit 308 sets the current bit rate, for example, to be the quotient of the previous bit rate divided by a decreasing parameter (e.g., RCurrt=RCurrt−1m ). In the example of the Fig. 10, the previous bit rate corresponds to a bit rate determined during a previous iteration of the machine-readable instructions and / or operations 1000. In the example of Fig. 10, the decreasing parameter is a configurable value. The decreasing parameter is set to two, for example. In block 1016, the rate calculation circuit 308 sets the current bit rate based on the first difference. In block 1016, the rate calculation circuit 308 sets, for example, the bit rate to be equal to the difference between the current bit rate and the product of a) the first difference and b) the sum of a first control parameter and a second control parameter for the first device (e.g., RCurrt=RCurrt−1−[Q(t)−Q(t−1)]∗[a+b] ).

[0153] In the illustrated example of Fig. 10, the rate calculation circuit 308 determines in block 1018 whether the current bit rate meets (e.g., is greater than) the upper bit rate threshold for the first device. Based on (e.g., in response to) the rate calculation circuit 308 determining that the current bit rate meets the upper bit rate threshold (block 1018: YES), the machine-readable instructions and / or operations 1000 proceed to block 1024. Based on (e.g., in response to) the rate calculation circuit 308 determining that the current bit rate does not meet the upper bit rate threshold (block 1018: NO), the machine-readable instructions and / or operations 1000 proceed to block 1020. In block 1020, the rate calculation circuit 308 determines whether the current bit rate meets (e.g., is less than) the lower bit rate threshold for the first device.

[0154] Based on (e.g., in response to) the rate calculation circuit 308 in the illustrated example of Fig. 10 determines that the current bit rate meets the lower bit rate threshold (block 1020: YES), the machine-readable instructions and / or operations 1000 proceed to block 1022. Based on (e.g., in response to) the rate calculation circuit 308 determining that the current bit rate does not meet the lower bit rate threshold (block 1020: NO), the machine-readable instructions and / or operations 1000 proceed to block 1026. In block 1022, the rate calculation circuit 308 sets the current bit rate equal to the value of the lower bit rate threshold.

[0155] Based on (e.g., in response to) the rate calculation circuit 308 in the illustrated example of Fig. 10 determines in block 1024 that the current bit rate meets (e.g., exceeds) the upper bit rate threshold, the rate calculation circuit 308 sets the current bit rate equal to the upper bit rate threshold. In block 1026, the rate calculation circuit 308 sets the current queue utilization of the first device by the first data packet flow equal to the previous queue utilization. In block 1028, the feedback traffic generation circuit 312 causes a transmission of a data packet to a second device to identify the current bit rate. In block 1028, the feedback traffic generation circuit 312 causes a transmission of a CNP.

[0156] In the illustrated example of Fig. 10, the rate calculation circuit 308 determines whether to continue its operation at block 1030. For example, if a period of time after the current bit rate has expired is to be recalculated, or the rate calculation circuit 308 has received a trigger, the rate calculation circuit 308 may determine whether to continue its operation. Based on (e.g., in response to) the rate calculation circuit 308 determining that it should continue its operation (block 1030: YES), the machine-readable instructions and / or the operations 1000 return to block 1002. Based on (e.g., in response to) the rate calculation circuit 308 determining that it should not continue its operation (block 1030: NO), the machine-readable instructions and / or the operations 1000 end.

[0157] Although the machine-readable instructions and / or operations 1000 with respect to the feedback-capable device 300 of the Fig. 3, the machine-readable instructions and / or operations 1000 are similarly applicable to the layered application device 200 of the Fig. 2. In some examples, the rate calculation circuit 226 of the Fig. 2 the machine-readable instructions and / or the work steps 1000 of the Fig. 10. The rate calculation circuit 226 of the Fig. 2 may, for example, implement the machine-readable instructions and / or the operations represented by blocks 1002, 1004, 1006, 1008, 1010, 1012, 1014, 1016, 1018, 1020, 1022, 1024, 1026 and / or 1030 of the Fig. 10 are represented.

[0158] Fig. 11 is a flowchart representing example machine-readable instructions and / or example operations 1100 that may be executed, instantiated, and / or performed by an example programmable circuit to implement the feedback-capable device 300 of the Fig. 3 for calculating control parameters of the feedback-capable device 300 based on delays in the network-internal computation. The example machine-readable instructions and / or the example operations 1100 of Fig. 11 begin in block 1102, where rate calculation circuit 308 initializes a scaling variable. For example, rate calculation circuit 308 initializes the scaling variable with a value of two.

[0159] In the illustrated example of Fig. 11, the rate calculation circuit 308 doubles the current value of the scaling variable in block 1104. In block 1106, the rate calculation circuit 308 determines whether a current bit rate used by a first device to transmit a first data packet flow meets (e.g., is less than) an upper bit rate threshold for the first device. Based on (e.g., in response to) the rate calculation circuit 308 determining that the current bit rate meets the upper bit rate threshold (block 1106: YES), the machine-readable instructions and / or operations 1100 proceed to block 1108. Based on (e.g., in response to) the rate calculation circuit 308 determining that the current bit rate does not meet the upper bit rate threshold (block 1106: NO), the machine-readable instructions and / or operations 1100 proceed to block 1112.

[0160] In the illustrated example of Fig. 11, at block 1108, the rate calculation circuit 308 determines whether a current delay associated with processing data associated with a workload in the first device meets (e.g., is less than) a delay threshold for the first device. Based on (e.g., in response to) the rate calculation circuit 308 determining that the current delay meets the delay threshold (block 1108: YES), the machine-readable instructions and / or operations 1100 proceed to block 1110. Based on (e.g., in response to) the rate calculation circuit 308 determining that the current delay does not meet the delay threshold (block 1108: NO), the machine-readable instructions and / or operations 1100 proceed to block 1112.

[0161] In the illustrated example of Fig. 11, at block 1110, the rate calculation circuit 308 determines whether a current value of the scaling variable satisfies (e.g., is less than) a scaling threshold for the first device. For example, the scaling threshold is equal to 64, which corresponds to 64 bytes. Based on (e.g., in response to) the rate calculation circuit 308 determining that the current value of the scaling variable satisfies the scaling threshold (block 1110: YES), the machine-readable instructions and / or operations 1100 return to block 1104. Based on (e.g., in response to) the rate calculation circuit 308 determining that the current value of the scaling variable does not satisfy the scaling threshold (block 1110: NO), the machine-readable instructions and / or operations 1100 proceed to block 1112.

[0162] In the illustrated example of Fig. 11, the rate calculation circuit 308 sets a factor variable equal to half the current value of the scaling variable in block 1112. In block 1114, the rate calculation circuit 308 determines a first control parameter for the first device based on the factor variable and a first parameter of the first device (e.g., a=a*Factor ). In the example of Fig. 11, the first parameter is a configurable value that varies based on a specification of the first device. For example, if the first device is a 40 gigabits per second (Gbps) network switch, the first parameter may be 0.3. Alternatively, if the first device is a 100 Gbps network switch, the first parameter may be 0.45.

[0163] In the illustrated example of Fig. 11, in block 1116, the rate calculation circuit 308 determines a second control parameter for the first device based on the factor variable and a second parameter of the first device (e.g., b=b*Factor ). In the example of Fig. 11, the second parameter is a configurable value that varies based on the specification of the first device. For example, if the first device is a 40 Gbps network switch, the second parameter may be 1.5. If the first device is a 100 Gbps network switch, the second parameter may be 2.25.

[0164] Although the machine-readable instructions and / or operations 1100 with respect to the feedback-capable device 300 of the Fig. 3, the machine-readable instructions and / or operations 1100 are similarly applicable to the layered application device 200 of the Fig. 2. In some examples, the rate calculation circuit 226 of the Fig. 2 the machine-readable instructions and / or the operations 1100 of the Fig. 11. The rate calculation circuit 226 of the Fig. 2 may, for example, implement the machine-readable instructions and / or the operations represented by blocks 1102, 1104, 1106, 1108, 1110, 1112, 1114 and / or 1116 of the Fig. 11 are represented.

[0165] Fig. 12 is a flowchart representing example machine-readable instructions and / or example operations 1200 that may be executed, instantiated, and / or performed by an example programmable circuit to implement the layered application device 200 of the Fig. 2 for processing a data packet for a congestion notification. The example machine-readable instructions and / or the example operations 1200 of Fig. 12 begin in block 1202, where the rate calculation circuit 226 in a first device (e.g., the layered application device 200) accesses a congestion notification data packet from a second congested device, the congestion notification data packet being associated with a data packet flow.

[0166] In the illustrated example of Fig. 12, at block 1204, the rate calculation circuit 226 determines whether the second congested device corresponds to a currently congested device associated with a rate limiter implemented in the first device, the currently congested device associated with the data packet flow. Based on (e.g., in response to) the rate calculation circuit 226 determining that the second congested device corresponds to the currently congested device (block 1204: YES), the machine-readable instructions and / or operations 1200 proceed to block 1208. Based on (e.g., in response to) the rate calculation circuit 226 determining that the second congested device does not correspond to the currently congested device (block 1204: NO), the machine-readable instructions and / or operations 1200 proceed to block 1206.

[0167] In the illustrated example of Fig. 12, at block 1206, the rate calculation circuit 226 determines whether a received bit rate identified in the congestion notification data packet satisfies (e.g., is less than or equal to) a current bit rate used by the first device to transmit the data packet flow. Based on (e.g., in response to) the rate calculation circuit 226 determining that the received bit rate satisfies the current bit rate (block 1206: YES), the machine-readable instructions and / or operations 1200 proceed to block 1208. Based on (e.g., in response to) the rate calculation circuit 226 determining that the received bit rate does not satisfy the current bit rate (block 1206: NO), the machine-readable instructions and / or operations 1200 proceed to block 1214.

[0168] In the illustrated example of Fig. 12, in block 1208, the rate calculation circuit 226 sets the current bit rate to the received bit rate. In block 1210, the rate calculation circuit 226 sets the currently congested device associated with the data packet flow as the second congested device. In block 1212, the rate calculation circuit 226 stops a bit rate recovery timer (e.g., a counter) for the data packet flow.

[0169] In the illustrated example of Fig. 12, at block 1214, the rate calculation circuit 226 determines whether to continue monitoring for a congestion notification data packet. Based on (e.g., in response to) the rate calculation circuit 226 determining that monitoring for a congestion notification data packet should continue (block 1214: YES), the machine-readable instructions and / or operations 1200 return to block 1202. Based on (e.g., in response to) the rate calculation circuit 226 determining that monitoring for a congestion notification data packet should not continue (block 1214: NO), the machine-readable instructions and / or operations 1200 end.

[0170] Although the machine-readable instructions and / or operations 1200 with respect to the layered application device 200 of the Fig. 2, the machine-readable instructions and / or the operations 1200 are similarly applicable to the feedback-capable device 300 of the Fig. 3. In some examples, the rate calculation circuit 308 of the Fig. 3 the machine-readable instructions and / or the operations 1200 of the Fig. 12. The rate calculation circuit 308 of the Fig. 3 may, for example, implement the machine-readable instructions and / or the operations represented by blocks 1202, 1204, 1206, 1208, 1210, 1212 and / or 1214 of the Fig. 12 are represented.

[0171] Fig. 13 is a flowchart representing example machine-readable instructions and / or example operations 1300 that may be executed, instantiated, and / or performed by an example programmable circuit to implement the layered application device 200 of the Fig. 2 for performing bit rate recovery after network congestion has subsided. The example machine-readable instructions and / or the example operations 1300 of Fig. 13 begin in block 1302, in which the rate calculation circuit 226 initializes a bit rate recovery timer (e.g., a counter) for a data packet flow.

[0172] In the illustrated example of Fig. 13, the rate calculation circuit 226 monitors the bit rate recovery timer at block 1304. The rate calculation circuit 226 determines at block 1306 whether the bit rate recovery timer has expired (e.g., whether the counter has reached zero, whether the counter has reached an upper threshold, etc.). Based on (e.g., in response to) the rate calculation circuit 226 determining that the bit rate recovery timer has expired (block 1306: YES), the machine-readable instructions and / or operations 1300 proceed to block 1308. Based on (e.g., in response to) the rate calculation circuit 226 determining that the bit rate recovery timer has not expired (block 1306: NO), the machine-readable instructions and / or operations 1300 return to block 1304.

[0173] In the illustrated example of Fig. 13, at block 1308, the rate calculation circuit 226 determines whether a current bit rate used by a device (e.g., the layered application device 200) to transmit the data packet flow meets (e.g., exceeds) an upper bit rate threshold for the device. Based on (e.g., in response to) the rate calculation circuit 226 determining that the current bit rate meets the upper bit rate threshold (block 1308: YES), the machine-readable instructions and / or operations 1300 proceed to block 1310. Based on (e.g., in response to) the rate calculation circuit 226 determining that the current bit rate does not meet the upper bit rate threshold (block 1308: NO), the machine-readable instructions and / or operations 1300 proceed to block 1314.

[0174] In the illustrated example of Fig. 13, at block 1310, the rate calculation circuit 226 determines whether a utilization of a queue of the device by the data packet flow meets (e.g., exceeds) a utilization threshold for the device. Based on (e.g., in response to) the rate calculation circuit 226 determining that the utilization meets the utilization threshold (block 1310: YES), the machine-readable instructions and / or operations 1300 proceed to block 1312. Based on (e.g., in response to) the rate calculation circuit 226 determining that the utilization does not meet the utilization threshold (block 1310: NO), the machine-readable instructions and / or operations 1300 proceed to block 1316. At block 1312, the rate calculation circuit 226 removes a rate limited for the data packet flow.

[0175] In the illustrated example of Fig. 13, the rate calculation circuit 226 doubles the current bit rate in block 1314. In block 1316, the rate calculation circuit 226 determines whether to continue its operation. Based on (e.g., in response to) the rate calculation circuit 226 determining that it should continue its operation (block 1316: YES), the machine-readable instructions and / or operations 1300 return to block 1302. Based on (e.g., in response to) the rate calculation circuit 226 determining that it should not continue its operation (block 1316: NO), the machine-readable instructions and / or operations 1300 end.

[0176] Although the machine-readable instructions and / or operations 1300 with respect to the layered application device 200 of the Fig. 2, the machine-readable instructions and / or the operations 1300 are similarly applicable to the feedback-capable device 300 of the Fig. 3. In some examples, the rate calculation circuit 308 of the Fig. 3 the machine-readable instructions and / or the work steps 1300 of the Fig. 13. The rate calculation circuit 308 of the Fig. 3 may, for example, implement the machine-readable instructions and / or the operations represented by blocks 1302, 1304, 1306, 1308, 1310, 1312, 1314 and / or 1316 of the Fig. 13 are represented.

[0177] Fig. 14 is a block diagram of an example programmable circuit platform 1400 structured to execute and / or instantiate the example machine-readable instructions and / or the example operations of the Fig. 8, Fig. 9, Fig. 10, Fig. 11, Fig. 12 and / or 13 to the layered application device 200 of the Fig. 2 and / or the feedback device 300 of the Fig. 3. The programmable circuit platform 1400 may be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network), a mobile device (e.g., a mobile phone, a smartphone, a tablet such as an iPad™), a personal digital assistant (PDA), an internet device, a DVD player, a CD player, a digital video recorder, a Blu-ray player, a game console, a personal video recorder, a set-top box, a headset (e.g., an augmented reality headset (AR headset), a virtual reality headset (VR headset), etc.), or another portable device, or any other type of computing and / or electronic device.

[0178] The programmable circuit platform 1400 of the illustrated example includes a programmable circuit 1412. The programmable circuit 1412 of the illustrated example is hardware. The programmable circuit 1412 may be implemented, for example, by one or more integrated circuits, logic circuits, FPGAs, microprocessors, CPUs, GPUs, DSPs, and / or microcontrollers from any desired product family or manufacturer. The programmable circuit 1412 may be implemented by one or more semiconductor-based (e.g., silicon-based) devices.In this example, programmable circuitry 1412 implements example application execution circuitry 208, example operation scheduling circuitry 212, example operation-to-flow mapping circuitry 214, example flow-to-operation mapping circuitry 216, example delay determination circuitry 222, example queue analysis circuitry 224, example rate calculation circuitry 226, example rate adaptation circuitry 228, and / or more generally example congestion control circuitry 218, example application processing circuitry 306, example rate calculation circuitry 308, and example feedback traffic generation circuitry 312.

[0179] The programmable circuit 1412 of the illustrated example includes a local memory 1413 (e.g., a cache, a register, etc.). The programmable circuit 1412 of the illustrated example is in communication with a main memory 1414, 1416, which includes a volatile memory 1414 and a non-volatile memory 1416, through a bus 1418. The volatile memory 1414 may be implemented by a synchronous dynamic random access memory (SDRAM), a dynamic random access memory (DRAM), a RAMBUS® dynamic random access memory (RDRAM®), and / or another type of RAM device.In this example, volatile memory 1414 implements example command and completion queues 210, example NIC queues 230, example data packet flow queues 304, and example data packet flow metadata store 310. Non-volatile memory 1416 may be implemented by flash memory and / or another desired type of memory device. Access to main memory 1414, 1416 of the illustrated example is controlled by a memory controller 1417. In some examples, memory controller 1417 may be implemented by one or more integrated circuits, logic circuits, microcontrollers from a desired product family or manufacturer, or another type of circuitry to manage the flow of data to and from main memory 1414, 1416.

[0180] The programmable circuit platform 1400 of the illustrated example also includes an interface circuit 1420. The interface circuit 1420 may be implemented by hardware according to any type of interface standard, such as an Ethernet interface, a Universal Serial Bus (USB) interface, a Bluetooth® interface, a Near Field Communication (NFC) interface, a Peripheral Component Interconnect (PCI) interface, and / or a Peripheral Component Interconnect Express (PCIe) interface.

[0181] In the illustrated example, one or more input devices 1422 are connected to the interface circuit 1420. The input devices 1422 allow a user (e.g., a human user, a machine user, etc.) to input data and / or commands into the programmable circuit 1412. The input devices 1422 may be implemented, for example, by an audio sensor, a microphone, a camera (still or video camera), a keyboard, a button, a mouse, a touchscreen, a trackpad, a trackball, an isopoint device, and / or a speech recognition system.

[0182] One or more output devices 1424 are also connected to the interface circuitry 1420 of the illustrated example. The output devices 1424 may be implemented, for example, by display devices (e.g., a light-emitting diode (LED), an organic light-emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube (CRT), an in-plane switching display (IPS display), a touchscreen, etc.), a haptic output device, a printer, and / or speakers. The interface circuitry 1420 of the illustrated example thus typically includes a graphics driver card, a graphics driver chip, and / or a graphics processor circuitry such as a GPU.

[0183] The interface circuitry 1420 of the illustrated example also includes a communication device, such as a transmitter, a receiver, a transceiver, a modem, a home gateway, a wireless access point, and / or a network interface, to facilitate data exchange with external machines (e.g., computing devices of any type) over a network 1426. Communication may occur, for example, through an Ethernet connection, a DSL (Digital Subscriber Line) connection, a telephone line connection, a coaxial cable system, a satellite system, a beyond-line-of-sight wireless system, a cellular system, an optical connection, etc. In this example, the interface circuitry 1420 implements the example transmitter and / or receiver circuitry 232, the example receiver circuitry 302, and the example transmitter circuitry 314.

[0184] The programmable circuit platform 1400 of the illustrated example also includes one or more mass storage drives or devices 1428 for storing firmware, software, and / or data. Examples of such mass storage drives or devices 1428 include magnetic data storage devices (e.g., floppy disks, drives, HDDs, etc.), optical data storage devices (e.g., Blu-ray disks, CDs, DVDs, etc.), RAID systems, and / or solid-state storage drives or devices such as flash memory devices and / or SSDs.

[0185] The machine-readable instructions 1432, which are replaced by the machine-readable instructions of the Fig. 8, Fig. 9, Fig. 10, Fig. 11, Fig. 12, and / or 13 may be stored in the mass storage device 1428, in the volatile memory 1414, in the non-volatile memory 1416, and / or in at least one non-volatile, computer-readable data storage medium, such as a CD or DVD, which may be removable.

[0186] Fig. 15 is a block diagram of an exemplary implementation of the programmable circuit 1412 of the Fig. 14. In this example, the programmable circuit 1412 of the Fig. 14 is implemented by a microprocessor 1500. The microprocessor 1500 may be, for example, a general-purpose microprocessor (e.g., a general-purpose microprocessor circuit). The microprocessor 1500 executes some or all of the machine-readable instructions from the flowcharts of the Fig. 8, Fig. 9, Fig. 10, Fig. 11, Fig. 12 and / or 13 to switch the circuit of the Fig. 2 and / or 3 as logic circuits for performing operations corresponding to those machine-readable instructions. In some such examples, the circuitry of the Fig. 2 and / or the switching of the Fig. 3 instantiated by the hardware circuitry of the microprocessor 1500 in combination with the machine-readable instructions. The microprocessor 1500 may be implemented, for example, by a multi-core hardware circuit such as a CPU, a DSP, a GPU, an XPU, etc. Although the microprocessor 1500 may include any number of exemplary cores 1502 (e.g., 1 core), in this example, the microprocessor 1500 is a multi-core semiconductor device with N cores. The cores 1502 of the microprocessor 1500 may function independently of one another, or they may cooperate to execute machine-readable instructions. For example, machine code corresponding to a firmware program, an embedded software program, or a software program may be executed by one of the cores 1502, or it may be executed by multiple cores 1502 simultaneously or at different times.In some examples, the machine code corresponding to the firmware program, the embedded software program, or the software program is divided into threads and executed in parallel by two or more of the cores 1502. The software program may correspond to some or all of the machine-readable instructions and / or operations represented by the flowcharts of the . Fig. 8, Fig. 9, Fig. 10, Fig. 11, Fig. 12 and / or 13 are represented.

[0187] The cores 1502 may communicate via a first example bus 1504. In some examples, the first bus 1504 may be implemented by a communication bus to realize communication associated with some of the cores 1502. The first bus 1504 may be implemented, for example, by at least one of an I2C (Inter-Integrated Circuit) bus, a SPI (Serial Peripheral Interface) bus, a PCI bus, or a PCIe bus. Additionally or alternatively, the first bus 1504 may be implemented by any other type of computer bus or electrical bus. The cores 1502 may receive data, instructions, and / or signals from one or more external devices, for example, using an example interface circuit 1506. The cores 1502 may output data, instructions, and / or signals to one or more external devices via the interface circuit 1506.Although in this example, the cores 1502 include an exemplary local memory 1520 (e.g., a Level 1 (L1) cache that can be divided into an L1 data cache and an L1 instruction cache), the microprocessor 1500 also includes an exemplary shared memory 1510 that can be shared between the cores (e.g., a Level 2 (L2) cache) for high-speed access to data and / or instructions. The data and / or instructions may be transferred (e.g., shared) by writing to and / or reading from the shared memory 1510. The local memory 1520 of each of the cores 1502 and the shared memory 1510 may be part of a hierarchy of data storage devices that include multiple levels of cache memory and main memory (e.g., the main memory 1414, 1416 of the . Fig. 14). Typically, higher memory levels in the hierarchy have lower access times and smaller storage capacities than lower memory levels. Changes at the different levels of the cache hierarchy are managed (e.g., coordinated) by a cache coherence policy.

[0188] Each core 1502 may be referred to as a CPU, DSP, GPU, etc., or any other type of hardware circuit. Each core 1502 includes a controller circuit 1514, an arithmetic and logic (AL) circuit (sometimes referred to as an ALU) 1516, a plurality of registers 1518, the local memory 1520, and a second example bus 1522. Other structures may also be present. For example, each core 1502 may include a vector unit circuit, a single-instruction multiple data unit (SIMD) circuit, a load / store unit (LSU) circuit, a branch / jump unit circuit, a floating-point unit (FPU) circuit, etc.Controller circuitry 1514 includes semiconductor-based circuitry structured to control (e.g., coordinate) data movement within the corresponding core 1502. AL circuitry 1516 includes semiconductor-based circuitry structured to perform one or more mathematical and / or logical operations on the data within the corresponding core 1502. AL circuitry 1516 of some examples performs integer operations. In other examples, AL circuitry 1516 also performs floating-point operations. In further examples, AL circuitry 1516 may include a first AL circuitry that performs integer operations and a second AL circuitry that performs floating-point operations. In some examples, AL circuitry 1516 may be referred to as an arithmetic logic unit (ALU).

[0189] Registers 1518 are semiconductor-based structures for storing data and / or instructions, such as results of one or more of the operations performed by the AL circuitry 1516 of the corresponding core 1502. Registers 1518 may include, for example, vector registers, SIMD registers, general-purpose registers, flag registers, segment registers, machine-specific registers, instruction pointer registers, control registers, debug registers, memory management registers, machine check registers, etc. Registers 1518 may be arranged in a bank, as shown in Fig. 15. Alternatively, registers 1518 may be organized in any other arrangement, format, or structure, such as distributing them throughout core 1502 to reduce access time. Second bus 1522 may be implemented by at least one of an I2C bus, an SPI bus, a PCI bus, or a PCIe bus.

[0190] Each core 1502 and / or, more generally, the microprocessor 1500 may include structures additional and / or alternative to those shown and described above. For example, one or more clock circuits, one or more power supplies, one or more power gates, one or more cache home agents (CHAs), one or more converged / common mesh stops (CMSs), one or more shifters (e.g., a barrel shifter), and / or other circuitry may be present. The microprocessor 1500 is a semiconductor device constructed to include many interconnected transistors to implement the structures described above in one or more integrated circuits (ICs) contained in one or more packages.

[0191] Microprocessor 1500 may include and / or cooperate with one or more accelerators (e.g., an accelerator circuit, a hardware accelerator, etc.). In some examples, the accelerators are implemented by logic circuitry to perform certain tasks faster and / or more efficiently than is possible with a general-purpose processor. Examples of accelerators include ASICs and FPGAs, such as those discussed herein. A GPU, a DSP, and / or other programmable device may also be accelerators. Accelerators may be included in microprocessor 1500, in the same die package as microprocessor 1500, and / or in one or more packages separate from microprocessor 1500.

[0192] Fig. 16 is a block diagram of another exemplary implementation of the programmable circuit 1412 of the Fig. 14. In this example, the programmable circuit 1412 is implemented by the FPGA circuit 1600. The FPGA circuit 1600 may, for example, be implemented by an FPGA. The FPGA circuit 1600 may, for example, be used to perform operations that would otherwise be performed by the exemplary microprocessor 1500 of the Fig. 15, which executes corresponding machine-readable instructions. However, after the FPGA circuit 1600 is configured, it instantiates the operations and / or functions corresponding to the machine-readable instructions in hardware and can thus often perform the operations / functions faster than they could be performed by a general-purpose microprocessor executing the corresponding software.

[0193] In particular, in contrast to the microprocessor 1500 described above, Fig. 15 (which is a general-purpose device that can be programmed to execute any or all of the machine-readable instructions represented by the flowcharts of the Fig. 8, Fig. 9, Fig. 10, Fig. 11, Fig. 12 and / or 13, but whose connections and logic circuitry are determined when they are manufactured), the FPGA circuit 1600 of the example of the Fig. 16 connections and a logic circuit that can be configured, structured, programmed and / or interconnected in different ways after manufacture, for example to instantiate some or all of the operations / functions corresponding to the machine-readable instructions represented by the flowcharts of the Fig. 8, Fig. 9, Fig. 10, Fig. 11, Fig. 12 and / or 13. In particular, the FPGA circuit 1600 can be thought of as an array of logic gates, interconnects, and switches. The switches can be programmed to change how the logic gates are connected to one another by the interconnects, thereby forming one or more dedicated logic circuits (except when and until the FPGA circuit 1600 is reprogrammed). The configured logic circuits enable the logic gates to cooperate in different ways to perform different operations on data received from an input circuit. These operations may correspond to some or all of the instructions (e.g., software and / or firmware) represented by the flowcharts of the Fig. 8, Fig. 9, Fig. 10, Fig. 11, Fig. 12 and / or 13. Therefore, the FPGA circuit 1600 may be configured and / or structured to perform some or all of the operations / functions corresponding to the machine-readable instructions of the flowchart of the Fig. 8, Fig. 9, Fig. 10, Fig. 11, Fig. 12 and / or 13, effectively instantiating them as dedicated logic circuits to perform the operations / functions corresponding to these software instructions in a dedicated manner analogous to an ASIC. Therefore, the FPGA circuit 1600 may perform the operations / functions corresponding to some or all of the machine-readable instructions of the Fig. 8, Fig. 9, Fig. 10, Fig. 11, Fig. 12 and / or 13 faster than the general purpose microprocessor can perform the same operations / functions.

[0194] In the example of Fig. 16, the FPGA circuit 1600 is configured and / or structured in response to being programmed (and / or reprogrammed one or more times) based on a binary file. In some examples, the binary file may be compiled and / or generated based on instructions in a Hardware Description Language (HDL) such as Lucid, a Very High Speed ​​Integrated Circuits (VHSIC-VHDL), or Verilog. For example, a user (e.g., a human user, a machine user, etc.) may write code or a program corresponding to one or more operations / functions in an HDL; where the code / program may be translated into a low-level language if necessary; and where the code / program (e.g., the code / program in the low-level language) (e.g.,by a compiler, a software application, etc.) into a binary file. In some examples, the FPGA circuit 1600 of the . Fig. 16 access and / or load the binary file to cause the FPGA circuit 1600 of the Fig. 16 is configured and / or structured to perform the one or more operations / functions. The binary file may be implemented, for example, by a bit stream (e.g., one or more computer-readable bits, one or more machine-readable bits, etc.), data (e.g., computer-readable data, machine-readable data, etc.), and / or machine-readable instructions available to the FPGA circuit 1600 of the Fig. 16 are accessible to configure and / or structure the FPGA circuit 1600 of the Fig. 16 or parts thereof.

[0195] In some examples, the binary file is compiled, generated, transformed, and / or otherwise output by a unified software platform used to program FPGAs. For example, the unified software platform may translate first instructions (e.g., a code or a program) corresponding to one or more operations / functions in a high-level language (e.g., C, C++, Python, etc.) into second instructions corresponding to the one or more operations / functions in an HDL. In some of these examples, the binary file is compiled, generated, and / or otherwise output by the unified software platform based on the second instructions. In some examples, the FPGA circuit 1600 of the Fig. 16 access and / or load the binary file to cause the FPGA circuit 1600 of the Fig. 16 is configured and / or structured to perform the one or more operations / functions. The binary file may be implemented, for example, by a bit stream (e.g., one or more computer-readable bits, one or more machine-readable bits, etc.), data (e.g., computer-readable data, machine-readable data, etc.), and / or machine-readable instructions available to the FPGA circuit 1600 of the Fig. 16 are accessible to configure and / or structure the FPGA circuit 1600 of the Fig. 16 or parts thereof.

[0196] The FPGA circuit 1600 of the Fig. 16 includes an example input / output (I / O) circuit 1602 for receiving and / or outputting data to or from an example configuration circuit 1604 and / or external hardware 1606. The configuration circuit 1604 may be implemented, for example, by an interface circuit that can receive a binary file that can be implemented by a bit stream, data, and / or machine-readable instructions to configure the FPGA circuit 1600 or portions thereof. In some such examples, the configuration circuit 1604 may receive the binary file from a user, a machine (e.g., a hardware circuit (e.g., a programmable or dedicated circuit) that can implement an artificial intelligence or machine learning (AI / ML) model to generate the binary file), etc., and / or any combination thereof.In some examples, external hardware 1606 may be implemented by external hardware circuitry. For example, external hardware 1606 may be implemented by microprocessor 1500 of the . Fig. 15 will be implemented.

[0197] The FPGA circuit 1600 also includes an array of an example logic gate circuit 1608, a plurality of example configurable connections 1610, and an example data storage circuit 1612. The logic gate circuit 1608 and the configurable connections 1610 are configurable to instantiate one or more operations / functions corresponding to at least some of the machine-readable instructions of the Fig. 8, Fig. 9, Fig. 10, Fig. 11, Fig. 12 and / or 13 and / or other desired operations. The logic gate circuit 1608 shown in Fig. 16 is fabricated in blocks or groups. Each block contains semiconductor-based electrical structures that can be configured as logic circuits. In some examples, the electrical structures include logic gates (e.g., AND gates, OR gates, NOR gates, etc.) that provide basic building blocks for the logic circuits. Electrically controllable switches (e.g., transistors) are located within each logic gate circuit 1608 to enable the configuration of the electrical structures and / or the logic gates to form circuits to perform the desired operations / functions. The logic gate circuit 1608 may include other electrical structures such as lookup tables (LUTs), registers (e.g., flip-flops or latches), multiplexers, etc.

[0198] The configurable connections 1610 of the illustrated example are conductive paths, traces, vias, or the like that may include electrically controllable switches (e.g., transistors) whose state may be changed by programming (e.g., using an HDL instruction language) to enable or disable one or more connections between one or more of the logic gate circuits 1608 to program the desired logic circuits.

[0199] The data storage circuit 1612 of the illustrated example is structured to store the results of the one or more operations performed by the corresponding logic gates. The storage circuit 1612 may be implemented by registers or the like. In the illustrated example, the storage circuit 1612 is distributed among the logic gate circuits 1608 to facilitate access and increase execution speed.

[0200] The exemplary FPGA circuit 1600 of the Fig.16 also includes an example dedicated operations circuit 1614. In this example, the dedicated operations circuit 1614 includes dedicated circuitry 1616 that can be called to implement frequently used functions to avoid the need to program those functions in-place. Examples of this dedicated circuitry 1616 include memory control circuitry (e.g., DRAM control circuitry), PCIe control circuitry, clock circuitry, transceiver circuitry, memory, and multiplier-accumulator circuitry. Other types of dedicated circuitry may also be present. In some examples, the FPGA circuitry 1600 may also include an example general-purpose programmable circuitry 1618, such as an example CPU 1620 and / or an example DSP 1622.Other programmable general-purpose circuitry 1618 may be additionally or alternatively present, such as a GPU, an XPU, etc., which may be programmed to perform other operations.

[0201] Although the Fig. 15 and Fig. 16 two exemplary implementations of the programmable circuit 1412 of the Fig. 14, many other approaches are considered. For example, an FPGA circuit may include an integrated CPU, such as one or more of the exemplary CPUs 1620 of Fig. 16. Therefore, the programmable circuit 1412 of the Fig. 14 may be additionally implemented by at least the exemplary microprocessor 1500 of the Fig. 15 and the exemplary FPGA circuit 1600 of the Fig. 16. In some such hybrid examples, one or more cores 1502 of the Fig. 15 execute a first part of the machine-readable instructions represented by the flowcharts of the Fig. 8, Fig. 9, Fig. 10, Fig. 11, Fig. 12 and / or 13 to perform first operations / functions, wherein the FPGA circuit 1600 of the Fig. 16 can be configured and / or structured to perform second operations / functions corresponding to a second part of the machine-readable instructions represented by the flowcharts of the Fig. 8, Fig. 9, Fig. 10, Fig. 11, Fig. 12 and / or 13, and / or wherein an ASIC can be configured and / or structured to perform third operations / functions corresponding to a third part of the machine-readable instructions represented by the flowcharts of the Fig. 8, Fig. 9, Fig. 10, Fig. 11, Fig. 12 and / or 13 are represented.

[0202] It is self-evident that part of the entire circuit of the Fig. 2 and Fig. 3 can be instantiated in this way simultaneously or at different times. For example, the same and / or different parts of the microprocessor 1500 of the Fig. 15 to execute portions of the machine-readable instructions simultaneously and / or at different times. In some examples, the same and / or different portions of the FPGA circuit 1600 of the Fig. 16 be configured and / or structured to perform operations / functions corresponding to parts of the machine-readable instructions simultaneously and / or at different times.

[0203] In some examples, part or all of the circuits of the Fig. 2 and the Fig. 3, for example, can be instantiated in one or more threads that execute concurrently and / or in series. The microprocessor 1500 of the Fig. 15 may, for example, execute machine-readable instructions in one or more threads that execute concurrently and / or sequentially. In some examples, the FPGA circuit 1600 of the Fig. 16 may be configured and / or structured to perform operations / functions simultaneously and / or sequentially. In some examples, furthermore, part or all of the circuitry of the Fig. 2 and the Fig. 3 be implemented within one or more virtual machines and / or containers located in the microprocessor 1500 of the Fig. 15 are executed.

[0204] In some examples, the programmable circuit 1412 of the Fig. 14 may be contained in one or more housings. For example, the microprocessor 1500 of the Fig. 15 and the FPGA circuit 1600 of the Fig. 16 in one or more packages. In some examples, an XPU may be programmable by the programmable circuit 1412 in Fig. 14, which can be located in one or more housings. The XPU can, for example, be a CPU (e.g., the microprocessor 1500 of the Fig. 15, the CPU 1620 of the Fig. 16 etc.) in one housing, a DSP (e.g. the DSP 1622 of the Fig. 16) in another package, a GPU in another package and an FPGA (e.g. the FPGA circuit 1600 of the Fig. 16) are contained in another data packet.

[0205] A block diagram illustrating an exemplary software distribution platform 1705 for distributing software, such as the machine-readable instructions 1432 of the Fig. 14 to other hardware devices (e.g. hardware devices owned or operated by third parties, the owner and / or operator of the software distribution platform) is Fig. 17. The exemplary software distribution platform 1705 may be implemented by any computer server, data facility, cloud service, etc. capable of storing and transmitting software to other computing devices. The third parties may be customers of the entity that owns and / or operates the software distribution platform 1705. For example, the entity that owns and / or operates the software distribution platform 1705 may be a developer, a vendor, and / or a licensor of software such as the exemplary machine-readable instructions 1432 of Fig. 14. The third parties may be consumers, users, retailers, OEMs, etc., who purchase and / or license and / or resell and / or sublicense the software for use. In the illustrated example, the software distribution platform 1705 includes one or more servers and one or more data storage devices. Stored in the data storage devices are the machine-readable instructions 1432, which correspond to the example machine-readable instructions in Fig. 8, Fig. 9, Fig. 10, Fig. 11, Fig. 12 and / or 13, as described above. One or more servers of the example software distribution platform 1705 are in communication with an example network 1710, which may correspond to one or more of the Internet and / or one of the example networks described above. In some examples, the one or more servers respond to requests to transfer the software to a requesting party as part of a commercial transaction. Payment for the delivery, sale, and / or license of the software may be handled by the one or more servers of the software distribution platform and / or by a third-party payment entity. The servers enable purchasers and / or licensors to download the machine-readable instructions 1432 from the software distribution platform 1705. For example, the software corresponding to the example machine-readable instructions of the Fig. 8, Fig. 9, Fig. 10, Fig. 11, Fig. 12 and / or 13, may be downloaded to the exemplary programmable circuit platform 1400, which executes the machine-readable instructions 1432 to implement the layered application device 200 of the Fig. 2 and / or the feedback device 300 of the Fig. 3. In some examples, one or more servers of the software distribution platform 1705 regularly offer, transmit, and / or enforce updates (e.g., the machine-readable instructions 1432 of the Fig. 14) to ensure that improvements, patches, updates, etc., are distributed and applied to the software on end-user devices. Although the "distributed software" is referred to above as software, it may alternatively be firmware.

[0206] Based on the foregoing, it should be understood that exemplary systems, devices, articles of manufacture, and methods for in-network computing and control have been disclosed, reducing network congestion based on delays in the in-network computing. For example, the disclosed systems, methods, devices, and articles of manufacture provide transport layer protocols to enable scalable in-network computing (INC) with a variety of hardware, including IPUs, DPUs, EPUs, FPGAs, ASICs, CPUs and / or XPUs of end-host devices, programmable switches, and middle boxes. The examples disclosed herein also improve a network's multiplexing gain, improve the fairness of data packet flows, increase link utilization, and improve multi-response handling.Therefore, the examples disclosed here reduce flow complete time (FCT) reduction, improve reliability, and enhance stability through rapid convergence of data packet flows.

[0207] Furthermore, the disclosed systems, devices, articles of manufacture, and methods improve the efficiency of using a computing device by increasing network throughput through dynamic bit rate control in EH devices and / or IN devices. For example, computation in intermediate network devices contributes to overall end-to-end delay. By compensating for delays introduced by in-network computing, the examples disclosed herein prevent premature triggering of congestion control techniques. Furthermore, the disclosed methods, devices, and articles of manufacture include application-specific context information in the data packet headers (e.g., indicating whether in-network computing with QoS parameters should be performed). The examples disclosed herein also generate accurate feedback (e.g.,Rate, queue size, delay) at congestion points in a network to assist EH devices in accurately determining potential causes of congestion and taking appropriate action to mitigate congestion. The disclosed systems, devices, articles of manufacture, and methods are accordingly directed to one or more improvements in the operation of a machine, such as a computer or other electronic and / or mechanical device.

[0208] Disclosed herein are exemplary methods, apparatus, systems, and articles of manufacture for in-network computation and network congestion control based on in-network computation delays. Further examples and combinations thereof include the following:

[0209] Example 1 includes a device deployed in a network, the device including: interface circuitry to access a data packet containing a header comprising: (1) a destination address field for identifying a first network address of a destination device capable of performing an action, and (2) a workload class field for identifying a workload class associated with the action; machine-readable instructions; and programmable circuitry for using the machine-readable instructions to perform the action in the device based on the workload class field;wherein the device has a second network address different from the first network address for modifying an indicator field of the data packet to indicate that the action has been performed and for causing the interface circuit to forward the data packet with the modified indicator field to the destination device;

[0210] Example 2 includes the apparatus of Example 1, wherein the action includes an in-network computation primitive, the in-network computation primitive including at least one of a vector summation operation, a counting operation, a caching operation, a storing operation, an obtaining operation, or a hash table operation.

[0211] Example 3 includes the apparatus of any one of examples 1 or 2, wherein the data packet is a first data packet, and wherein the programmable circuit is configured to cause the interface circuit to transmit a second data packet to a source device of the data packet, the second data packet identifying at least one capability of an in-network computation primitive of the programmable circuit.

[0212] Example 4 includes the apparatus of any one of examples 1, 2, or 3, wherein the data packet is to be transmitted along a communication path from a source device to the destination device, and wherein the programmable circuit is configured to determine a congestion level in another network device in the communication path and to determine that the action is performed in the device based on the congestion level.

[0213] Example 5 includes the apparatus of Example 4, wherein the programmable circuit is configured to determine the congestion level based on at least one of a congestion notification field, a delay level field, a queue state field, or a fair rate field in the header of the data packet.

[0214] Example 6 includes the device of any one of examples 4 or 5, wherein the programmable circuit is configured to perform the action in the device during a delay introduced by the congestion level.

[0215] Example 7 includes the apparatus of any of Examples 1, 2, 3, 4, 5, or 6, wherein the data packet is a first data packet, the action is a first action, and the first data packet is transmitted along a communication path from a source device to the destination device, and wherein the programmable circuit is configured to determine a level of congestion in a subsequent device in the communication path based on a feedback data packet from the subsequent device, the feedback data packet including at least one of a bit rate used by the subsequent device to transmit a second data packet, a utilization of a queue in the subsequent device by the second data packet, or a processing delay associated with performing a second action associated with the second data packet in the subsequent device rather than the destination device.and to determine that the action is performed in the device based on the overload level in the subsequent device.

[0216] Example 8 includes the apparatus of Example 7, wherein the programmable circuit is configured to perform the action in the apparatus during a delay introduced by the congestion level in the subsequent apparatus.

[0217] Example 9 includes the apparatus of any of examples 1, 2, 3, 4, 5, 6, 7, or 8, wherein the action is to be performed based on a payload of the data packet, the payload being associated with an application, and the programmable circuitry configured to modify the payload before forwarding the data packet to the destination device.

[0218] Example 10 includes the apparatus of any one of examples 1, 2, 3, 4, 5, 6, 7, 8, or 9, wherein the data packet is a first data packet, wherein the programmable circuit is configured to determine a bit rate used to transmit the first data packet to the destination device, the bit rate based on a delay associated with performing the action at the device, and wherein the interface circuit is configured to transmit a second data packet to identify the bit rate to at least one of a source device of the data packet or the destination device.

[0219] Example 11 includes a source device comprising machine-readable instructions and programmable circuitry to use the machine-readable instructions to generate a data packet including a header comprising: (1) a destination address field for identifying a network address of a destination device; and (2) a workload class field for identifying a workload class associated with an action suitable to be performed by the destination device, wherein the workload class is accessible by an intermediate device along a communication path between the source device and the destination device, wherein the intermediate device is configured to perform the action in place of the destination device, and wherein the interface circuitry is configured to transmit the data packet to the intermediate device.

[0220] Example 12 includes the source device of Example 11, wherein the data packet is a first data packet, and wherein the programmable circuit is configured to, based on a second data packet from the intermediate device, determine at least one in-network computation primitive that may be performed by the intermediate device, and to select the communication path of the data packet based on at least one in-network computation primitive and the workload class.

[0221] Example 13 includes the source device of Example 12, wherein at least one in-network computation primitive is identified in the second data packet.

[0222] Example 14 includes the source device of any one of examples 12 or 13, wherein the at least one in-network computation primitive includes at least one of a vector summation operation, a counting operation, a caching operation, a storing operation, a fetching operation, or a hash table operation.

[0223] Example 15 includes the source device of any one of examples 11, 12, 13, or 14, wherein the programmable circuit is configured to populate at least one of a congestion notification field, a delay level field, a queue state field, or a fair rate field in the header of the data packet before transmitting the data packet to the intermediate device.

[0224] Example 16 includes the source device of any one of Examples 11, 12, 13, 14, or 15, wherein the data packet is a first data packet, the action is a first action, and the programmable circuit is configured to access a feedback data packet from the intermediate device, the feedback data packet including at least one of a first bit rate used by the intermediate device to transmit a second data packet, a load on a queue in the intermediate device by the second data packet, or a processing delay associated with performing a second action associated with the second data packet in the intermediate device rather than the destination device; and to cause the interface circuit to transmit the first data packet to the intermediate device at a second bit rate based on the feedback data packet.

[0225] Example 17 includes the source device of any one of examples 11, 12, 13, 14, 15, or 16, wherein the data packet is a first data packet, wherein the interface circuit is associated with a queue, and wherein the programmable circuit is configured to cause the interface circuit to transmit the first data packet to the intermediate device at a bit rate based on a predicted delay and a utilization of the queue, wherein the predicted delay is based on a delay measurement of a second data packet.

[0226] Example 18 includes the source device of any of Examples 11, 12, 13, 14, 15, 16, or 17, wherein the header includes the workload class field and one or more second fields that are different from the workload class field, the one or more second fields including one or more of a version field, an internet header length field, a data packet length field, a data packet identifier field, a data packet fragmentation flag field, a data packet fragmentation offset field, a validity period field, a header checksum field, a source address field, a destination address field, a timestamp field, a retransmission field, or an acknowledgment field.

[0227] Example 19 includes an apparatus configured to be deployed in a network, the apparatus including machine-readable instructions and programmable circuitry to use the machine-readable instructions to analyze a first data packet from a source device, the first data packet addressed to a destination device in the network, the first data packet including a payload and a header, the header including a workload class field and one or more second fields different from the workload class field, the workload class field configured to indicate a workload class configured to be used to process the payload of the first data packet;wherein the one or more second fields include one or more of a version field, an internet header length field, a packet length field, a packet identifier field, a packet fragmentation flag field, a packet fragmentation offset field, a validity period field, a header checksum field, a source address field, a destination address field, a timestamp field, a retransmission field, or an acknowledgment field; and to, before the first data packet reaches the destination device, perform an operation on the payload that is not associated with routing the first data packet, the operation for producing an output, and an interface circuit for forwarding a second data packet corresponding to the output to the destination device.

[0228] Example 20 includes the apparatus of Example 19, wherein the programmable circuit is configured to determine a bit rate used to transmit the first data packet to the destination device, the bit rate based on a delay associated with performing the operation at the device, and wherein the interface circuit is configured to transmit a third data packet to identify the bit rate to at least one of the source device of the first data packet or the destination device.

[0229] Example 21 includes the apparatus of any one of examples 19 or 20, wherein the workload class field is located between the internet header length field and the data packet length field.

[0230] Example 22 includes an in-network device configured to be deployed in a network, wherein the in-network device comprises machine-readable instructions and programmable circuitry to use the machine-readable instructions to analyze a data packet to identify a header field of the data packet, the data packet originating from a first device in the network, and to process a payload included in the data packet in the in-network device based on the header field; and wherein interface circuitry is configured to transmit a result of processing the payload to at least one of the first and a second device.

[0231] Example 23 includes the in-network device of Example 22, wherein the data packet is a first data packet, wherein the programmable circuit is configured to determine a bit rate to be used to transmit a second data packet containing the result, wherein the bit rate is based on a delay associated with processing the payload in the in-network device, and wherein the interface circuit is configured to transmit a third data packet to identify the bit rate for at least one of the first device or the second devices.

[0232] Example 24 includes the in-network device of any one of examples 22 or 23, wherein the programmable circuit is configured to determine whether a delay associated with processing the payload in the in-network device satisfies a delay threshold.

[0233] Example 25 includes the in-network device of Example 24, wherein the data packet is a first data packet, and wherein the programmable circuit is configured to determine, based on the delay satisfying the delay threshold, whether a bit rate currently used to transmit a second data packet containing the result satisfies an average of an upper bit rate threshold for the in-network device and a number of third data packets to be transmitted by the in-network device, wherein the first data packet is associated with a first workload, and wherein the third data packets are associated with corresponding second workloads.

[0234] Example 26 includes the in-network device of Example 25, wherein the delay is a first delay, the bit rate is a first bit rate, the delay threshold is a first delay threshold, and wherein the programmable circuit is configured to, based on the first bit rate not satisfying the average, set a second bit rate to be used to transmit the second data packet based on whether a difference between the first delay and a second delay satisfies a second delay threshold.

[0235] Example 27 includes the in-network device of any one of examples 22, 23, 24, 25, or 26, wherein the data packet is a first data packet, wherein the interface circuit is associated with a queue, and wherein the programmable circuit is configured to adjust a bit rate used to transmit a second data packet containing the result based on a delay associated with processing the payload in the in-network device and a load of the queue.

[0236] Example 28 includes the in-network device of any one of examples 22, 23, 24, 25, 26, or 27, wherein the payload is a first payload, the data packet is a first data packet, and wherein the programmable circuit is configured to decrease a bit rate used by the in-network device to transmit a third data packet based on a first delay associated with processing the first payload at the in-network device being greater than a second delay associated with processing a second payload at the in-network device, the first and third data packets being associated with the same workload type.

[0237] Example 29 includes the in-network device of any one of Examples 22, 23, 24, 25, 26, 27, or 28, wherein the data packet is a first data packet, and wherein the interface circuitry is configured to transmit a second data packet to identify: (a) a bit rate used to transmit a third data packet containing the result, and (b) at least one of: a delay associated with processing the payload at the in-network device; a load of a queue of the interface circuitry, the load associated with the third data packet; a first identifier of a third device from which the first data packet originated; or a second identifier of a fourth device at which the third data packet will terminate.

[0238] Example 30 includes the in-network device of any one of examples 22, 23, 24, 25, 26, 27, 28, or 29, wherein the header field is a first header field and wherein the data packet includes second header fields, the second header fields relating to a protocol for communication of the data packet.

[0239] Example 31 includes the in-network device of any of Examples 22, 23, 24, 25, 26, 27, 28, 29, or 30, wherein the payload includes a first payload, and wherein the programmable circuit is configured to process the first payload to generate a second payload, the second payload corresponding to a result of processing the payload; and to replace the first payload of the payload with the second payload.

[0240] Example 32 includes the in-network device of any of examples 22, 23, 24, 25, 26, 27, 28, 29, 30, or 31, wherein the header field identifies a workload type associated with the payload, and wherein the programmable circuit is configured to trigger a protocol corresponding to the workload type.

[0241] Example 33 includes the in-network device of any of Examples 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, or 32, wherein the header field identifies a workload type associated with the payload, and wherein the workload type corresponds to at least one of a compression workload, an encryption workload, a distribution / collection workload, a load balancing workload, a network address translation workload, a domain name system server workload, a caching workload, a data reduction workload, and a data aggregation workload.

[0242] Example 34 includes a non-transitory machine-readable data storage medium comprising instructions to cause programmable circuitry of an in-network device to at least: analyze a data packet to identify a header field of the data packet, the data packet originating from a first device in a network; process a payload included in the data packet in the in-network device based on the header field; and cause interface circuitry of the in-network device to transmit a result of processing the payload to at least one of the first device and a second device.

[0243] Example 35 includes the non-transitory machine-readable data storage medium of example 34, wherein the data packet is a first data packet, and wherein the instructions cause the programmable circuitry to determine a bit rate used to transmit a second data packet containing the result, the bit rate based on a delay associated with processing the payload at the in-network device, and causing the interface circuitry to transmit a third data packet to identify the bit rate for at least one of the first device or the second devices.

[0244] Example 36 includes the non-transitory machine-readable data storage medium of any one of examples 34 or 35, wherein the instructions cause the programmable circuit to determine whether a delay associated with processing the payload at the in-network device meets a delay threshold.

[0245] Example 37 includes the non-transitory machine-readable data storage medium of Example 36, wherein the data packet is a first data packet, and wherein the instructions cause the programmable circuit to determine, based on the delay satisfying the delay threshold, whether a bit rate currently used to transmit a second data packet containing the result satisfies an average of an upper bit rate threshold for the in-network device and a number of third data packets to be transmitted by the in-network device, wherein the first data packet is associated with a first workload, and wherein the third data packets are associated with corresponding second workloads.

[0246] Example 38 includes the non-transitory machine-readable data storage medium of example 37, wherein the delay is a first delay, the bit rate is a first bit rate, the delay threshold is a first delay threshold, and wherein the instructions cause the programmable circuit to, based on the first bit rate not meeting the average, set a second bit rate to be used to transmit the second data packet based on whether a difference between the first delay and the second delay meets a second delay threshold.

[0247] Example 39 includes the non-transitory machine-readable data storage medium of any one of examples 34, 35, 36, 37, or 38, wherein the data packet is a first data packet, wherein the interface circuit is associated with a queue, and wherein the instructions cause the programmable circuit to set a bit rate used to transmit a second data packet containing the result based on a delay associated with processing the payload at the in-network device and a load of the queue.

[0248] Example 40 includes the non-transitory machine-readable data storage medium of any one of examples 34, 35, 36, 37, 38, or 39, wherein the payload is a first payload, the data packet is a first data packet, and wherein the instructions cause the programmable circuit to decrease a bit rate used by the in-network device to transmit a third data packet based on a first delay associated with processing the first payload at the in-network device being greater than a second delay associated with processing a second payload at the in-network device, the first and third data packets being associated with the same workload type.

[0249] Example 41 includes the non-transitory machine-readable data storage medium of any one of Examples 34, 35, 36, 37, 38, 39, or 40, wherein the data packet is a first data packet, and wherein the instructions cause the programmable circuitry to transmit a second data packet to identify: (a) a bit rate used to transmit a third data packet containing the result, and (b) at least one of: a delay associated with processing the payload at the in-network device; a load of a queue of the interface circuitry, the load associated with the third data packet; a first identifier of a third device from which the first data packet originated; or a second identifier of a fourth device at which the third data packet will terminate.

[0250] Example 42 includes the non-transitory machine-readable data storage medium of any one of examples 34, 35, 36, 37, 38, 39, 40, or 41, wherein the header field is a first header field and wherein the data packet includes second header fields, the second header fields relating to a protocol for communication of the data packet.

[0251] Example 43 includes the non-transitory machine-readable data storage medium of any one of examples 34, 35, 36, 37, 38, 39, 40, 41, or 42, wherein the payload includes first payload data, and wherein the instructions cause the programmable circuit to process the first payload data to generate second payload data, the second payload data corresponding to the result of processing the payload; and to replace the first payload data of the payload with the second payload data.

[0252] Example 44 includes the non-transitory machine-readable data storage medium of any of Examples 34, 35, 36, 37, 38, 39, 40, 41, 42, or 43, wherein the header field identifies a workload type associated with the payload, and wherein the instructions cause the programmable circuitry to trigger a protocol corresponding to the workload type.

[0253] Example 45 includes the non-transitory machine-readable data storage medium of any one of Examples 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, or 44, wherein the header field identifies a workload type associated with the payload, and wherein the workload type corresponds to at least one of a compression workload, an encryption workload, a distribution / collection workload, a load balancing workload, a network address translation workload, a domain name system server workload, a caching workload, a data reduction workload, and a data aggregation workload.

[0254] Example 46 includes a method comprising: analyzing, using an instruction by programmable circuitry of an in-network device, a header field of a data packet, the data packet originating from a first device in a network; processing, by the programmable circuitry, in the in-network device, a payload included in the data packet based on the header field; and transmitting, by interface circuitry of the in-network device, a result of processing the payload to at least one of the first device and a second device.

[0255] Example 47 includes the method of Example 46, wherein the data packet is a first data packet, the method further comprising: determining a bit rate used to transmit a second data packet containing the result, the bit rate based on a delay associated with processing the payload at the in-network device; and transmitting a third data packet to identify the bit rate for at least one of the first device or the second devices.

[0256] Example 48 includes the method of any one of Examples 46 and 47, further including determining whether a delay associated with processing the payload at the in-network device satisfies a delay threshold.

[0257] Example 49 includes the method of Example 48, wherein the data packet is a first data packet, and wherein the method further includes determining, based on the delay satisfying the delay threshold, whether a bit rate currently used to transmit a second data packet containing the result satisfies an average of an upper bit rate threshold for the in-network device and a number of third data packets transmitted by the in-network device, wherein the first data packet is associated with a first workload, and wherein the third data packets are associated with corresponding second workloads.

[0258] Example 50 includes the method of Example 49, wherein the delay is a first delay, the bit rate is a first bit rate, the delay threshold is a first delay threshold, and wherein the method further includes setting, based on the first bit rate not satisfying the average, a second bit rate used to transmit the second data packet based on whether a difference between the first delay and the second delay satisfies a second delay threshold.

[0259] Example 51 includes the method of any one of examples 46, 47, 48, 49, or 50, wherein the data packet is a first data packet, wherein the interface circuit is associated with a queue, and wherein the method further includes adjusting, based on a delay associated with processing the payload at the in-network device and a load of the queue, a bit rate used to transmit a second data packet containing the result.

[0260] Example 52 includes the method of any one of Examples 46, 47, 48, 49, 50, or 51, wherein the payload is a first payload, the data packet is a first data packet, and wherein the method further includes decreasing a bit rate used by the in-network device to transmit a third data packet based on a delay associated with processing the first payload at the in-network device being greater than a second delay associated with processing a second payload at the in-network device, wherein the first and third data packets are associated with the same workload type.

[0261] Example 53 includes the method of any one of Examples 46, 47, 48, 49, 50, 51, or 52, wherein the data packet is a first data packet, and wherein the method further includes transmitting a second data packet to identify: (a) a bit rate used to transmit a third data packet containing the result, and (b) at least one of: a delay associated with processing the payload at the in-network device; a load of a queue of the interface circuit, the load associated with the third data packet; a first identifier of a third device from which the first data packet originated; or a second identifier of a fourth device at which the third data packet will terminate.

[0262] Example 54 includes the method of any one of examples 46, 47, 48, 49, 50, 51, 52, or 53, wherein the header field is a first header field and wherein the data packet includes second header fields, the second header fields relating to a protocol for communication of the data packet.

[0263] Example 55 includes the method of any one of Examples 46, 47, 48, 49, 50, 51, 52, 53, or 54, wherein the payload includes first payload data, and wherein processing the first payload data is configured to generate second payload data, the second payload data corresponding to a result of processing the payload; and wherein transmitting includes: (i) replacing the first payload data of the payload with the second payload data to update the data packet; and (ii) transmitting the updated data packet to the second device.

[0264] Example 56 includes the method of any of examples 46, 47, 48, 49, 50, 51, 52, 53, 54, or 55, wherein the header field identifies a workload type associated with the payload, and wherein the method further includes triggering a protocol corresponding to the workload type.

[0265] Example 57 includes the method of any one of Examples 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, or 56, wherein the header field identifies a workload type associated with the payload, and wherein the workload type corresponds to at least one of a compression workload, an encryption workload, a distribution / collection workload, a load balancing workload, a network address translation workload, a domain name system server workload, a caching workload, a data reduction workload, and a data aggregation workload.

[0266] Example 58 includes a first device comprising: machine-readable instructions; programmable circuitry configured to use the machine-readable instructions to generate a data packet including a payload and a header field, the header field configured to identify a workload type associated with the data packet; and interface circuitry configured to transmit the data packet to a network-internal device.

[0267] Example 59 includes the first device of Example 58, wherein the data packet is a first data packet, wherein the interface circuit is associated with a queue, and wherein the programmable circuit is configured to cause the interface circuit to transmit the first data packet to the first device at a bit rate based on a predicted delay and a utilization of the queue, wherein the predicted delay is based on a delay measurement of a second data packet.

[0268] Example 60 includes the first apparatus of Example 59, wherein the threshold is a first threshold and wherein the programmable circuit is configured to determine whether the predicted delay meets a second threshold.

[0269] Example 61 includes the first apparatus of any one of examples 58, 59, or 60, wherein the data packet is a first data packet, the header field is a first header field, the interface circuit is associated with a queue, and wherein the programmable circuit is configured to set a flag in a second header field of the first data packet based on a load of the queue satisfying a first threshold and based on a predicted delay satisfying a second threshold, the predicted delay being based on a delay measurement of a second data packet.

[0270] Example 62 includes the first apparatus of any one of examples 58, 59, 60, or 61, wherein the programmable circuit is configured to insert a workload type identifier into the header field of the data packet.

[0271] Example 63 includes the first apparatus of any one of examples 58, 59, 60, 61, or 62, wherein the data packet is a first data packet, the header field is a first header field, the interface circuit is associated with a queue, and wherein the programmable circuit is configured to identify a second data packet based on a second header field included in the second data packet, the second header field associated with the workload type, and to predict a first load of the queue based on a second load of the queue during a transmission of the second data packet.

[0272] Example 64 includes the first apparatus of any one of examples 58, 59, 60, 61, 62, or 63, wherein the programmable circuit is configured to determine an operation according to which an instruction needs to be processed, generating the instruction based on a workload corresponding to the workload type, and converting the instruction into the data packet based on the operation and a priority of the workload type.

[0273] Example 65 includes the first apparatus of any one of examples 58, 59, 60, 61, 62, 63, or 64, wherein the header field is a first header field and wherein the data packet includes second header fields, the second header fields relating to a protocol for communication of the data packet.

[0274] Example 66 includes the first device of any of examples 58, 59, 60, 61, 62, 63, 64, or 65, wherein the header field is configured to trigger a protocol in the in-network device corresponding to the workload type.

[0275] Example 67 includes the first apparatus of any one of Examples 58, 59, 60, 61, 62, 63, 64, 65, or 66, wherein the workload type corresponds to at least one of a compression workload, an encryption workload, a distribution / collection workload, a load balancing workload, a network address translation workload, a domain name system server workload, a caching workload, a data reduction workload, or a data aggregation workload.

[0276] Example 68 includes a non-transitory machine-readable data storage medium comprising instructions configured to cause a programmable circuit to generate at least one data packet including a payload and a header field, the header field configured to identify a workload type associated with the data packet; and to cause an interface circuit to transmit the data packet to an in-network device.

[0277] Example 69 includes the non-transitory machine-readable data storage medium of Example 68, wherein the data packet is a first data packet, wherein the interface circuitry is associated with a queue, and wherein the instructions cause the programmable circuitry to cause the interface circuitry to transmit the first data packet to the first device at a bit rate based on a predicted delay and a utilization of the queue, wherein the predicted delay is based on a delay measurement of a second data packet.

[0278] Example 70 includes the non-transitory machine-readable data storage medium of Example 69, wherein the threshold is a first threshold, and wherein the instructions cause the programmable circuit to determine whether the predicted delay meets a second threshold.

[0279] Example 71 includes the non-transitory machine-readable data storage medium of any one of examples 68, 69, or 70, wherein the data packet is a first data packet, the header field is a first header field, the interface circuit is associated with a queue, and wherein the instructions cause the programmable circuit to set a flag in a second header field of the first data packet based on a load of the queue satisfying a first threshold and based on a predicted delay satisfying a second threshold, the predicted delay based on a delay measurement of a second data packet.

[0280] Example 72 includes the non-transitory machine-readable data storage medium of any one of Examples 68, 69, 70, or 71, wherein the instructions cause the programmable circuit to insert a workload type identifier into the header field of the data packet.

[0281] Example 73 includes the non-transitory machine-readable data storage medium of any one of examples 68, 69, 70, 71, or 72, wherein the data packet is a first data packet, the header field is a first header field, the interface circuit is associated with a queue, and wherein the instructions cause the programmable circuit to identify a second data packet based on a second header field included in the second data packet, the second header field associated with the workload type, and to predict a first load of the queue based on a second load of the queue during a transmission of the second data packet.

[0282] Example 74 includes the non-transitory machine-readable data storage medium of any one of examples 68, 69, 70, 71, 72, or 73, wherein the instructions cause the programmable circuit to determine an operation according to which an instruction is to be processed, generating the instruction based on a workload corresponding to the workload type, and converting the instruction into the data packet based on the operation and a priority of the workload type.

[0283] Example 75 includes the non-transitory machine-readable data storage medium of any one of examples 68, 69, 70, 71, 72, 73, or 74, wherein the header field is a first header field and wherein the data packet includes second header fields, the second header fields relating to a protocol for communication of the data packet.

[0284] Example 76 includes the non-transitory machine-readable data storage medium of any of Examples 68, 69, 70, 71, 72, 73, 74, or 75, wherein the header field is configured to trigger a protocol in the in-network device corresponding to the workload type.

[0285] Example 77 includes the non-transitory machine-readable data storage medium of any one of Examples 68, 69, 70, 71, 72, 73, 74, 75, or 76, wherein the workload type corresponds to at least one of a compression workload, an encryption workload, a distribution / collection workload, a load balancing workload, a network address translation workload, a domain name system server workload, a caching workload, a data reduction workload, or a data aggregation workload.

[0286] Example 78 includes a method comprising: generating a data packet including a payload and a header field using an instruction for a programmable circuit, wherein the header field is configured to identify a workload type associated with the data packet; and transmitting, by an interface circuit, the data packet to a network-internal device.

[0287] Example 79 includes the method of Example 78, wherein the data packet is a first data packet, the interface circuitry is associated with a queue, and the method further includes transmitting the first data packet to the first device at a bit rate based on a predicted delay and a queue load, the predicted delay being based on a delay measurement of a second data packet.

[0288] Example 80 includes the method of Example 79, wherein the threshold is a first threshold, and wherein the method further includes determining whether the predicted delay meets a second threshold.

[0289] Example 81 includes the method of any one of examples 78, 79, or 80, wherein the data packet is a first data packet, the header field is a first header field, the interface circuit is associated with a queue, and wherein the method further includes setting a flag in a second header field of the first data packet based on a load of the queue satisfying a first threshold and based on a predicted delay satisfying a second threshold, the predicted delay based on a delay measurement of a second data packet.

[0290] Example 82 includes the method of any one of Examples 78, 79, 80, or 81, further including inserting a workload type identifier into the header field of the data packet.

[0291] Example 83 includes the method of any one of Examples 78, 79, 80, 81, or 82, wherein the data packet is a first data packet, the header field is a first header field, the interface circuit is associated with a queue, and wherein the method further includes: identifying a second data packet based on a second header field included in the second data packet, the second header field associated with the workload type; and predicting a first load of the queue based on a second load of the queue during transmission of the second data packet.

[0292] Example 84 includes the method of any of Examples 78, 79, 80, 81, 82, or 83, further including: determining an operation according to which a command needs to be processed, wherein the command is generated based on a workload corresponding to the workload type; and converting the command into the data packet based on the operation and a priority of the workload type.

[0293] Example 85 includes the method of any one of examples 78, 79, 80, 81, 82, 83, or 84, wherein the header field is a first header field and wherein the data packet includes second header fields, the second header fields relating to a protocol for communication of the data packet.

[0294] Example 86 includes the method of any of Examples 78, 79, 80, 81, 82, 83, 84, or 85, wherein the header field is configured to trigger a protocol in the in-network device corresponding to the workload type.

[0295] Example 87 includes the method of any one of Examples 78, 79, 80, 81, 82, 83, 84, 85, or 86, wherein the workload type corresponds to at least one of a compression workload, an encryption workload, a distribution / collection workload, a load balancing workload, a network address translation workload, a domain name system server workload, a caching workload, a data reduction workload, or a data aggregation workload.

[0296] Example 88 includes an in-network device comprising: interface circuitry to receive a data packet transmitted from a source to a destination in a network; machine-readable instructions; and programmable circuitry to use the machine-readable instructions to: access a payload of the data packet, wherein the payload is associated with a workload class identified by a header field of the data packet; perform an action associated with the workload class on a first payload of the data packet to generate a second payload; and cause the interface circuitry to forward the data packet containing an updated payload to the destination, wherein the updated payload is based on the second payload.

[0297] Example 89 includes the in-network device of Example 88, wherein the header field is a first header field and wherein the data packet includes second header fields, the second header fields relating to a protocol for communication of the data packet.

[0298] Example 90 includes the in-network device of any one of examples 88 or 89, wherein the second payload includes a result of the action, and wherein the programmable circuit replaces the first payload with the second payload to generate the updated payload.

[0299] Example 91 includes the in-network device of any one of examples 88, 89, or 90, wherein the programmable circuit is configured to trigger a protocol corresponding to the workload type.

[0300] Example 92 includes the in-network device of any one of Examples 88, 89, 90, or 91, wherein the workload type corresponds to at least one of a compression workload, an encryption workload, a distribution / collection workload, a load balancing workload, a network address translation workload, a domain name system server workload, a caching workload, a data reduction workload, or a data aggregation workload.

[0301] Example 93 includes the in-network device of any one of examples 88, 89, 90, 91, or 92, wherein the data packet is a first data packet, the programmable circuitry configured to determine a bit rate used to transmit the first data packet containing the updated payload, the bit rate based on a delay associated with processing the first payload at the in-network device, and the interface circuitry configured to transmit a second data packet to the source, the second data packet configured to identify the bit rate.

[0302] Example 94 includes the in-network device of any one of examples 88, 89, 90, 91, 92, or 93, wherein the interface circuit is associated with a queue, and wherein the programmable circuit is configured to adjust a bit rate used to transmit the data packet containing the updated payload based on a delay associated with processing the payload at the in-network device and a load of the queue.

[0303] The following claims are hereby incorporated by reference into this Detailed Description in their entirety. Although certain exemplary systems, devices, articles of manufacture, and methods have been disclosed herein, the scope of the invention of this patent is not so limited. On the contrary, this patent covers all systems, devices, articles of manufacture, and methods that reasonably fall within the scope of the claims of this patent.

Claims

[1] A device used in a network, the device comprising: an interface circuit to access a data packet containing a header comprising: (1) a destination address field for identifying a first network address of a destination device capable of performing an action, and (2) a workload class field for identifying a workload class associated with the action; machine-readable instructions; and at least one processor circuit to use the machine-readable instructions to: performing the action in the device based on the workload class field, the device having a second network address that is different from the first network address; Modifying an indicator field of the data packet to indicate that the action has been performed; and Cause the interface circuit to forward the data packet with the modified display field to the destination device. [2] The apparatus of claim 1, wherein the action includes an in-network computation primitive, the in-network computation primitive including at least one of a vector summation operation, a counting operation, a caching operation, a storing operation, a fetching operation, or a hash table operation. [3] The apparatus of claim 1 or claim 2, wherein the data packet is a first data packet, and wherein one or more of the at least one processor circuitry causes the interface circuitry to transmit a second data packet to a source device of the first data packet, the second data packet identifying at least one capability of an in-network computation primitive of the programmable circuitry. [4] The apparatus of claim 1 or claim 2, wherein the data packet is transmitted along a communication path from a source device to the destination device, and wherein one or more of the at least one processor circuits are configured to: Determining a level of congestion in another network device in the communication path; and Determine that the action should be performed based on the overload level in the device. [5] The apparatus of claim 4, wherein one or more of the at least one processor circuitry is configured to determine the congestion level based on at least one of a congestion notification field, a delay level field, a queue state field, or a fair rate field in the header of the data packet. [6] Apparatus according to claim 4 or claim 5, wherein one or more of the at least one processor circuits are arranged to perform the action in the apparatus during a delay introduced by the congestion level. [7] The apparatus of claim 1 or claim 2, wherein the data packet is a first data packet, wherein the action is a first action, wherein the first data packet is transmitted along a communication path from a source device to the destination device, and wherein one or more of the at least one processor circuits are configured to: Determining a level of congestion in a subsequent device in the communication path based on a feedback data packet from the subsequent device, wherein the feedback data packet includes at least one of: a bit rate used by the subsequent device to transmit a second data packet; a utilization of a queue in the subsequent device by the second data packet; or a processing delay associated with performing a second action associated with the second data packet in the subsequent device rather than in the destination device; and Determine that the action is performed based on the overload level in the subsequent device in the device. [8] The apparatus of claim 7, wherein one or more of the at least one processor circuits are configured to perform the action in the apparatus during a delay introduced by the congestion level in the subsequent apparatus. [9] The apparatus of any one of claims 1 to 8, wherein the action is configured to be performed based on a payload of the data packet, the payload being associated with an application, and wherein one or more of the at least one processor circuitry is configured to modify the payload before forwarding the data packet to the destination device. [10] Device according to one of claims 1, 2 or 9, wherein: the data packet is a first data packet; one or more of the at least one processor circuits are configured to determine a bit rate used to transmit the first data packet to the destination device, the bit rate being based on a delay associated with performing the action in the device; and the interface circuit is configured to transmit a second data packet to identify the bit rate to at least one of a source device of the data packet or the destination device. [11] Method comprising: Accessing a data packet in a second device, the data packet including a header comprising: (1) a destination address field for identifying a first network address of a first device capable of performing an action, and (2) a workload class field for identifying a workload class associated with the action; performing the action in the second device based on the workload class field, the second device having a second network address that is different from the first network address; Modifying an indicator field of the data packet to indicate that the action has been performed; and Forwarding the data packet with the modified display field to the first device. [12] The method of claim 11, wherein the action includes an in-network computation primitive, the in-network computation primitive including at least one of a vector summation operation, a counting operation, a caching operation, a storing operation, a fetching operation, or a hash table operation. [13] The method of claim 11 or claim 12, wherein the data packet is a first data packet, and wherein the method further includes transmitting a second data packet to a source device of the first data packet, the second data packet identifying at least one capability of an in-network computation primitive of the programmable circuit. [14] A method according to claim 11 or claim 12, wherein the data packet is transmitted along a communication path from a source device to the destination device, and wherein the method further comprises: Determining a level of congestion in another network device in the communication path; and Determine that the action should be performed based on the congestion level in the second device. [15] The method of claim 14, further comprising determining the congestion level based on at least one of a congestion notification field, a delay level field, a queue state field, or a fair rate field in the header of the data packet. [16] A method according to claim 14 or claim 15, further comprising performing the action in the second device during a delay introduced by the congestion level. [17] The method of claim 11 or claim 12, wherein the data packet is a first data packet, wherein the action is a first action, wherein the first data packet is transmitted along a communication path from a source device to the destination device, and wherein the method further comprises: Determining a level of congestion in a subsequent device in the communication path based on a feedback data packet from the subsequent device, wherein the feedback data packet includes at least one of: a bit rate used by the subsequent device to transmit a second data packet; a utilization of a queue in the subsequent device by the second data packet; or a processing delay associated with performing a second action associated with the second data packet in the subsequent device rather than in the destination device; and Determining that the action is performed in the second device based on the congestion level of the subsequent device. [18] The method of claim 17, further comprising performing the action in the second device during a delay introduced by the congestion level in the subsequent device. [19] The method of any one of claims 11 to 18, wherein the action is configured to be performed based on a payload of the data packet, the payload being associated with an application, and wherein the method further includes modifying the payload before forwarding the data packet to the first device. [20] A method according to any one of claims 11, 12 or 19, wherein the data packet is a first data packet and wherein the method further comprises: Determining a bit rate used to transmit the first data packet to the first device, the bit rate being based on a delay associated with performing the action in the second device; and Transmitting a second data packet to identify the bit rate to at least one of a source device of the data packet or the first device. [21] System comprising: a source device for generating a data packet including a header comprising: (1) a destination address field for identifying a first network address of a destination device and (2) a workload class field for identifying a workload class associated with an action capable of being performed by the destination device; and an intermediate device in a communication path between the source device and the destination device, the intermediate device being configured to: performing the action in the intermediate device based on the workload class field, wherein the intermediate device has a second network address that is different from the first network address; Modifying an indicator field of the data packet to indicate that the action has been performed; and Forward the data packet with the modified display field to the target device. [22] The system of claim 21, wherein the action includes an in-network computation primitive, the in-network computation primitive including at least one of a vector summation operation, a counting operation, a caching operation, a storing operation, a fetching operation, or a hash table operation. [23] A system according to claim 21 or claim 22, wherein: the data packet is a first data packet; the intermediate device is configured to transmit a second data packet to the source device of the first data packet, wherein the second data packet is configured to identify at least one capability of an in-network computing primitive of the intermediate device; and wherein the source device is designed to: Determining, based on the second data packet, at least one intra-network computation primitive that can be performed by the intermediate device; and Selecting the communication path of the first data packet based on at least one intra-network computation primitive and the workload class. [24] The system of any of claims 21 to 23, wherein the action is configured to be performed based on a payload of the data packet, the payload being associated with an application, and wherein the intermediate device is configured to modify the payload before forwarding the data packet to the destination device. [25] A computer-readable data storage medium comprising machine-readable instructions for causing a device to implement the method of any one of claims 11 to 20.

Images