Monitor and direct service requests to acceleration components

Abstract

Systems and methods for monitoring and directing service requests to acceleration components are provided. An example method in a system including a host configured to send a message corresponding to a service to any acceleration component in a set of acceleration components may include: tracking an input buffer occupancy corresponding to each acceleration component in the set of acceleration components. The method may also include: when a first input buffer occupancy of a first acceleration component in the set of acceleration components meets a predetermined criterion, generating a first penalty value for the first acceleration component based at least on the first input buffer occupancy associated with the first acceleration component, and comparing a second value indicating the availability of a second acceleration component for receiving messages from the host with a second dynamically updated value to determine whether the host can send a next message to the second acceleration component.

Description

Background Art

[0001] More and more users are accessing applications provided by computing, networking and storage resources located in data centers. These applications run in a distributed computing environment, sometimes also referred to as a cloud computing environment. Computer servers in a data center are interconnected via a network, and applications running on the computer servers can therefore communicate with each other via the network. In large data centers, message communication between computer servers can cause network congestion, resulting in poor performance.

[0002] Therefore, there is a need for methods and systems that alleviate at least some of these problems. Summary of the invention

[0003] In one example, the present disclosure relates to a method in a system, the system including a host configured to send a message corresponding to a service to any one of an acceleration component in a set of acceleration components. The method may include tracking an input buffer occupancy corresponding to each acceleration component in the set of acceleration components. The method may also include, when a first input buffer occupancy of a first acceleration component in the set of acceleration components meets a predetermined criterion, generating a first penalty value for the first acceleration component based at least on the first input buffer occupancy associated with the first acceleration component, and comparing a second value indicating the availability of a second acceleration component for receiving a message from the host with a second dynamically updated value to determine whether the host can send a next message to the second acceleration component.

[0004] In another example, the present disclosure relates to a system including a set of acceleration components and a host, the host being configured to send a message corresponding to a service to any acceleration component in the set of acceleration components. The system may also include a load balancer, the load balancer being configured to: (1) track an input buffer occupancy corresponding to each acceleration component in the set of acceleration components, and (2) when a first input buffer occupancy of a first acceleration component in the set of acceleration components meets a predetermined criterion, generate a first penalty value for the first acceleration component based at least on the first input buffer occupancy associated with the first acceleration component, and compare a second value indicating the availability of a second acceleration component for receiving a message from the host with a second dynamically updated value to determine whether the host can send a next message to the second acceleration component.

[0005] In another example, the present disclosure relates to a method in a system including a host, the host being configured to send a message corresponding to a service to any one of a set of acceleration components. The method may include: tracking an input buffer occupancy and a processing rate corresponding to each of the set of acceleration components. The method may also include: as long as the first acceleration component is determined to be in a healthy state based on a comparison of a first value indicating the availability of the first acceleration component with a first dynamically updated value, the host initiates the transmission of a first message to a first acceleration component selected from the set of acceleration components. The method may also include, when the first input buffer occupancy of the first acceleration component meets a predetermined criterion, generating a first penalty value for the first acceleration component based at least on the first input buffer occupancy and the first processing rate associated with the first acceleration component, and comparing a second value indicating the availability of a second acceleration component for receiving messages from the host with a second dynamically updated value to determine whether the host can send the next message to the second acceleration component.

[0006] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The present disclosure is illustrated by way of example and not limitation in the accompanying drawings, in which like reference numerals indicate similar elements. Elements in the drawings are illustrated for simplicity and clarity and are not necessarily drawn to scale.

[0008] Figure 1 is a diagram of an architecture that may include a software plane and an acceleration plane according to an example;

[0009] Figure 2 A diagram of a system for transmitting messages through an acceleration component configured to accelerate a service according to an example is shown;

[0010] Figure 3 A diagram showing a system environment for transmitting messages through an acceleration component configured to accelerate a service according to an example;

[0011] Figure 4 A diagram showing an acceleration component according to an example;

[0012] Figure 5 An arrangement of components for transmitting a service request from a host according to one example is shown;

[0013] Figure 6 A diagram showing a load balancer according to an example;

[0014] Figure 7 A system including load balancing according to an example is shown;

[0015] Figure 8 A flow chart showing a method for load balancing according to an example; and

[0016] Fig. 9 A flow chart of another method for load balancing according to an example is shown. DETAILED DESCRIPTION

[0017] The examples described in the present disclosure relate to methods and systems for providing message management between acceleration components that can be configured to accelerate services. Certain aspects of the present disclosure relate to monitoring and guiding service requests for acceleration components. Acceleration components include, but are not limited to, hardware components that can be configured (or configured) to perform functions corresponding to services provided by, for example, a data center more efficiently than software running on a general-purpose central processing unit (CPU). Acceleration components may include field programmable gate arrays (FPGAs), graphics processing units (GPUs), application-specific integrated circuits (ASICs), erasable and / or complex programmable logic devices (PLDs), programmable array logic (PAL) devices, general-purpose array logic (GAL) devices, and massively parallel processor arrays (MPPA) devices. Image files may be used to configure or reconfigure acceleration components, such as FPGAs. The information contained in the image files may be used to program the hardware components of the acceleration components (e.g., logic blocks and reconfigurable interconnects of FPGAs) to achieve desired functionality. The desired functionality may be implemented to support any service that may be provided via a combination of computing, networking, and storage resources (such as via a data center or other infrastructure for delivering services).

[0018] The various aspects described may also be implemented in a cloud computing environment. Cloud computing may refer to a model for implementing on-demand network access to a shared pool of configurable computing resources. For example, cloud computing may be employed in a market to provide ubiquitous and convenient on-demand access to a shared pool of configurable computing resources. A shared pool of configurable computing resources may be quickly configured via virtualization and released with a low management workload or service provider interaction, and then expanded accordingly. The cloud computing model may consist of various features, such as, for example, on-demand self-service, extensive network access, resource pools, rapid elasticity, measured services, and the like. The cloud computing model may also disclose various service models, such as, for example, software as a service ("SaaS"), platform as a service ("PaaS"), and infrastructure as a service ("IaaS"). The cloud computing model may also be deployed using different deployment models, such as private clouds, community clouds, public clouds, hybrid clouds, and the like.

[0019] Data center deployment may include a hardware acceleration plane and a software plane. The hardware acceleration plane may include multiple networked acceleration components (e.g., FPGA). The software plane may include multiple networked software-implemented host components (e.g., central processing unit (CPU)). Network infrastructure may be shared between the hardware acceleration plane and the software plane. In some environments, the software-implemented host components are locally linked to the corresponding acceleration components. The acceleration components may communicate with each other via a network protocol. In order to provide reliable services to users of services provided via a data center, any communication mechanism may be required to meet certain performance requirements, including reliability. In some examples, the present disclosure provides a lightweight transport layer for meeting such requirements. In one example, the acceleration components may communicate with each other via a lightweight transport layer (LTL).

[0020] An application running on a host can send a request to a collection of acceleration components to perform a task. When the computation required by the request is completed, a response can be sent to the application. Some tasks have strict real-time requirements, so restrictions are imposed on the processing time of the acceleration components. Each acceleration component in the group is independent of each other, but can perform the same computation and therefore appear uniform from the application's perspective. However, from the hardware's perspective, each acceleration component can be in a different state and can even have a different processing speed. This uneven behavior can be caused by power throttling, hardware errors, overloading of certain nodes, or hardware differences associated with the acceleration components. This uneven behavior can be a problem for time-critical applications.

[0021] One metric used to evaluate time-critical applications is tail latency, which can indicate the maximum time taken to process a single request for a given percentile of all requests. In an ideal scenario, in order to maintain low tail latency, each request can be directed to the node that can generate a response in the shortest time. However, in practice, in a distributed computing environment, it is difficult to achieve the ideal result. In order to execute the task, the software associated with a particular service can pick an acceleration component at the beginning of a round-robin queue or some other type of queue. Next, a message can be sent to the acceleration component. While the software can include some monitoring of the acceleration component to ensure that the software does not send messages to an acceleration component that is out of service or does not belong to the group, any status update on the acceleration component will cause significant latency introduced by the network, the interconnection with the acceleration component, and (multiple) device drivers. In addition, if a problem occurs with the acceleration component after the message has been sent to the acceleration component, the software corresponding to the service may not be aware of the failure until it receives an indication of a fault message from the relevant acceleration component. For example, an acceleration component may be subject to thermal throttling, but the software corresponding to the service may not be aware of this until it has sent a service request to the acceleration component. Therefore, the software may use stale information about the system when deciding which acceleration component should receive the next request.

[0022] Certain examples associated with the present disclosure relate to using a load balancer to monitor and direct service requests. An example hardware unit for load balancing can be designed to collect information from a processing node (e.g., an acceleration component) and react to state changes depending on its configuration. Any processing node (e.g., an acceleration component) can be disabled by a load balancer based on information collected at a specified time and enabled by a specific event. All requests that are monitored by a load balancer and directed to a processing node under the control of the load balancer are subject to load balancing restrictions. In one example, load balancing can be performed by checking the target processing node of the request and deciding whether it should be changed depending on the state of the target node. In the case where the load balancer changes the target processing node, an algorithm can be used to select an alternative processing node. Therefore, when software associated with a service selects an original destination for a request (e.g., based on a round-robin scheme), the load balancer can swap out the destination node without the software knowing. In this way, requests from a service can be transparently redirected.

[0023] Figure 1An architecture 100 is shown that may include a software plane 104 and an acceleration plane 106 according to one example. The software plane 104 may include a collection of software-driven host components (each indicated by the symbol "S"), while the acceleration plane may include a collection of acceleration components (each indicated by the symbol "A"). In this example, each host component may correspond to a server computer that executes machine-readable instructions using one or more central processing units (CPUs). In one example, these instructions may correspond to a service, such as a text / image / video search service, a translation service, or any other service that may be configured to provide useful results to a user of the device. Each CPU may execute instructions corresponding to various components (e.g., software modules or libraries) of the service. Each acceleration component may include hardware logic for implementing functionality, such as, for example, part of a service provided by a data center.

[0024] The acceleration plane 106 can be constructed using a heterogeneous or homogeneous collection of acceleration components, including different types of acceleration components and / or the same type of acceleration components with different capabilities. For example, the acceleration plane 106 can include a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), an application specific standard product (ASSP), a system on a chip (SOC), a complex programmable logic device (CPLD), other types of programmable hardware logic devices, etc. The acceleration plane 106 can provide a reconfigurable structure of acceleration components.

[0025] The host component can generally be any computing component that can execute machine-readable instructions to perform operations by using each of its CPU hardware threads. The acceleration component can use several parallel logic elements to perform operations to perform computing tasks. For example, an FPGA can include several gate arrays that can be configured to perform certain computing tasks in parallel. Therefore, compared to a software-driven host component, the acceleration component can perform some operations in a shorter time. In the context of architecture 100, "acceleration" reflects its potential to accelerate the functions performed by the host component.

[0026] In one example, architecture 100 may correspond to a data center environment including a large number of servers. The servers may correspond to host components in software plane 104. In another example, architecture 100 may correspond to an enterprise system. In yet another example, architecture 100 may correspond to a user device or appliance using at least one host component that is capable of accessing two or more acceleration components. In practice, other implementations for architecture 100 are possible, depending on the requirements of the service.

[0027] The network 120 may couple host components in the software plane 104 to other host components and acceleration components in the acceleration plane 106 to other acceleration components. In this example, the host components may interact with each other using the network 120 and the acceleration components may interact with each other using the network 120. The interactions between the host components in the software plane 104 may be independent of the interactions between the acceleration components in the acceleration plane 106. In this example, two or more acceleration components may communicate in a transparent manner with respect to the host components in the software plane 104, outside of the direction of the host components and without the host components even being "aware" of the particular interactions occurring in the acceleration plane 106.

[0028] The architecture 100 may use any of a variety of different protocols to facilitate communication between acceleration components through the network 120, and may use any of a variety of different protocols to facilitate communication between host components through the network 120. For example, the architecture 100 may use the Ethernet protocol to transmit Internet Protocol (IP) packets through the network 120. In one implementation, each local host component in the server is given a single physical IP address. Local acceleration components in the same server may take the same IP address. The server may determine in different ways whether an incoming packet is destined for a local host component or a local acceleration component. For example, a packet destined for a local acceleration component may be formulated as a UDP packet with a specific port; on the other hand, a host-defined packet may not be expressed in this manner. In another example, packets belonging to the acceleration plane 106 and packets belonging to the software plane 104 may be distinguished based on the value of a status flag in each packet. In one example, the architecture 100 may be viewed as two logical networks (software plane 104 and acceleration plane 106) that may share the same physical network communication link. Packets associated with the two logical networks may be distinguished from each other by their respective service classes.

[0029] On the other hand, each host component in the architecture 100 is coupled to at least one acceleration component in the acceleration plane 104 via a local link. For example, the host component and the acceleration component can be arranged together and maintained as a single serviceable unit (e.g., a server) within the architecture 100. In this arrangement, the server can be referred to as a "local" host component to distinguish it from other host components associated with other servers. Similarly, the acceleration component(s) of the server can be referred to as (multiple) "local" acceleration components to distinguish them from other acceleration components associated with other servers.

[0030] As depicted in architecture 100, host component 108 may be coupled to acceleration component 110 via local link 112 (e.g., a peripheral component interconnect express (PCIe) link). Thus, from the perspective of acceleration component 110, host component 108 may be a local host component, and from the perspective of host component 108, acceleration component 110 may be a local acceleration component. The local links of host component 108 and acceleration component 110 may form part of a server. More generally, host components in software plane 104 may be collectively represented as local A -to-local S The host component 108 is locally coupled to the acceleration component in the acceleration plane 106 by many separate links of the coupling 114. In this example, the host component can interact directly with any locally linked acceleration component. The host component can initiate communication with the locally linked acceleration component to cause further communication between multiple acceleration components. For example, the host component can issue a request for a service (or part thereof), wherein the functionality for the service or part thereof is composed of a set of one or more acceleration components in the acceleration plane 106. The host component can also interact indirectly with other acceleration components in the acceleration plane 106 (to which the host component is not locally linked). For example, the host component 108 can communicate indirectly with the acceleration component 116 via the acceleration component 110. In this example, the acceleration component 110 communicates with the acceleration component 116 via the link 118 of the network (e.g., the network 120).

[0031] The acceleration components in the acceleration plane 106 can be advantageously used to robustly accelerate larger scale services in a data center. By using low latency interconnects to perform computations across multiple acceleration components, large portions of complex data center services can be mapped to acceleration components (e.g., FPGAs). Acceleration components can also be appropriately reconfigured to provide different service functionality at different times. Figure 1 A certain number of components of the architecture 100 are shown arranged in a certain manner, but there may be a greater or lesser number of components arranged differently. In addition, the various components of the architecture 100 may also be implemented using other technologies.

[0032] Figure 2A diagram of a system 200 for transmitting or retransmitting a message by an acceleration component configured to accelerate a service according to an example is shown. In one example, the system 200 can be implemented as a server rack in a data center. Servers 204, 206, and 208 can be included in a rack in a data center. Each of the servers 204, 206, and 208 can be coupled to a top-of-rack (TOR) switch 210. Other racks, although not shown, can have similar configurations. Server 204 can also include a host component 212, including CPUs 214, 216, etc. The host component 212 can be included in the software plane 104 along with the host components from servers 206 and 208. Server 204 can also include an acceleration component 218. The acceleration component 218 can be included in the acceleration plane 106 along with the acceleration components from servers 206 and 208.

[0033] The acceleration component 218 can be directly coupled to the host component 212 via a local link 220 (e.g., a PCIe link). Thus, the acceleration component 218 can treat the host component 212 as a local host component. The acceleration component 218 and the host component 212 can also be indirectly coupled through a network interface controller 222 (e.g., used to communicate across the network infrastructure 120). In this example, the server 204 can load an image representing service functionality onto the acceleration component 218.

[0034] The acceleration component 218 can also be coupled to the TOR switch 210. Thus, in the system 200, the acceleration component 218 can represent a path for the host component 212 to interact with other components in the data center (including other host components and other acceleration components). The system 200 allows the acceleration component 218 to perform processing (e.g., by performing encryption, compression, etc.) on packets received from (and / or sent to) the TOR switch 210 without burdening the CPU-based operations performed by the host component 212. Although Figure 2 A certain number of components of system 200 arranged in a certain manner is shown, but there may be a greater or lesser number of components arranged differently. In addition, various components of system 200 may also be implemented using other technologies.

[0035] Figure 3A diagram of a system environment 300 for transmitting or retransmitting messages through an acceleration component configured to accelerate a service according to an example is shown. In this example, IP routing can be used to transmit or receive messages between TOR switches (including TOR switch 1 302, TOR switch 2 304, and TOR switch N 306). Each server or server group can have a single "physical" IP address that can be provided by a network administrator. Therefore, in this example, server group 1 320, server group 2 322, and server group N 324 can each include servers, each of which can have a "physical" IP address. The acceleration component can use the physical IP of its server as its address. In order to distinguish between IP packets destined for the host and packets destined for the acceleration component, a UDP packet with a specific port indicating the acceleration component as the destination can be used. The acceleration component can use the layer 3 functionality corresponding to the seven-layer open system interconnection (OSI) model to transmit messages to a selected set of acceleration components associated with different TOR switches. The layer 3 functionality can be similar to the functionality provided by the network layer of the OSI model. In this example, the acceleration component can transmit a point-to-point message to each of the other related acceleration components associated with the corresponding TOR switch. These acceleration components can then use layer 2 Ethernet broadcast packets to send data to all acceleration components associated with the TOR switch. Layer 2 functionality can be similar to the functionality provided by the data link layer of the OSI model. Layer 2 functionality may include media access control, flow control, and error checking. In one example, this step will not require any broadcast support from the network that interconnects the acceleration plane and the software plane. This can advantageously alleviate the need for multicast functionality provided by routers or other network infrastructure. This in turn can reduce the complexity of deploying and managing acceleration components. In addition, in general, higher-level networks (e.g., networks including routers and other TOR switches) may be oversubscribed, which in turn can reduce the bandwidth available for acceleration components that use higher networks for communication. In contrast, in this example, acceleration components that share a TOR switch can advantageously have higher bandwidth available for them to use for any message transmission from one acceleration component to another. Although Figure 3 A certain arrangement of switches and routers is shown, but other arrangements may be used. In addition, the server may be connected to Figure 3 The groups are grouped differently as shown in .

[0036] Figure 4 A diagram of an acceleration component 400 according to one example is shown. The acceleration component 400 can be included in the acceleration plane 106. Components included in the acceleration component 400 can be implemented on hardware resources of the acceleration component 400 (eg, logic blocks and programmable interconnects).

[0037] The acceleration component 400 may include an application logic 406, a soft shell 404 associated with a first resource set, and a shell 402 associated with a second resource set. The resources associated with the shell 402 may correspond to lower-level interface-related components that may generally remain the same in many different application scenarios. The resources associated with the soft shell 404 may remain the same across at least some different application scenarios. The application logic 406 may be further conceptualized as including an application domain (e.g., a "role"). An application domain or role may represent a portion of the functionality included in a composite service spread across multiple acceleration components. The roles at each acceleration component in the acceleration component set may be linked together to create a group that provides service acceleration for an application domain.

[0038] The application domain hosts application logic 406 that performs service-specific tasks, such as a portion of functionality for ranking documents, encrypting data, compressing data, facilitating computer vision, facilitating speech translation, machine learning, etc. Resources associated with the soft shell 404 are generally less restricted from changing than application resources, and resources associated with the shell 402 are less restricted from changing than resources associated with the soft shell 404 (although changing (reconfiguring) any component of the acceleration component 400 is possible).

[0039] In operation, in this example, application logic 406 interacts with shell resources and soft shell resources in a manner similar to the way software-implemented applications interact with their underlying operating system resources. From an application development perspective, the use of generic shell resources and soft shell resources enables developers to avoid having to recreate these generic components for each service.

[0040] Referring first to the housing 402, the housing resources may include a bridge 408 for coupling the acceleration component 400 to a network interface controller (via a NIC interface 410) and a local top-of-rack switch (via a TOR interface 412). The bridge 408 also includes a data path that allows traffic from the NIC or TOR to flow into the acceleration component 400 and allows traffic from the acceleration component 400 to flow out to the NIC or TOR. Internally, the bridge 408 may consist of various FIFOs (414, 416) that buffer received packets and various selectors and arbitration logic that route packets to their intended destinations. The bypass control component 418, when activated, may control the bridge 408 to transmit packets between the NIC and the TOR without further processing by the acceleration component 400.

[0041] The memory controller 420 manages the interaction between the acceleration component 400 and a local memory 422, such as DRAM memory. As part of its services, the memory controller 420 may perform error correction.

[0042] The host interface 424 may provide functionality that enables the acceleration component 400 to interact with a local host component (not shown). In one implementation, the host interface 424 may use a peripheral component interconnect express (PCIe) in conjunction with direct memory access (DMA) to exchange information with the local host component. The external housing may also include various other features 426, such as a clock signal generator, status LEDs, error correction functionality, etc.

[0043] The elastic router 428 can be used to route messages between various internal components of the acceleration component 400 and between the acceleration component and external entities (e.g., via the transmission component 430). Each such endpoint can be associated with a corresponding port. For example, the elastic router 428 is coupled to the memory controller 420, the host interface 424, the application logic 406, and the transmission component 430.

[0044] A load balancer 436 may be coupled to the host interface 424 and the elastic router 428. The load balancer 436 may be designed to collect information and react to state changes depending on its configuration. Although the load balancer 436 is shown as part of the acceleration component 400, the load balancer 436 may be part of a different arrangement.

[0045] The transport component 430 can formulate packets for transmission to a remote entity (such as other acceleration components) and receive packets from a remote entity (such as other acceleration components). In this example, the 3-port switch 432 takes over the function of routing packets between the NIC and the TOR and between the NIC or TOR and the local port associated with the acceleration component 400 when activated.

[0046] The 3-port switch 432 can provide a feature to prevent packets for acceleration components from being sent to the host system. If the data network supports several lossless categories of business, the 3-port switch 432 can be configured to provide sufficient support to buffer and pause incoming lossless streams to allow it to insert its own business into the network. To support this, the 3-port switch 432 can be configured to distinguish lossless business categories (e.g., remote direct memory access (RDMA)) from lossy (e.g., TCP / IP) flow categories. Fields in the packet header can be used to identify which business category the packet belongs to. Configuration memory can be used to store any configuration files or data structures corresponding to the 3-port switch 432.

[0047] The 3-port switch 432 may have a host side port connected to a first MAC and a network side port connected to a second MAC. A third local port may provide internal services to a transport component (e.g., transport component 430). With some limitations, the 3-port switch 432 may generally operate as a network switch.

[0048] The diagnostic logger 434 can store information about the operations performed by the router 428, the transport component 430, and the 3-port switch 432 in a circular buffer. For example, the information can include data about the origin and destination IP addresses of the packet, host-specific data, or timestamps. The logs can be stored as part of a telemetry system (not shown) so that a technician can study the logs to diagnose the cause of a failure or suboptimal performance in the acceleration component 400.

[0049] Acceleration plane 106 may include multiple acceleration components, such as acceleration component 400. Acceleration components may communicate with each other using different network topologies (rather than using network 120 for communication). In one aspect, acceleration components are directly connected to each other, such as, for example, in a two-dimensional torus. Figure 4 A specific number of components of the acceleration assembly 400 arranged in a certain manner are shown, but there may be a greater or lesser number of components arranged differently. In addition, the various components of the acceleration assembly 400 may also be implemented using other technologies.

[0050] Figure 5 An arrangement of components for transmitting a service request from a host interface 508 according to one example is shown. According to one example, a transmitting component 500 (corresponding to Figure 4 ) is coupled to a 3-port switch 502 (corresponding to Figure 4 3 ports 432 of the example) and an elastic router 504. The elastic router 504 can be coupled to a load balancer 506, which in turn can be coupled to a host interface 508.

[0051] In this example, the load balancer 506 is a hardware unit that is configured to monitor the acceleration components in the acceleration group and direct service-related requests to the appropriate acceleration component. In this example, one of the main purposes of the load balancer is to direct incoming requests to the acceleration component that will generate the correct response without violating the quality of service specified by the application logic. Additional details about the load balancer 506 are as follows: Figure 6 To provide.

[0052] In one example, application logic corresponding to a service such as a search result ranking service (e.g., Figure 4The application logic 406 of the accelerator components can be divided and mapped to the roles of multiple accelerator components. As described earlier, the application logic can be conceptualized as including application domains (e.g., "roles"). An application domain or role can represent a portion of the functionality included in a combined service spread across multiple acceleration components. The roles of each acceleration component in the acceleration component collection can be linked together to create a group that provides service acceleration for the application domain. Each application domain can host application logic that performs service-specific tasks (such as a portion of the functionality for ranking documents, encrypting data, compressing data, facilitating computer vision, facilitating speech translation, machine learning, extracting features, etc.).

[0053] Acceleration components may be grouped together as part of a graph. Grouped acceleration components do not need to be physically close to each other; instead, they may be associated with different parts of a data center and may still be grouped together by linking them as part of an acceleration plane. In one example, the graph may have a specific network topology, depending on which acceleration components associated with which TOR switches are coupled together to accelerate a service. The network topology may be dynamically created based on configuration information received from a service manager of the service. The service manager may be a higher-level software associated with a service. In one example, the network topology may be dynamically adjusted based on at least one performance metric associated with a network (e.g., network 120) interconnecting an acceleration plane and a software plane, the software plane including a host component configured to execute instructions corresponding to at least one service. The service manager may monitor network performance using a telemetry service. A network performance metric may be selected in substantially real time based at least on the requirements of at least one service. At least one network performance metric may include latency, bandwidth, or any other performance metric specified by a service manager or application logic corresponding to at least one service.

[0054] The transport component 500 can be configured to act as an autonomous node on the network. In one embodiment, the transport component 500 can be configured in an environment or shell in which any process or execution unit can be instantiated. The use of the transport component 500 can be advantageous because of the proximity between the application logic and the network, and the removal of host-based burdens such as navigating complex network stacks, interrupt handling, and resource sharing. Therefore, applications or services using acceleration components and transport components (such as the transport component 500) may be able to communicate with lower latency and higher throughput. The transport component 500 itself can be an agent that generates and consumes network traffic for its own purposes.

[0055] The transmission component 500 can be used to implement functionality associated with a mechanism or protocol for exchanging data, including transmitting or retransmitting messages. In this example, the transmission component 500 can include transmission logic 510, reception logic 512, soft shell 514, connection management logic 516, configuration memory 518, transmission buffer 520, and reception buffer 522. These elements can operate to provide efficient and reliable communication between transmission components that can be included in the acceleration component.

[0056] In one example, the transport component 500 can be used to implement functionality associated with a lightweight transport layer (LTL). Consistent with this example of LTL, the transport component 500 can expose two main interfaces for the LTL: one for communication with a 3-port switch 502 (e.g., a local network interface, which can then be connected to a network switch, such as a TOR switch), and the other for communication with an elastic router 504 (e.g., an elastic router interface). In this example, the local network interface (local_*) can contain NeworkStream, Ready, and Valid for both Rx and Tx directions. In this example, the elastic router interface (router_*) can expose a FIFO-like interface that supports multiple virtual channels and a credit-based flow control scheme. The transport component 500 can be configured via a configuration data structure (struct) for runtime controllable parameters, and can output a status data structure (a structure for status monitoring by a host or other soft shell logic). Table 1 below shows an example of an LTL top-level module interface.

[0057]

[0058] Table 1

[0059] Table 2 below shows example static parameters that can be set for an LTL instance at compile time. The values ​​used for these parameters are examples only, and more or fewer parameters may be specified.

[0060] Parameter name Configuration Values MAX_VIRTUAL CHANNELS 8 ER_PHITS_PER_FLIT 4 MAX_ER_CREDITS 256 EXTRA_SFQ_ENTRIES 32

[0061] Table 2

[0062] Thus, as noted in Table 2 above, this will configure a buffer of MAX_VIRTUAL_CHANNELS + EXTRA_SFQ_ENTRIES MTU size for the LTL instance. Elastic router credits (ER_CREDITS) may guarantee at least 1 credit per virtual channel (VC), as well as a dynamically calculated number of additional credits. The transport component 500 may expose a configuration input port that sets multiple runtime values. The configuration port may be defined as part of the LTLConfiguration structure data structure. The following table (Table 3) lists the fields of the example data structure:

[0063]

[0064]

[0065] Table 3

[0066] The functionality corresponding to the fields shown in Table 3 may be combined or further separated. Certain fields may also be located in memory indexed by address or descriptor fields in the LTLConfiguration structure data structure. Similarly, a special instruction may provide information related to any one of the fields in Table 3, or it may combine information from these fields. Other changes may be made to the LTLConfiguration structure data structure and format without departing from the scope of this disclosure.

[0067] As part of LTL, in one example, all messages can be encapsulated in an IPv4 / UDP frame. Table 4 below shows an example packet format for encapsulating messages in such a frame. The Group column shows the various field groups in the packet structure. The Description column shows the fields corresponding to each group in the packet structure. The Size column shows the size of each field in bits. The Value column provides the value of the field and, if necessary, provides an example description of the relevant field.

[0068]

[0069] Table 4

[0070] The functionality corresponding to the fields shown in Table 4 may be combined or further separated. Certain fields may also be in memory indexed by an address or descriptor field in the packet. Similarly, a special instruction may provide information associated with any one of the fields in Table 4, or it may combine information from these fields. Other changes may be made to the packet structure and format without departing from the scope of this disclosure.

[0071] The connection management logic 516 may provide a register interface to establish a connection between the transport components. The connection management logic 516, together with software (e.g., a soft shell), may establish a connection before data may be transmitted or received. In one example, there are two connection tables that may control the state of a connection: a send connection table (SCT) and a receive connection table (RCT). Each of these tables may be stored as part of a configuration memory 518 or some other memory associated with the transport component 500. Each entry in the SCT, a send connection table entry (SCTE), may store the current sequence number of a packet and other connection states used to construct the packet, such as a destination MAC address. By comparing the destination IP address and the virtual channel field provided by the elastic router 504, a request arriving from the elastic router 504 may be matched to the SCTE. At most one connection may target a destination IP address and a VC pair. Therefore, the tuple {IP, VC} may be a unique key (database term) in the table. There may be two entries in the table with the same VC—for example, {IP: 10.0.0.1, VC: 0} and {IP: 10.0.0.2, VC: 0}. It is also possible to have two entries with the same IP address and different VCs: {IP:10.0.0.1,VC:0} and {IP:10.0.0.1,VC:1}. However, two entries with the same {IP,VC} pair are not allowed. The number of entries supported by LTL is configurable at compile time.

[0072] The elastic router 504 can move data in units of flits, which are 128B in size (32B×4 cycles). A message can consist of multiple flits, de-marked by start and last markers. In one example, when the elastic router 504 selects a flow to be sent from an input port to an output port for a given virtual channel, the entire message must be delivered before another message begins to arrive on the same virtual channel. The connection management logic 516 may need to package messages from the elastic router 504 into fragments of the maximum transmission unit (MTU) size of the network. This can be done by buffering data on each virtual channel until one of the following conditions is met: (1) the last marker is seen in the flits or (2) the MTU's worth of data (or a size appropriately reduced to accommodate header and alignment requirements). In this implementation, the MTU for LTL payloads can be 1408 bytes. When one of the requirements is met, the transport component 500 can attempt to send that packet via the transmission logic 510. The packet destination may be determined by a combination of which virtual channel the message arrives on at the transport component 500 input (from the elastic router 504) and the message header that may arrive during the first cycle of the message from the elastic router 504. These two values ​​may be used to index into the send connection table, which may provide the destination IP address and sequence number for the connection. In this example, the sequence number of each packet transmitted on a given connection should be one greater than the previous packet for that connection. The only exception may be a retransmission, which may see a discarded or unacknowledged packet retransmitted with the same sequence number as when it was originally sent. The first packet sent on a connection may have the sequence number set to 1. Thus, as an example, for a collection of microslices arriving on various virtual channels (VCs) in the transport component 500 from the elastic router 504, a buffer (e.g., receive buffer 522) may be used to buffer the data until the end of the message or the MTU's worth of data has been received, and then a packet may be output. In this example, the LTL instance may send the message immediately after buffering the data - i.e., it does not wait for an ACK for the first message before sending the next message. There may be no maximum message size. The LTL instance might simply chunk the message into MTU-sized packets and transmit them as soon as MTU's worth of data is ready. Similarly, in this example, there is no "message length" field anywhere in the packet - only the payload size of each packet. The transport component 500 may have no a priori knowledge of how much data the message will contain. Preferably, the LTL instance associated with the transport component 500 can deliver arriving micro-slices matching a given SCT entry in sequence, even in the face of drops and timeouts. There may be no ordering guarantees for micro-slices matching different SCT entries.

[0073] In this example, the transport component 500 will output one credit for each virtual channel and then one credit for each shared buffer. Credits will be returned after each flits - except when a flits completes the MTU buffer. This can occur if the last flag is received or when the flits contain the MTUth byte of the message. Credits consumed in this manner can be held by the transport component 500 until the packet is acknowledged.

[0074] In terms of the reception of packets by the LTL instance associated with the transport component 500, in one example, packets arriving from the network are matched to RCT entries (RCTEs) by fields in the packet header. Each RCTE stores the last sequence number and which virtual channel (VC) outputs the packet from the transport component 500 to the elastic router 504. Multiple entries in the RCT can point to the same output virtual channel. The number of entries supported by the LTL can be configured at compile time. When a packet arrives at a local port from a network switch, the transport component 500 can determine which entry in the receive connection table (RCT) the packet is paired with. If there is no matching RCT table, the packet can be discarded. The transport component 500 can check whether the sequence number matches the expected value from the RCT entry. If the sequence number is greater than the RCT entry, the packet can be discarded. If the sequence number is less than the RCT entry expectation, an acknowledgment (ACK) can be generated and the packet can be discarded. If it matches, the transport component 500 can grab the virtual channel field of the RCT entry. If the number of available elastic router (ER) credits for the virtual channel is sufficient to cover the packet size, the transport component 500 can accept the packet. If insufficient credits are available, the transport component 500 may drop the packet. When a packet is accepted, an acknowledgment (ACK) may be generated and the RCT entry sequence number may be incremented. The elastic router 504 may use the packet header to determine the final endpoint to which the message is destined. The transport component 500 may require sufficient credits to deliver an entire packet's worth of data to the elastic router 504 to make progress. To help ensure that all VCs can make progress, the transport component 500 may require the elastic router 504 to provide dedicated credits for each VC to process at least one MTU of data for each VC. In this example, it may be assumed that there is no shared credit.

[0075] SCT / RCT entries can be written by software. In one example, the software can keep a mirror of the connection settings. To update an SCT or RCT entry, the user can write to the register_wrdata_in port, which can be hooked to a register in the soft shell or environment corresponding to the application logic. Table 5 below is an example of a data structure format that can be used to update an entry in the SCT or RCT.

[0076]

[0077] Table 5

[0078] In order to write to the SCT entry, scte_not_rcte can be set to 1, the sCTI value can be set to the index value of the SCT being written, and then the other fields of the data structure in Table 5 can be appropriately set. Regarding timing, the value of register_write_in can be switched to a high level for at least one cycle. rCTI can be set to the RCT entry of the remote acceleration component (in this example, rCTI is included in the UDP packet sent to the acceleration component, and this is the way to find the correct connection on the other end). IPAddr can be set to the IP address of the destination acceleration component. MacAddr can be set to the MAC address of the host on the same LAN segment as the acceleration component or the MAC address of the router of the remote host. VirtualChannel can be set by looking it up in the micro slice reached from the elastic router 504. In order to write to the RCT entry, scte_not_rcte can be set to 0, the rCTI value can be set to the index value of the RCT being written, and then the other fields of the data structure in Table 5 can be appropriately set. rCTI can be set to the RCT entry of the sending acceleration component. IPAddr can be set to the IP address of the sending acceleration component. For the purpose of writing to the RCT, the MacAddr may be ignored.The VirtualChannel may be set to the channel on which the message will be sent to the elastic router 504.

[0079] As an example, to establish a unidirectional link from node A (e.g., transport component A (10.0.0.1)) to node B (e.g., transport component B (10.0.0.2)), one can: (1) create a SCTE {sCTI:1, rCTI:4, IP:10.0.0.2, VC:1, Mac:01-02-03-04-05-06} on transport component A; and (2) create a RCTE {rCTI:4, sCTI:1, IP:10.0.0.1, VC:2} on transport component B. In this example, this will take messages arriving from the elastic router on transport component A with DestIP == 10.0.0.2 and VC == 1 and send them in packets to transport component B. The packet header will have the rCTI field set to 4 (the rCTI value read from the SCT). Transport component B will access its RCT entry 4 and learn that the message should be output on VC2. It will also generate an ACK back to transmitting component A. In this packet, the value of the sCTI field is 1 (filled according to the sCTI value read from the RCT).

[0080] The LTL instance associated with the transport component 500 can buffer all send packets until it receives an acknowledgment (ACK) from the receive acceleration component. If the ACK for the connection does not arrive within a configurable timeout period, the packet can be retransmitted. In this example, all unacknowledged packets starting from the earliest will be retransmitted. The discarding of packets belonging to a given SCT may not change the behavior of any other connection - that is, packets for other connections may not be retransmitted. Since the LTL instance may require a reliable communication channel and packets may occasionally be lost on the network, in one example, a timeout-based retry mechanism may be used. If a packet does not receive an acknowledgment within a certain period of time, it may be retransmitted. The timeout period can be set via a configuration parameter.

[0081] The transport component 500 may also provide congestion control. If the LTL instance transmits data to a receiver that cannot absorb traffic at the full line rate, the congestion control functionality may allow it to gracefully reduce the frequency of packets sent to the destination node. Each LTL connection may have an associated inter-packet gap state that controls the minimum number of cycles between packet transmissions in the flow. When a new connection is created, the IPG may be set to 1, effectively allowing full use of any available bandwidth. If a timeout, ECN notification, or NACK occurs on a flow, the delay may be multiplied by the cfg.throttle_credit_multiple parameter (see Table 2) or increased by the cfg.throttle_credits_per_scrub parameter (see Table 2; depending on whether linear or exponential backoff is selected). Each ACK received may reduce the IPG by the cfg.throttle_credits_per_scrub parameter (see Table 2; depending on whether linear or exponential backoff is selected). The LTL instance may not increase the IPG of a flow more than once every predetermined period of time; for example, no more than every 2 microseconds (in this example, this may be controlled by the cfg.throttle_scrub_delay parameter (see Table 2)).

[0082] When the LTL instance associated with the transport component 500 successfully receives a packet, it will generate an acknowledgment (e.g., a packet with the ACK flag set). The acknowledgment (ACK) can include a sequence number that tells the sender the last packet that was successfully received and the SCTI to which the sender should credit the ACK (this value can be stored in the RCT of the ACK generator). For one example of LTL, the following rules can be used to generate ACKs: (1) If the RX sequence number matches the expected sequence number (in the RCT), an ACK is generated with the received sequence number; (2) If the RX sequence number is less than the expected sequence number, the packet is discarded, but an ACK is generated with the highest sequence number received (this can cover the case where a packet is sent twice (perhaps because of a timeout) but is then received correctly); (3) If the RX sequence number is greater than the expected sequence number, the packet is discarded and no ACK is generated. As explained below, in one example, acknowledgments (ACKs) can also be used to provide information updates to a load balancer.

[0083] Figure 6 A load balancer 600 according to an example is shown. In this example, the load balancer 600 can maintain a table of acceleration components that it monitors. Periodically, the load balancer 600 can receive information updates from the acceleration components. The information updates can be received as part of an ACK received by the intended recipient of a packet transmitted from a host (e.g., the host component described earlier). In one example, each update can contain the number of bytes processed by the acceleration component since the last update, the number of cycles spent processing the number of bytes, the number of bytes in the processing queue of the acceleration component, and the acceleration component identifier. The load balancer 600 can accumulate the total number of updates from each acceleration component, the number of bytes processed by the acceleration component, and the number of cycles spent processing the specified number of bytes. In this example, the load balancer 600 can calculate the processing rate associated with each acceleration component that it monitors. In addition, the load balancer 600 can track the hardware as a service (HaaS) input buffer occupancy threshold of each acceleration component. The use of the term HaaS in this disclosure does not mean that the disclosure is limited to use in a hardware as a service model. Instead, the term is used only to refer to the input buffer associated with the acceleration component that receives the input request from the service. By doing so, the load balancer 600 can track the health of the acceleration components, and the load balancer 600 can direct the request to the healthy acceleration component.

[0084] Continue to refer Figure 6, the load balancer 600 may include a register file (RF) 610, a request steering logic (RSL) 620, and a node monitoring logic (NML) 630. In this example, the RF 610 may be a module including configuration and statistical registers and may be read / written through a SOFTREG_* interface. The RSL 620 may include logic configured to direct packets to acceleration components based on the status of acceleration components in an available acceleration component group. The NML 630 may process information updates from acceleration components and track their status. Table 6 below shows a set of example configuration and statistical registers included in the example RF 610. The first column corresponds to the address of the register. The address for each register may be specified as an offset from a base address register in the RF 610. The second column corresponds to the name of the register (name); the third column corresponds to an access parameter (read, write, or read / write) (access); the fourth column corresponds to a default value (default value); and the fifth column provides a description (description).

[0085]

[0086]

[0087]

[0088]

[0089]

[0090]

[0091] Table 6

[0092] Still reference Figure 6 , RF 610 can control RSL 620 and NML 630 through LB_CTRL_* interface. RF 610 can receive information from RSL 620 and NML 630 via LB-STAT interface. RF 610 can also keep a table with acceleration component addresses so as to track each packet monitored by load balancer 600. The LBID of the current request can be sent to RF 610 via DST_TAG interface, and the corresponding address of the acceleration component can be read from the table with the acceleration component address. The address of the acceleration component can be sent to RSL 620 via LB_CTRL_* interface.

[0093] Continue to refer Figure 6, NML 630 can track the status of the acceleration component associated with the service. NML 630 can receive information from the acceleration component via REMOTE_INFO_* interface. When an information update is received from the acceleration component, the status can be updated. The acceleration component can be a local component or a remote acceleration component. Each information update contains the next information about the acceleration component. Table 7 below shows example information as part of the information update:

[0094]

[0095] Table 7

[0096] The NML 630 may maintain the next state of each acceleration component using the LB_DST_TABLE (an example is shown in Table 8 below).

[0097]

[0098] Table 8

[0099] In addition to the states described in Table 8 above, NML 630 tracks the NODE_ENABLETIME value for each acceleration component. In this example, the value indicates the time when the acceleration component is available (tracked via a counter). In this example, a round-robin scheme is used to read the NODE_ENABLETIME value for each acceleration component in each cycle. The read value is compared with the free-running counter (FREE-RUNNING-COUNTER), which is incremented in each cycle in this example. In those cases where the read NODE_ENABLETIME value is less than or equal to the free-running counter (FREE-RUNNING-COUNTER) value, the NODE_HEALTH value is set to a healthy state; in the opposite case, the NODE_HEALTH value is set to an unhealthy state. Alternatively, when, for example, the input buffer occupancy is lower than a low threshold (e.g., by comparing the buffer occupancy in bytes (e.g., received via an information update containing INFO_FIFOWORD) with the node's specified threshold (e.g., the threshold indicated in the LTHRESHOP_REG register of Table 6)), NML 630 can re-mark the node. The unhealthy state may be caused by some aspect of the acceleration component being negatively affected. For example, the FPGA may have entered an error state, or the FPGA may have been restarted, or its power may have been throttled. Although the software associated with the service can discover the problem with the FPGA, it may take several minutes to discover the problem. In contrast, a hardware-implemented load balancer can obtain status about the health of the FPGA in a much shorter time; for example, the time it takes for the FPGA to confirm (ACK or NACK) a packet. In addition, advantageously, the NML 630 does not need to sense these conditions, but can simply rely on tracking certain values ​​(e.g., NODE_HEALTH) values ​​to determine the state of the acceleration component. Finally, the software associated with the service may be sending requests corresponding to the service to different types of acceleration components (e.g., acceleration components with different FPGAs or different versions of FPGAs). Some of these acceleration components may be slower than other acceleration components. Since the software associated with the service may not be aware of these differences, it may continue to send requests to slower acceleration components, causing performance problems. However, faster feedback through the load balancer can help alleviate these performance problems.

[0100] At reset, all NODE_ENABLETIME values ​​and FREE-RUNNING-COUNTER values ​​can be set to zero, which can set all acceleration components to a healthy state. Since the NODE_ENABLETIME value is read in a round-robin manner (or some other fairness-based scheme), an acceleration component can be enabled for longer than it is set. However, in this example, since the maximum number of acceleration components is 8, this difference is not significant.

[0101] In this example, the NODE_ENABLETIME value is updated for each incoming information update. When the disabled period of the acceleration component based on the value in the information update expires, it is set to the current FREE-RUNNING-COUNTER value. When the value of the number of information updates received from the acceleration component (e.g., as indicated by NODE_WINCNT in Table 8) reaches the window size of the number of information updates from each acceleration component used to average its processing rate, (e.g., tracked in a register named WNDSIZE_REG), the processing rate of the acceleration component (e.g., NODE_RDRATE in Table 8) is updated based on the following calculation. In addition, each value corresponding to NODE_WINCNT, NODE_RDBYTES, and NODE_RDCYCLES in Table 7 is reset to zero. In one example, the processing rate of the acceleration component is represented by an 8-bit NODE_RDRATE vector, where each bit corresponds to the next rate, as shown in Table 9 below. Although Table 9 shows a certain correspondence between bit values ​​and processing rates, other mappings can also be used to specify processing rates.

[0102]

[0103] Table 9

[0104] In this example, each information update from the acceleration component contains information about the HaaS ingress buffer occupancy of the acceleration component - INFO_FIFOWORD. When its value is greater than or equal to the value specified by the HTHRESHVAL_REG register, the acceleration component is assigned a penalty weight, which effectively changes its status to unhealthy. In one example, the penalty is calculated based on the mode specified in the HTHRESHACT_REG register and the value specified in the HTHRESHOP_REG register.

[0105] In one example, the penalty is calculated using the following equation: Penalty = OCCUPANCY_WEIGHT + HTHRESHOP_REG. OCCUPANCY_WEIGHT is calculated based on the bit values ​​specified by the HTHRESHACT_REG register described earlier with respect to Table 6. Table 10 below shows an example of calculation of the bit values ​​specified by the HTHRESHACT_REG register and the PENALTY value.

[0106]

[0107] Table 10

[0108] Although Table 10 shows a certain correspondence between the bit values ​​in the left column and the penalty values, additional soft registers can be used to change the parameters in the table. When the HTHRESHACT_REG register is set to scaling mode (bit 5), OCCUPANCY_WEIGHT is calculated by multiplying the HaaS input buffer occupancy of the acceleration component by the scaling factor. As shown in Table 11 below, the scaling factor is set based on the processing rate as shown in Table 9 above.

[0109] NODE_RDRATE setting bits 7 6 5 4 3 2 1 0 Scaling factor value 512 64 16 2 2 16 64 512

[0110] Table 11

[0111] The scaling factor values ​​may also vary depending on the type of application or workload running on the platform. Therefore, these scaling factors are only examples and they may be configured in other ways.

[0112] In this example, the request from the host contains a load balancer identifier (LBID). The RSL 620 can use the LBID to retrieve the health status of the corresponding acceleration component (e.g., NODE_HEALTH). In this example, all requests from the host can be received by the load balancer 600 via the RTR_*_IN interface and sent to the elastic router (e.g., RTR_*_OUT interface) via the RTR_*_OUT interface. Figure 5 ER 504). ER 504 can route the request to a specific processing node (e.g., another acceleration component) based on the address (e.g., IP address) specified in the request. In this way, load balancer 600 can perform request steering by specifying the IP address of the acceleration component to which the elastic router should route the request. The specified IP address can be the original IP address provided by the host or another different IP address selected by RSL 620. Table 12 below shows an example interface used by load balancer 600 to direct requests from the host.

[0113] name width describe RTR_VALID_IN 1 The signal is asserted when there is a valid phit on the interface. RTR_IF_IN.DATA 256 Data phit RTR_IF_IN.FIRST 1 When the signal is asserted, it indicates the first phit of the logic message. RTR_IF_IN.DST_IP 32 The destination IP address of the message

[0114] Table 12

[0115] In this example, the term phit (physical flit) can mean a unit of data transmitted to the load balancer 600 via an interface in one clock cycle. In other words, the term phit can refer to the physical width of the interface. As previously noted, the term flit can refer to a number of units of phit. In this example, the load balancer 600 can assume that the first data phit of a message contains a network header starting from the least significant bit of the phit. Table 13 shown below provides an example of a header structure.

[0116]

[0117]

[0118] Table 13

[0119] In this example, when the load balancer 600 receives a valid (e.g., indicated by RTF_VALID_IN_) first phit (e.g., indicated by RTR_IF_IN.FIRST) of a message (e.g., RTR_IF_IN.DATA), the load balancer 600 may extract the LBID from the network header (e.g., a network header having the structure shown in Table 13). Based on the state provided by the NML 630 to the processing node with the extracted LBID, the load balancer 600 either uses the same LBID, RTR_IF_IN.DST_IP, and DST_IP in the network header for the output router interface (RTR_IF_OUT), or it changes the values ​​used for these fields based on information from the RSL 620. In the case when the load balancer 600 changes the values ​​of these fields, the processing request is sent to a different acceleration component (e.g., FPGA node) instead of to the original destination specified by the requesting host.

[0120] When the specified acceleration component is in an unhealthy state and there are other available acceleration components, RSL 620 effectively forwards the request to the next acceleration component pointed to by the global round-robin scheme or another scheme. When no other acceleration components are available, in this example, the request will be sent to its original destination. In addition, in one example, sending a request to the original acceleration component can change the state of the original acceleration component to a healthy state.

[0121] Figure 7A system 700 including load balancing according to one example is shown. The system 700 may include a host 702 coupled to a local acceleration component (AC) 706 and to a load balancer (LB) 710 via a high-speed link (e.g., PCIe 704). The LB 710 may be coupled to a resilient router (ER) 712. The ER 712 may be coupled to a lightweight transport layer (LTL) 714, which may be coupled to a network 720. The network 720 may allow the host 702 to send a request message corresponding to a service not only to the local AC 706, but also to a remote acceleration component. Thus, the network 720 may be coupled to the LTL 722, which may be coupled to the ER 724. The ER 724 may be further coupled to the acceleration component (AC) 726. The network 720 may be further coupled to the LTL 732, which may be coupled to the ER 734. ER 734 may be further coupled to an acceleration component (AC) 736. Network 720 may be further coupled to LTL 742, which may be coupled to ER 744. ER 744 may be further coupled to an acceleration component (AC) 746. In this example, host 702 may send a message 770 corresponding to a service (e.g., a ranking service) to any one of AC 706, AC 726, AC 736, and AC 746. Message 770 may be split into packets that may have a size determined by a maximum transmission unit (MTU) of network 720. Thus, in this example, during transmission through network 720, message 770 may be split into MTU 772, MTU 774, MTU 776, and MTU 778. Each LTL block may add metadata about load balancing to each MTU. Each acceleration component may include an input buffer for storing packets received from host 702. Thus, in this example, AC 726 may receive packets via input buffer 725; AC 736 may receive packets via input buffer 735; and AC 746 may receive packets via buffer 745. Figure 7 A certain number of components are shown arranged in a certain manner, but system 700 may include more or fewer components arranged differently.

[0122] Continue to refer Figure 7As explained earlier, LB 710 may include node monitoring logic (e.g., NML 630) and request steering logic (e.g., RSL 620). NML 630 may use register files (e.g., RF 610) to track the input buffer occupancy of any acceleration component to which the host may send messages. As an example, NML 630 may track the LBID associated with each destination acceleration component. NML 630 may therefore have a proxy (e.g., input buffer occupancy metric) for the load experienced by the destination acceleration component. Each ACK or NACK from the acceleration component may convey information to NML 630. As described earlier, the information may relate to the total number of bytes processed by the acceleration component and the time spent processing these bytes. In this way, NML 630 may determine the processing rate of the request.

[0123] Figure 8 A flow chart 800 of a method for load balancing in a system including a host according to an example is shown, the host being configured to send a message corresponding to a service to any one of a set of acceleration components. Step 810 may include tracking an input buffer occupancy corresponding to each of the set of acceleration components. In one example, the load balancer 600 may perform this step. Thus, in this example, the NML 630 may track the input buffer occupancy on the host side, as explained earlier with respect to Table 6 and the related description.

[0124] Step 820 may include: when the first input buffer occupancy of the first acceleration component in the set of acceleration components meets the predetermined criteria, generating a first penalty value for the first acceleration component based on at least the first input buffer occupancy associated with the first acceleration component, and comparing a second value indicating the availability of the second acceleration component to receive messages from the host with a second dynamically updated value to determine whether the host can send the next message to the second acceleration component. As part of this step, the NML 630 may determine when the predetermined criteria are met. The predetermined criteria may include determining when the value of INFO_FIFOWORD (e.g., received by the NML 630 as part of an information update from the acceleration component) is greater than or equal to a value specified by the HTHRESHVAL_REG register. The NML 630 may generate the penalty value by calculating the penalty value based on a mode specified in the HTHRESHACT_REG register and a value specified in the HTHRESHOP_REG register. For example, the NML 630 may generate the penalty value using the following equation: Penalty=OCCUPANCY_WEIGHT+HTHRESHOP_REG. Table 10 described earlier provides various modes for the penalty value. The NML 630 can compare a value such as the NODE_ENABLETIME value to a free-running counter. In this example, a round-robin scheme is used for each acceleration component to read the NODE_ENABLETIME value in each cycle. The read value is compared to the free-running counter (FREE-RUNNING-COUNTER), which in this example is incremented in each cycle. When it is determined that the acceleration component is in a position to receive a request, the NML 630 can communicate it to the RSL 620. In addition, if necessary, the RSL 620 can also use the LBID to retrieve the health status of the corresponding acceleration component (e.g., NODE_HEALTH). The RSL 620 can forward the request to the next acceleration component pointed to by the global round-robin scheme or other scheme. Although Figure 8 A specific number of steps are shown listed in a specific order, but there may be fewer or more steps and the steps may be performed in a different order.

[0125] Fig. 9 A flow chart 900 for load balancing in a system including a host according to an example is shown, the host being configured to send a message corresponding to a service to any one of a set of acceleration components. Step 910 may include tracking an input buffer occupancy and a processing rate corresponding to each acceleration component in the set of acceleration components. In one example, the load balancer 600 may perform this step. Thus, in this example, the NML 630 may track the input buffer occupancy on the host side, as explained earlier with respect to Table 6 and the related description.

[0126] Step 920 may include: as long as the first acceleration component is determined to be in a healthy state based on a comparison of the first value indicating the availability of the first acceleration component with the first dynamically updated value, the host initiates the transmission of a first message to the first acceleration component selected from the set of acceleration components. NML 630 can compare a value such as NODE_ENABLETIME value with a free-running counter. In this example, a loop scheme is used to read the NODE_ENABLETIME value for each acceleration component in each cycle. The read value is compared with a free-running counter (FREE-RUNNING-COUNTER), which is incremented in each cycle in this example. When it is determined that the acceleration component is in a position to receive a request, NML 630 can communicate it to RSL 620, and RSL 620 can send the request to the acceleration component.

[0127] Step 930 may include: when the first input buffer occupancy of the first acceleration component in the set of acceleration components meets the predetermined criteria, at least based on the first input buffer occupancy associated with the first acceleration component, generating a first penalty value for the first acceleration component, and comparing a second value indicating the availability of the second acceleration component to receive messages from the host with a second dynamically updated value to determine whether the host can send the next message to the second acceleration component. The NML 630 may determine when the predetermined criteria are met. The predetermined criteria may include determining when the value of INFO_FIFOWORD (e.g., received by the NML 630 as part of an information update from the acceleration component) is greater than or equal to a value specified by the HTHRESHVAL_REG register. The NML 630 may generate the penalty value by calculating the penalty value based on a mode specified in the HTHRESHACT_REG register and a value specified in the HTHRESHOP_REG register. For example, the NML 630 may generate the penalty value using the following equation: Penalty=OCCUPANCY_WEIGHT+HTHRESHOP_REG. Table 10 described earlier provides various modes for the penalty value. The NML 630 can compare a value such as the NODE_ENABLETIME value to a free-running counter. In this example, a round-robin scheme is used to read the NODE_ENABLETIME value in each cycle for each acceleration component. The read value is compared to the free-running counter (FREE-RUNNING-COUNTER), which in this example is incremented in each cycle. When it is determined that the acceleration component is in a position to receive the request, the NML 630 can communicate it to the RSL 620. In addition, if necessary, the RSL 620 can also use the LBID to retrieve the health status of the corresponding acceleration component (e.g., NODE_HEALTH). The RSL 620 can forward the request to the next acceleration component pointed to by the global round-robin scheme or other scheme. Although Fig. 9 A specific number of steps are shown listed in a specific order, but there may be fewer or more steps and the steps may be performed in a different order.

[0128] In summary, the present disclosure relates to a method in a system including a host, the host being configured to send a message corresponding to a service to any one acceleration component in a set of acceleration components. The method may include: tracking the input buffer occupancy corresponding to each acceleration component in the set of acceleration components. The method may also include: when the first input buffer occupancy of a first acceleration component in the set of acceleration components meets a predetermined criterion, generating a first penalty value for the first acceleration component based at least on the first input buffer occupancy associated with the first acceleration component, and comparing a second value indicating the availability of a second acceleration component of a message received from the host with a second dynamically updated value to determine whether the host can send a next message to the second acceleration component.

[0129] The method may further include: if the second value indicating the availability of the second acceleration component is greater than the second dynamic update value, directing the next message from the host to the second acceleration component. The method may further include: if the second value indicating the availability of the second acceleration component is less than or equal to the second dynamic update value, directing the next message from the host to a third acceleration component selected from the set of acceleration components as long as the third acceleration component is determined to be in a healthy state based on a comparison of a third value indicating the availability of the third acceleration component with the third dynamic update value.

[0130] The first dynamically updated value may include a first value of a first free running counter managed by the load balancer, and the second dynamically updated value may include a second value of a second free running counter managed by the load balancer.The method may also include periodically decrementing the first penalty value by a specified value.

[0131] The first penalty value may include a fixed penalty value equal to a high occupancy threshold associated with the first input buffer occupancy. The first penalty value may include a sum of the fixed penalty value and the first input buffer occupancy in bytes. The first penalty value may include one of the following: a sum of the fixed penalty and the first input buffer occupancy multiplied by two, a sum of the fixed penalty and the first input buffer occupancy multiplied by sixteen, a sum of the fixed penalty and the first input buffer occupancy multiplied by sixty-four.

[0132] In another example, the present disclosure relates to a system including an acceleration component set and a host, the host being configured to send a message corresponding to a service to any acceleration component in the acceleration component set. The system may also include a load balancer, the load balancer being configured to: (1) track an input buffer occupancy corresponding to each acceleration component in the acceleration component set, and (2) when a first input buffer occupancy of a first acceleration component in the acceleration component set meets a predetermined criterion, generate a first penalty value for the first acceleration component based at least on the first input buffer occupancy associated with the first acceleration component, and compare a second value indicating the availability of a second acceleration component to receive a message from the host with a second dynamically updated value to determine whether the host can send a next message to the second acceleration component.

[0133] The load balancer may also be configured to: if the second value indicating the availability of the second acceleration component is greater than the second dynamically updated value, then direct the next message from the host to the second acceleration component. The load balancer may also be configured to: if the second value indicating the availability of the second acceleration component is less than or equal to the second dynamically updated value, then as long as the third acceleration component is determined to be in a healthy state based on a comparison of the third value indicating the availability of the third acceleration component with the third dynamically updated value, then direct the next message from the host to a third acceleration component selected from the set of acceleration components.

[0134] The first dynamically updated value may include a first value of a first free-running counter managed by the load balancer, and the second dynamically updated value may include a second value of a second free-running counter managed by the load balancer. The load balancer may also be configured to periodically decrement the first penalty value by a specified value. The first penalty value may include at least a fixed penalty value equal to a high occupancy threshold associated with the first input buffer occupancy.

[0135] In yet another example, the present disclosure relates to a method in a system including a host, the host being configured to send a message corresponding to a service to any one of a set of acceleration components. The method may include: tracking an input buffer occupancy and a processing rate corresponding to each of the set of acceleration components. The method may also include: as long as the first acceleration component is determined to be in a healthy state based on a comparison of a first value indicating the availability of the first acceleration component with a first dynamically updated value, the host initiates the transmission of a first message to a first acceleration component selected from the set of acceleration components. The method may also include: when the first input buffer occupancy of the first acceleration component meets a predetermined criterion, generating a first penalty value for the first acceleration component based at least on the first input buffer occupancy and the first processing rate associated with the first acceleration component, and comparing a second value indicating the availability of a second acceleration component receiving a message from the host with a second dynamically updated value to determine whether the host can send the next message to the second acceleration component.

[0136] The method may further include: if the second value indicating the availability of the second acceleration component is greater than the second dynamic update value, directing the next message from the host to the second acceleration component. The method may further include: if the second value indicating the availability of the second acceleration component is less than or equal to the second dynamic update value, directing the next message from the host to a third acceleration component selected from the set of acceleration components as long as the third acceleration component is determined to be in a healthy state based on a comparison of a third value indicating the availability of the third acceleration component with the third dynamic update value.

[0137] The first dynamically updated value may include a first value of a first free running counter managed by the load balancer, and the second dynamically updated value may include a second value of a second free running counter managed by the load balancer. The method may also include periodically decrementing the first penalty value by a specified value. The first penalty value may include a fixed penalty value equal to a high occupancy threshold associated with the first input buffer occupancy.

[0138] It should be understood that the systems, methods, modules and components described herein are merely exemplary. Alternatively or additionally, the functionality described herein may be performed at least in part by one or more hardware logic components. For example, but not limited to, illustrative types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip systems (SOCs), complex programmable logic devices (CPLDs), etc. In an abstract but still clear sense, any component arrangement that achieves the same functionality is effectively "associated" so that the desired functionality is achieved. Therefore, any two components that are combined to achieve a specific functionality herein can be considered to be "associated" with each other so that the desired functionality is achieved, regardless of the architecture or intermediate components. Similarly, any two components so associated can also be considered to be "operably connected" or "coupled" to each other to achieve the desired functionality.

[0139] Functionality associated with some examples described in this disclosure may also include instructions stored in non-transient media. The term "non-transient media" as used herein refers to any medium that stores data and / or instructions that cause a machine to operate in a particular manner. Exemplary non-transient media include non-volatile media and / or volatile media. Non-volatile media include, for example, hard disks, solid-state drives, magnetic disks or tapes, optical disks or tapes, flash memory, EPROM, NVRAM, PRAM or other such media, or networked versions of such media. Volatile media include, for example, dynamic memories such as DRAM, SRAM, cache or other such media. Non-transient media are different from transmission media, but can be used in conjunction with transmission media. Transmission media are used to transmit data and / or instructions to or from a machine. Exemplary transmission media include coaxial cables, fiber optic cables, copper wires, and wireless media, such as radio waves.

[0140] In addition, those skilled in the art will recognize that the boundaries between the functionality of the above operations are merely illustrative. The functionality of multiple operations can be combined into a single operation, and / or the functionality of a single operation can be distributed in additional operations. In addition, alternative embodiments can include multiple instances of specific operations, and the order of operations can be changed in various other embodiments.

[0141] Although the present disclosure provides specific examples, various modifications and changes may be made without departing from the scope of the present disclosure as set forth in the following claims. Therefore, the specification and drawings are to be regarded as illustrative rather than restrictive, and all such modifications are intended to be included within the scope of the present disclosure. Any benefits, advantages, or solutions to problems described herein with respect to specific examples should not be construed as key, required, or essential features or elements of any or all claims.

[0142] Furthermore, as used herein, the terms "a" or "an" are defined as one or more than one. Furthermore, the use of introductory phrases such as "at least one" and "one or more" in the claims should not be interpreted as implying that the introduction of another claim element by the indefinite article "a" or "an" limits any particular claim containing such introduced claim element to an invention containing only one such element, even if the same claim includes the introductory phrases "one or more" or "at least one" and an indefinite article such as "a" or "an". The same applies to the use of definite articles.

[0143] Unless otherwise stated, terms such as "first" and "second" are used to arbitrarily distinguish between the elements such terms describe. Therefore, these terms are not necessarily intended to indicate temporal or other prioritization of such elements.

Claims

1. A method implemented in a system including a host, wherein the host is configured to send a message corresponding to a service to any acceleration component in a set of acceleration components, the method comprising: Tracking an input buffer occupancy corresponding to each acceleration component in the set of acceleration components; as well as When a first input buffer occupancy of a first acceleration component in the set of acceleration components meets a predetermined criterion, a first penalty value for the first acceleration component is generated based at least on the first input buffer occupancy associated with the first acceleration component, and a second value indicating the availability of a second acceleration component for receiving messages from the host is compared with a second dynamically updated value to determine whether the host can send a next message to the second acceleration component.

2. The method according to claim 1, further comprising: If the second value indicating the availability of the second acceleration component is greater than the second dynamically updated value, directing the next message from the host to the second acceleration component.

3. The method according to claim 1, further comprising: If the second value indicating the availability of the second acceleration component is less than or equal to the second dynamic update value, the next message is directed from the host to the third acceleration component selected from the set of acceleration components as long as the third acceleration component is determined to be in a healthy state based on a comparison of the third value indicating the availability of the third acceleration component with the third dynamic update value.

4. The method according to claim 1, further comprising: The first penalty value is periodically reduced by a specified value. 5 . The method of claim 1 , wherein the first penalty value comprises a fixed penalty value equal to a high occupancy threshold associated with the first input buffer occupancy. The method of claim 1 , wherein the first penalty value comprises a sum of a fixed penalty value in bytes and the first input buffer occupancy rate.

7. The method according to claim 6, wherein the first penalty value comprises one of the following: the sum of the fixed penalty and the first input buffer occupancy multiplied by two, the sum of the fixed penalty and the first input buffer occupancy multiplied by sixteen, and the sum of the fixed penalty and the first input buffer occupancy multiplied by sixty-four.

8. The method of claim 3, wherein the second dynamically updated value comprises a first value of a first free running counter managed by a load balancer, and wherein the third dynamically updated value comprises a second value of a second free running counter managed by the load balancer.

9. A system for monitoring and directing service requests of an acceleration component, comprising: Acceleration component collection; A host, wherein the host is configured to send a message corresponding to a service to any acceleration component in the acceleration component set; as well as A load balancer, the load balancer being configured to: (1) track an input buffer occupancy corresponding to each acceleration component in the set of acceleration components, and (2) when a first input buffer occupancy of a first acceleration component in the set of acceleration components meets a predetermined criterion, generate a first penalty value for the first acceleration component based at least on the first input buffer occupancy associated with the first acceleration component, and compare a second value indicating the availability of a second acceleration component for receiving a message from the host with a second dynamically updated value to determine whether the host is able to send a next message to the second acceleration component.

10. The system of claim 9, wherein the load balancer is further configured to direct the next message from the host to the second acceleration component if the second value indicating the availability of the second acceleration component is greater than the second dynamically updated value.

11. The system of claim 9, wherein the load balancer is further configured to direct the next message from the host to the third acceleration component selected from the set of acceleration components as long as the third acceleration component is determined to be in a healthy state based on a comparison of the third value indicating the availability of the second acceleration component with the third dynamic update value if the second value indicating the availability of the second acceleration component is less than or equal to the second dynamic update value.

12. The system of claim 9, wherein the load balancer is further configured to periodically decrement the first penalty value by a specified value.

13. The system of claim 9, wherein the first penalty value comprises at least a fixed penalty value equal to a high occupancy threshold associated with the first input buffer occupancy.

14. The system of claim 11, wherein the second dynamically updated value comprises a first value of a first free running counter managed by a load balancer, and wherein the third dynamically updated value comprises a second value of a second free running counter managed by the load balancer.

15. A method implemented in a system including a host, wherein the host is configured to send a message corresponding to a service to any acceleration component in a set of acceleration components, the method comprising: Tracking an input buffer occupancy and a processing rate corresponding to each acceleration component in the set of acceleration components; The host initiates transmission of a first message to a first acceleration component selected from among the set of acceleration components as long as the first acceleration component is determined to be in a healthy state based on a comparison of a first value indicating availability of the first acceleration component and a first dynamically updated value; as well as When the first input buffer occupancy of the first acceleration component meets a predetermined criterion, a first penalty value for the first acceleration component is generated based at least on the first input buffer occupancy and a first processing rate associated with the first acceleration component, and a second value indicating the availability of a second acceleration component for receiving messages from the host is compared with a second dynamically updated value to determine whether the host can send a next message to the second acceleration component.

16. The method according to claim 15, further comprising: If the second value indicating the availability of the second acceleration component is greater than the second dynamically updated value, directing the next message from the host to the second acceleration component.

17. The method according to claim 15, further comprising: If the second value indicating the availability of the second acceleration component is less than or equal to the second dynamic update value, the next message is directed from the host to the third acceleration component selected from the set of acceleration components as long as the third acceleration component is determined to be in a healthy state based on a comparison of the third value indicating the availability of the third acceleration component with the third dynamic update value.

18. The method according to claim 15, further comprising: The first penalty value is periodically reduced by a specified value.

19. The method of claim 15, wherein the first penalty value comprises at least a fixed penalty value equal to a high occupancy threshold associated with the first input buffer occupancy.

20. The method of claim 17, wherein the second dynamically updated value comprises a first value of a first free running counter managed by a load balancer, and wherein the third dynamically updated value comprises a second value of a second free running counter managed by the load balancer.

Images